2023-10-06T03:17:00.058 INFO:root:teuthology version: 0.0.1.dev194+g54e62bc 2023-10-06T03:17:00.058 DEBUG:teuthology.run:Teuthology command: teuthology --name yuriw-2023-10-05_23:52:50-fs-wip-yuri-testing-2023-10-05-0730-quincy-distro-default-smithi --description fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/snap_schedule_snapdir} --verbose --owner scheduled_yuriw@teuthology --archive /home/teuthworker/archive/yuriw-2023-10-05_23:52:50-fs-wip-yuri-testing-2023-10-05-0730-quincy-distro-default-smithi/7412204 -- /home/teuthworker/archive/yuriw-2023-10-05_23:52:50-fs-wip-yuri-testing-2023-10-05-0730-quincy-distro-default-smithi/7412204/orig.config.yaml 2023-10-06T03:17:00.111 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-10-06T03:17:00.206 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2023-10-05_23:52:50-fs-wip-yuri-testing-2023-10-05-0730-quincy-distro-default-smithi/7412204 branch: wip-yuri-testing-2023-10-05-0730-quincy description: fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/snap_schedule_snapdir} email: yweinste@redhat.com first_in_suite: false job_id: '7412204' kernel: kdb: true sha1: distro ktype: distro last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi name: yuriw-2023-10-05_23:52:50-fs-wip-yuri-testing-2023-10-05-0730-quincy-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: ubuntu os_version: '20.04' overrides: admin_socket: branch: wip-yuri-testing-2023-10-05-0730-quincy ceph: cephfs: ec_profile: - m=2 - k=2 - crush-failure-domain=osd conf: client: client mount timeout: 600 client snapdir: .customsnapfuse debug client: 20 debug ms: 1 pid file: '' rados mon op timeout: 900 rados osd op timeout: 900 mds: debug mds: 20 debug ms: 1 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - overall HEALTH_ - \(FS_DEGRADED\) - \(MDS_FAILED\) - \(MDS_DEGRADED\) - \(FS_WITH_FAILED_MDS\) - \(MDS_DAMAGE\) - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(FS_INLINE_DATA_DEPRECATED\) - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding sha1: 736ac5ba72cb47af7d6f6666b1e100f7e20a20e2 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: 736ac5ba72cb47af7d6f6666b1e100f7e20a20e2 kclient: snapdirname: .customsnapkernel thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-yuri-testing-2023-10-05-0730-quincy sha1: f1e315a9732d36c05ec316678e2397dc58561c7f owner: scheduled_yuriw@teuthology priority: 75 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mgr.y - mds.a - mds.c - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - client.2 - - mon.b - mon.c - mgr.x - mds.b - mds.d - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - client.3 seed: 7519 sha1: 736ac5ba72cb47af7d6f6666b1e100f7e20a20e2 sleep_before_teardown: 0 subset: 111/12000 suite: fs suite_branch: wip-yuri-testing-2023-10-05-0730-quincy suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_f1e315a9732d36c05ec316678e2397dc58561c7f/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: f1e315a9732d36c05ec316678e2397dc58561c7f targets: smithi110.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFyT/r7xuKfmg50w0OdQebr4A/ptJo1Zk6DyZNZNZM/q5rW+sznh+wvHqeT3e4YtCdZLaXyvMms8o8u8H5nXOYM= smithi192.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFyT/r7xuKfmg50w0OdQebr4A/ptJo1Zk6DyZNZNZM/q5rW+sznh+wvHqeT3e4YtCdZLaXyvMms8o8u8H5nXOYM= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - ceph: null - ceph-fuse: null - cephfs_test_runner: modules: - tasks.cephfs.test_snap_schedules.TestSnapSchedulesSnapdir teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return false end - local function is_kdistro() return false end - local function is_fuse() return true end - local function syntax_version() return '' end teuthology_branch: main teuthology_sha1: 54e62bcbac4e53d9685e08328b790d3b20d71cae timestamp: 2023-10-05_23:52:50 tube: smithi user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2830 2023-10-06T03:17:00.206 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_f1e315a9732d36c05ec316678e2397dc58561c7f/qa; will attempt to use it 2023-10-06T03:17:00.207 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_f1e315a9732d36c05ec316678e2397dc58561c7f/qa/tasks 2023-10-06T03:17:00.207 INFO:teuthology.run_tasks:Running task internal.check_packages... 2023-10-06T03:17:00.209 INFO:teuthology.task.internal:Checking packages... 2023-10-06T03:17:00.232 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'default' and ceph hash '736ac5ba72cb47af7d6f6666b1e100f7e20a20e2' 2023-10-06T03:17:00.232 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-10-06T03:17:00.232 INFO:teuthology.packaging:ref: None 2023-10-06T03:17:00.232 INFO:teuthology.packaging:tag: None 2023-10-06T03:17:00.233 INFO:teuthology.packaging:branch: wip-yuri-testing-2023-10-05-0730-quincy 2023-10-06T03:17:00.233 INFO:teuthology.packaging:sha1: 736ac5ba72cb47af7d6f6666b1e100f7e20a20e2 2023-10-06T03:17:00.233 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&ref=wip-yuri-testing-2023-10-05-0730-quincy 2023-10-06T03:17:00.375 INFO:teuthology.task.internal:Found packages for ceph version 17.2.6-1296-gf1e315a9-1focal 2023-10-06T03:17:00.375 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2023-10-06T03:17:00.382 INFO:teuthology.task.internal:no buildpackages task found 2023-10-06T03:17:00.382 INFO:teuthology.run_tasks:Running task internal.save_config... 2023-10-06T03:17:00.406 INFO:teuthology.task.internal:Saving configuration 2023-10-06T03:17:00.422 INFO:teuthology.run_tasks:Running task internal.check_lock... 2023-10-06T03:17:00.428 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-10-06T03:17:00.454 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi110.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-10-05_23:52:50-fs-wip-yuri-testing-2023-10-05-0730-quincy-distro-default-smithi/7412204', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '20.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-10-06 03:11:12.020242', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIGETOy3xONfbx7kUFCNiUtRLNfkcrcEsWzPdq6Np/6Bw'} 2023-10-06T03:17:00.478 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi192.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-10-05_23:52:50-fs-wip-yuri-testing-2023-10-05-0730-quincy-distro-default-smithi/7412204', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '20.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-10-06 03:11:12.018725', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIGETOy3xONfbx7kUFCNiUtRLNfkcrcEsWzPdq6Np/6Bw'} 2023-10-06T03:17:00.478 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2023-10-06T03:17:00.484 INFO:teuthology.task.internal:roles: ubuntu@smithi110.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'client.2'] 2023-10-06T03:17:00.484 INFO:teuthology.task.internal:roles: ubuntu@smithi192.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'client.3'] 2023-10-06T03:17:00.484 INFO:teuthology.run_tasks:Running task console_log... 2023-10-06T03:17:00.553 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f4a66a4b040>, signals=[15]) 2023-10-06T03:17:00.554 INFO:teuthology.run_tasks:Running task internal.connect... 2023-10-06T03:17:00.559 INFO:teuthology.task.internal:Opening connections... 2023-10-06T03:17:00.560 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi110.front.sepia.ceph.com 2023-10-06T03:17:00.561 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi110.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-06T03:17:00.611 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi192.front.sepia.ceph.com 2023-10-06T03:17:00.611 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi192.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-06T03:17:00.660 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2023-10-06T03:17:00.666 DEBUG:teuthology.orchestra.run.smithi110:> uname -m 2023-10-06T03:17:00.698 INFO:teuthology.orchestra.run.smithi110.stdout:x86_64 2023-10-06T03:17:00.699 DEBUG:teuthology.orchestra.run.smithi110:> cat /etc/os-release 2023-10-06T03:17:00.751 INFO:teuthology.orchestra.run.smithi110.stdout:NAME="Ubuntu" 2023-10-06T03:17:00.751 INFO:teuthology.orchestra.run.smithi110.stdout:VERSION="20.04.4 LTS (Focal Fossa)" 2023-10-06T03:17:00.751 INFO:teuthology.orchestra.run.smithi110.stdout:ID=ubuntu 2023-10-06T03:17:00.751 INFO:teuthology.orchestra.run.smithi110.stdout:ID_LIKE=debian 2023-10-06T03:17:00.751 INFO:teuthology.orchestra.run.smithi110.stdout:PRETTY_NAME="Ubuntu 20.04.4 LTS" 2023-10-06T03:17:00.751 INFO:teuthology.orchestra.run.smithi110.stdout:VERSION_ID="20.04" 2023-10-06T03:17:00.752 INFO:teuthology.orchestra.run.smithi110.stdout:HOME_URL="https://www.ubuntu.com/" 2023-10-06T03:17:00.752 INFO:teuthology.orchestra.run.smithi110.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2023-10-06T03:17:00.752 INFO:teuthology.orchestra.run.smithi110.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2023-10-06T03:17:00.752 INFO:teuthology.orchestra.run.smithi110.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2023-10-06T03:17:00.752 INFO:teuthology.orchestra.run.smithi110.stdout:VERSION_CODENAME=focal 2023-10-06T03:17:00.752 INFO:teuthology.orchestra.run.smithi110.stdout:UBUNTU_CODENAME=focal 2023-10-06T03:17:00.753 INFO:teuthology.lock.ops:Updating smithi110.front.sepia.ceph.com on lock server 2023-10-06T03:17:00.779 DEBUG:teuthology.orchestra.run.smithi192:> uname -m 2023-10-06T03:17:00.785 INFO:teuthology.orchestra.run.smithi192.stdout:x86_64 2023-10-06T03:17:00.786 DEBUG:teuthology.orchestra.run.smithi192:> cat /etc/os-release 2023-10-06T03:17:00.791 INFO:teuthology.orchestra.run.smithi192.stdout:NAME="Ubuntu" 2023-10-06T03:17:00.791 INFO:teuthology.orchestra.run.smithi192.stdout:VERSION="20.04.4 LTS (Focal Fossa)" 2023-10-06T03:17:00.791 INFO:teuthology.orchestra.run.smithi192.stdout:ID=ubuntu 2023-10-06T03:17:00.791 INFO:teuthology.orchestra.run.smithi192.stdout:ID_LIKE=debian 2023-10-06T03:17:00.791 INFO:teuthology.orchestra.run.smithi192.stdout:PRETTY_NAME="Ubuntu 20.04.4 LTS" 2023-10-06T03:17:00.792 INFO:teuthology.orchestra.run.smithi192.stdout:VERSION_ID="20.04" 2023-10-06T03:17:00.792 INFO:teuthology.orchestra.run.smithi192.stdout:HOME_URL="https://www.ubuntu.com/" 2023-10-06T03:17:00.792 INFO:teuthology.orchestra.run.smithi192.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2023-10-06T03:17:00.792 INFO:teuthology.orchestra.run.smithi192.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2023-10-06T03:17:00.792 INFO:teuthology.orchestra.run.smithi192.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2023-10-06T03:17:00.792 INFO:teuthology.orchestra.run.smithi192.stdout:VERSION_CODENAME=focal 2023-10-06T03:17:00.792 INFO:teuthology.orchestra.run.smithi192.stdout:UBUNTU_CODENAME=focal 2023-10-06T03:17:00.792 INFO:teuthology.lock.ops:Updating smithi192.front.sepia.ceph.com on lock server 2023-10-06T03:17:00.811 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2023-10-06T03:17:00.821 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2023-10-06T03:17:00.826 INFO:teuthology.task.internal:Checking for old test directory... 2023-10-06T03:17:00.826 DEBUG:teuthology.orchestra.run.smithi110:> test '!' -e /home/ubuntu/cephtest 2023-10-06T03:17:00.828 DEBUG:teuthology.orchestra.run.smithi192:> test '!' -e /home/ubuntu/cephtest 2023-10-06T03:17:00.840 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2023-10-06T03:17:00.846 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2023-10-06T03:17:00.846 DEBUG:teuthology.orchestra.run.smithi110:> test -z $(ls -A /var/lib/ceph) 2023-10-06T03:17:00.875 DEBUG:teuthology.orchestra.run.smithi192:> test -z $(ls -A /var/lib/ceph) 2023-10-06T03:17:00.892 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2023-10-06T03:17:00.950 INFO:teuthology.run_tasks:Running task kernel... 2023-10-06T03:17:00.965 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2023-10-06T03:17:00.965 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2023-10-06T03:17:00.966 DEBUG:teuthology.orchestra.run.smithi110:> test -f /run/.containerenv -o -f /.dockerenv 2023-10-06T03:17:00.966 DEBUG:teuthology.orchestra.run.smithi192:> test -f /run/.containerenv -o -f /.dockerenv 2023-10-06T03:17:00.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-06T03:17:00.972 DEBUG:teuthology.orchestra.run.smithi110:> uname -r 2023-10-06T03:17:00.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-06T03:17:00.973 DEBUG:teuthology.orchestra.run.smithi192:> uname -r 2023-10-06T03:17:00.979 INFO:teuthology.orchestra.run.smithi110.stdout:5.4.0-124-generic 2023-10-06T03:17:00.979 INFO:teuthology.task.kernel:Running kernel on smithi110: 5.4.0-124-generic 2023-10-06T03:17:00.979 DEBUG:teuthology.orchestra.run.smithi110:> sudo apt-get clean 2023-10-06T03:17:01.021 INFO:teuthology.orchestra.run.smithi192.stdout:5.4.0-124-generic 2023-10-06T03:17:01.022 INFO:teuthology.task.kernel:Running kernel on smithi192: 5.4.0-124-generic 2023-10-06T03:17:01.022 DEBUG:teuthology.orchestra.run.smithi192:> sudo apt-get clean 2023-10-06T03:17:01.133 DEBUG:teuthology.orchestra.run.smithi192:> sudo apt-get update 2023-10-06T03:17:01.136 DEBUG:teuthology.orchestra.run.smithi110:> sudo apt-get update 2023-10-06T03:17:01.390 INFO:teuthology.orchestra.run.smithi192.stdout:Get:1 http://security.ubuntu.com/ubuntu focal-security InRelease [114 kB] 2023-10-06T03:17:01.413 INFO:teuthology.orchestra.run.smithi110.stdout:Get:1 http://security.ubuntu.com/ubuntu focal-security InRelease [114 kB] 2023-10-06T03:17:01.479 INFO:teuthology.orchestra.run.smithi192.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal InRelease 2023-10-06T03:17:01.494 INFO:teuthology.orchestra.run.smithi110.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal InRelease 2023-10-06T03:17:01.561 INFO:teuthology.orchestra.run.smithi192.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease [114 kB] 2023-10-06T03:17:01.574 INFO:teuthology.orchestra.run.smithi110.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease [114 kB] 2023-10-06T03:17:01.665 INFO:teuthology.orchestra.run.smithi192.stdout:Get:4 http://security.ubuntu.com/ubuntu focal-security/main amd64 Packages [2,500 kB] 2023-10-06T03:17:01.730 INFO:teuthology.orchestra.run.smithi110.stdout:Get:4 http://security.ubuntu.com/ubuntu focal-security/main amd64 Packages [2,500 kB] 2023-10-06T03:17:01.800 INFO:teuthology.orchestra.run.smithi192.stdout:Get:5 http://security.ubuntu.com/ubuntu focal-security/main i386 Packages [660 kB] 2023-10-06T03:17:01.808 INFO:teuthology.orchestra.run.smithi192.stdout:Get:6 http://security.ubuntu.com/ubuntu focal-security/main Translation-en [390 kB] 2023-10-06T03:17:01.812 INFO:teuthology.orchestra.run.smithi192.stdout:Get:7 http://security.ubuntu.com/ubuntu focal-security/main amd64 c-n-f Metadata [13.2 kB] 2023-10-06T03:17:01.813 INFO:teuthology.orchestra.run.smithi192.stdout:Get:8 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 Packages [2,267 kB] 2023-10-06T03:17:01.829 INFO:teuthology.orchestra.run.smithi192.stdout:Get:9 http://security.ubuntu.com/ubuntu focal-security/restricted i386 Packages [33.3 kB] 2023-10-06T03:17:01.829 INFO:teuthology.orchestra.run.smithi192.stdout:Get:10 http://security.ubuntu.com/ubuntu focal-security/restricted Translation-en [317 kB] 2023-10-06T03:17:01.831 INFO:teuthology.orchestra.run.smithi192.stdout:Get:11 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 c-n-f Metadata [572 B] 2023-10-06T03:17:01.832 INFO:teuthology.orchestra.run.smithi192.stdout:Get:12 http://security.ubuntu.com/ubuntu focal-security/universe amd64 Packages [890 kB] 2023-10-06T03:17:01.837 INFO:teuthology.orchestra.run.smithi192.stdout:Get:13 http://security.ubuntu.com/ubuntu focal-security/universe i386 Packages [616 kB] 2023-10-06T03:17:01.841 INFO:teuthology.orchestra.run.smithi192.stdout:Get:14 http://security.ubuntu.com/ubuntu focal-security/universe Translation-en [186 kB] 2023-10-06T03:17:01.843 INFO:teuthology.orchestra.run.smithi192.stdout:Get:15 http://security.ubuntu.com/ubuntu focal-security/universe amd64 c-n-f Metadata [19.1 kB] 2023-10-06T03:17:01.843 INFO:teuthology.orchestra.run.smithi192.stdout:Get:16 http://security.ubuntu.com/ubuntu focal-security/multiverse i386 Packages [7,196 B] 2023-10-06T03:17:01.843 INFO:teuthology.orchestra.run.smithi192.stdout:Get:17 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 Packages [23.6 kB] 2023-10-06T03:17:01.859 INFO:teuthology.orchestra.run.smithi192.stdout:Get:18 http://security.ubuntu.com/ubuntu focal-security/multiverse Translation-en [5,504 B] 2023-10-06T03:17:01.859 INFO:teuthology.orchestra.run.smithi192.stdout:Get:19 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 c-n-f Metadata [548 B] 2023-10-06T03:17:01.905 INFO:teuthology.orchestra.run.smithi192.stdout:Get:20 http://archive.ubuntu.com/ubuntu focal-backports InRelease [108 kB] 2023-10-06T03:17:01.917 INFO:teuthology.orchestra.run.smithi110.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal-backports InRelease [108 kB] 2023-10-06T03:17:01.922 INFO:teuthology.orchestra.run.smithi110.stdout:Get:6 http://security.ubuntu.com/ubuntu focal-security/main i386 Packages [660 kB] 2023-10-06T03:17:01.934 INFO:teuthology.orchestra.run.smithi110.stdout:Get:7 http://security.ubuntu.com/ubuntu focal-security/main Translation-en [390 kB] 2023-10-06T03:17:01.939 INFO:teuthology.orchestra.run.smithi110.stdout:Get:8 http://security.ubuntu.com/ubuntu focal-security/main amd64 c-n-f Metadata [13.2 kB] 2023-10-06T03:17:01.939 INFO:teuthology.orchestra.run.smithi110.stdout:Get:9 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 Packages [2,267 kB] 2023-10-06T03:17:01.959 INFO:teuthology.orchestra.run.smithi110.stdout:Get:10 http://security.ubuntu.com/ubuntu focal-security/restricted i386 Packages [33.3 kB] 2023-10-06T03:17:01.959 INFO:teuthology.orchestra.run.smithi110.stdout:Get:11 http://security.ubuntu.com/ubuntu focal-security/restricted Translation-en [317 kB] 2023-10-06T03:17:01.961 INFO:teuthology.orchestra.run.smithi110.stdout:Get:12 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 c-n-f Metadata [572 B] 2023-10-06T03:17:01.961 INFO:teuthology.orchestra.run.smithi110.stdout:Get:13 http://security.ubuntu.com/ubuntu focal-security/universe amd64 Packages [890 kB] 2023-10-06T03:17:01.969 INFO:teuthology.orchestra.run.smithi110.stdout:Get:14 http://security.ubuntu.com/ubuntu focal-security/universe i386 Packages [616 kB] 2023-10-06T03:17:01.976 INFO:teuthology.orchestra.run.smithi110.stdout:Get:15 http://security.ubuntu.com/ubuntu focal-security/universe Translation-en [186 kB] 2023-10-06T03:17:01.976 INFO:teuthology.orchestra.run.smithi110.stdout:Get:16 http://security.ubuntu.com/ubuntu focal-security/universe amd64 c-n-f Metadata [19.1 kB] 2023-10-06T03:17:01.983 INFO:teuthology.orchestra.run.smithi110.stdout:Get:17 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 Packages [23.6 kB] 2023-10-06T03:17:01.983 INFO:teuthology.orchestra.run.smithi110.stdout:Get:18 http://security.ubuntu.com/ubuntu focal-security/multiverse i386 Packages [7,196 B] 2023-10-06T03:17:01.983 INFO:teuthology.orchestra.run.smithi110.stdout:Get:19 http://security.ubuntu.com/ubuntu focal-security/multiverse Translation-en [5,504 B] 2023-10-06T03:17:01.984 INFO:teuthology.orchestra.run.smithi110.stdout:Get:20 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 c-n-f Metadata [548 B] 2023-10-06T03:17:02.047 INFO:teuthology.orchestra.run.smithi192.stdout:Get:21 http://archive.ubuntu.com/ubuntu focal-updates/main i386 Packages [893 kB] 2023-10-06T03:17:02.064 INFO:teuthology.orchestra.run.smithi110.stdout:Get:21 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages [2,888 kB] 2023-10-06T03:17:02.296 INFO:teuthology.orchestra.run.smithi192.stdout:Get:22 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages [2,888 kB] 2023-10-06T03:17:02.460 INFO:teuthology.orchestra.run.smithi192.stdout:Get:23 http://archive.ubuntu.com/ubuntu focal-updates/main Translation-en [473 kB] 2023-10-06T03:17:02.473 INFO:teuthology.orchestra.run.smithi192.stdout:Get:24 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 c-n-f Metadata [17.1 kB] 2023-10-06T03:17:02.473 INFO:teuthology.orchestra.run.smithi192.stdout:Get:25 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 Packages [2,383 kB] 2023-10-06T03:17:02.529 INFO:teuthology.orchestra.run.smithi110.stdout:Get:22 http://archive.ubuntu.com/ubuntu focal-updates/main i386 Packages [893 kB] 2023-10-06T03:17:02.534 INFO:teuthology.orchestra.run.smithi192.stdout:Get:26 http://archive.ubuntu.com/ubuntu focal-updates/restricted i386 Packages [34.6 kB] 2023-10-06T03:17:02.535 INFO:teuthology.orchestra.run.smithi192.stdout:Get:27 http://archive.ubuntu.com/ubuntu focal-updates/restricted Translation-en [334 kB] 2023-10-06T03:17:02.547 INFO:teuthology.orchestra.run.smithi110.stdout:Get:23 http://archive.ubuntu.com/ubuntu focal-updates/main Translation-en [473 kB] 2023-10-06T03:17:02.554 INFO:teuthology.orchestra.run.smithi192.stdout:Get:28 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 c-n-f Metadata [572 B] 2023-10-06T03:17:02.556 INFO:teuthology.orchestra.run.smithi110.stdout:Get:24 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 c-n-f Metadata [17.1 kB] 2023-10-06T03:17:02.556 INFO:teuthology.orchestra.run.smithi110.stdout:Get:25 http://archive.ubuntu.com/ubuntu focal-updates/restricted i386 Packages [34.6 kB] 2023-10-06T03:17:02.557 INFO:teuthology.orchestra.run.smithi110.stdout:Get:26 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 Packages [2,383 kB] 2023-10-06T03:17:02.631 INFO:teuthology.orchestra.run.smithi110.stdout:Get:27 http://archive.ubuntu.com/ubuntu focal-updates/restricted Translation-en [334 kB] 2023-10-06T03:17:02.638 INFO:teuthology.orchestra.run.smithi110.stdout:Get:28 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 c-n-f Metadata [572 B] 2023-10-06T03:17:02.718 INFO:teuthology.orchestra.run.smithi192.stdout:Get:29 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 Packages [1,121 kB] 2023-10-06T03:17:02.802 INFO:teuthology.orchestra.run.smithi110.stdout:Get:29 http://archive.ubuntu.com/ubuntu focal-updates/universe i386 Packages [749 kB] 2023-10-06T03:17:03.355 INFO:teuthology.orchestra.run.smithi192.stdout:Get:30 http://archive.ubuntu.com/ubuntu focal-updates/universe i386 Packages [749 kB] 2023-10-06T03:17:03.659 INFO:teuthology.orchestra.run.smithi110.stdout:Get:30 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 Packages [1,121 kB] 2023-10-06T03:17:03.660 INFO:teuthology.orchestra.run.smithi110.stdout:Get:31 http://archive.ubuntu.com/ubuntu focal-updates/universe Translation-en [268 kB] 2023-10-06T03:17:03.660 INFO:teuthology.orchestra.run.smithi110.stdout:Get:32 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 c-n-f Metadata [25.6 kB] 2023-10-06T03:17:03.660 INFO:teuthology.orchestra.run.smithi110.stdout:Get:33 http://archive.ubuntu.com/ubuntu focal-updates/multiverse i386 Packages [8,436 B] 2023-10-06T03:17:03.660 INFO:teuthology.orchestra.run.smithi110.stdout:Get:34 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 Packages [25.8 kB] 2023-10-06T03:17:03.660 INFO:teuthology.orchestra.run.smithi110.stdout:Get:35 http://archive.ubuntu.com/ubuntu focal-updates/multiverse Translation-en [7,484 B] 2023-10-06T03:17:03.660 INFO:teuthology.orchestra.run.smithi110.stdout:Get:36 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 c-n-f Metadata [620 B] 2023-10-06T03:17:03.660 INFO:teuthology.orchestra.run.smithi110.stdout:Get:37 http://archive.ubuntu.com/ubuntu focal-backports/main amd64 Packages [45.7 kB] 2023-10-06T03:17:03.660 INFO:teuthology.orchestra.run.smithi110.stdout:Get:38 http://archive.ubuntu.com/ubuntu focal-backports/main i386 Packages [36.1 kB] 2023-10-06T03:17:03.661 INFO:teuthology.orchestra.run.smithi110.stdout:Get:39 http://archive.ubuntu.com/ubuntu focal-backports/main amd64 c-n-f Metadata [1,420 B] 2023-10-06T03:17:03.661 INFO:teuthology.orchestra.run.smithi192.stdout:Get:31 http://archive.ubuntu.com/ubuntu focal-updates/universe Translation-en [268 kB] 2023-10-06T03:17:03.661 INFO:teuthology.orchestra.run.smithi192.stdout:Get:32 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 c-n-f Metadata [25.6 kB] 2023-10-06T03:17:03.661 INFO:teuthology.orchestra.run.smithi192.stdout:Get:33 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 Packages [25.8 kB] 2023-10-06T03:17:03.661 INFO:teuthology.orchestra.run.smithi192.stdout:Get:34 http://archive.ubuntu.com/ubuntu focal-updates/multiverse i386 Packages [8,436 B] 2023-10-06T03:17:03.661 INFO:teuthology.orchestra.run.smithi192.stdout:Get:35 http://archive.ubuntu.com/ubuntu focal-updates/multiverse Translation-en [7,484 B] 2023-10-06T03:17:03.661 INFO:teuthology.orchestra.run.smithi192.stdout:Get:36 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 c-n-f Metadata [620 B] 2023-10-06T03:17:03.661 INFO:teuthology.orchestra.run.smithi192.stdout:Get:37 http://archive.ubuntu.com/ubuntu focal-backports/main i386 Packages [36.1 kB] 2023-10-06T03:17:03.661 INFO:teuthology.orchestra.run.smithi192.stdout:Get:38 http://archive.ubuntu.com/ubuntu focal-backports/main amd64 Packages [45.7 kB] 2023-10-06T03:17:03.661 INFO:teuthology.orchestra.run.smithi192.stdout:Get:39 http://archive.ubuntu.com/ubuntu focal-backports/main amd64 c-n-f Metadata [1,420 B] 2023-10-06T03:17:03.662 INFO:teuthology.orchestra.run.smithi192.stdout:Get:40 http://archive.ubuntu.com/ubuntu focal-backports/universe i386 Packages [13.8 kB] 2023-10-06T03:17:03.693 INFO:teuthology.orchestra.run.smithi110.stdout:Get:40 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 Packages [25.0 kB] 2023-10-06T03:17:03.765 INFO:teuthology.orchestra.run.smithi192.stdout:Get:41 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 Packages [25.0 kB] 2023-10-06T03:17:03.817 INFO:teuthology.orchestra.run.smithi192.stdout:Get:42 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 c-n-f Metadata [880 B] 2023-10-06T03:17:03.877 INFO:teuthology.orchestra.run.smithi110.stdout:Get:41 http://archive.ubuntu.com/ubuntu focal-backports/universe i386 Packages [13.8 kB] 2023-10-06T03:17:03.899 INFO:teuthology.orchestra.run.smithi110.stdout:Get:42 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 c-n-f Metadata [880 B] 2023-10-06T03:17:05.776 INFO:teuthology.orchestra.run.smithi192.stdout:Fetched 17.6 MB in 3s (6,610 kB/s) 2023-10-06T03:17:05.886 INFO:teuthology.orchestra.run.smithi110.stdout:Fetched 17.6 MB in 3s (6,527 kB/s) 2023-10-06T03:17:07.132 INFO:teuthology.orchestra.run.smithi192.stdout:Reading package lists... 2023-10-06T03:17:07.156 DEBUG:teuthology.orchestra.run.smithi192:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2023-10-06T03:17:07.228 INFO:teuthology.orchestra.run.smithi192.stdout:Reading package lists... 2023-10-06T03:17:07.370 INFO:teuthology.orchestra.run.smithi110.stdout:Reading package lists... 2023-10-06T03:17:07.398 DEBUG:teuthology.orchestra.run.smithi110:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2023-10-06T03:17:07.428 INFO:teuthology.orchestra.run.smithi192.stdout:Building dependency tree... 2023-10-06T03:17:07.429 INFO:teuthology.orchestra.run.smithi192.stdout:Reading state information... 2023-10-06T03:17:07.470 INFO:teuthology.orchestra.run.smithi110.stdout:Reading package lists... 2023-10-06T03:17:07.609 INFO:teuthology.orchestra.run.smithi192.stdout:The following additional packages will be installed: 2023-10-06T03:17:07.610 INFO:teuthology.orchestra.run.smithi192.stdout: linux-generic linux-headers-5.4.0-164 linux-headers-5.4.0-164-generic 2023-10-06T03:17:07.610 INFO:teuthology.orchestra.run.smithi192.stdout: linux-headers-generic linux-image-5.4.0-164-generic 2023-10-06T03:17:07.610 INFO:teuthology.orchestra.run.smithi192.stdout: linux-modules-5.4.0-164-generic linux-modules-extra-5.4.0-164-generic 2023-10-06T03:17:07.611 INFO:teuthology.orchestra.run.smithi192.stdout:Suggested packages: 2023-10-06T03:17:07.611 INFO:teuthology.orchestra.run.smithi192.stdout: fdutils linux-doc | linux-source-5.4.0 linux-tools 2023-10-06T03:17:07.611 INFO:teuthology.orchestra.run.smithi192.stdout:Recommended packages: 2023-10-06T03:17:07.611 INFO:teuthology.orchestra.run.smithi192.stdout: thermald 2023-10-06T03:17:07.672 INFO:teuthology.orchestra.run.smithi110.stdout:Building dependency tree... 2023-10-06T03:17:07.672 INFO:teuthology.orchestra.run.smithi110.stdout:Reading state information... 2023-10-06T03:17:07.674 INFO:teuthology.orchestra.run.smithi192.stdout:The following NEW packages will be installed: 2023-10-06T03:17:07.674 INFO:teuthology.orchestra.run.smithi192.stdout: linux-headers-5.4.0-164 linux-headers-5.4.0-164-generic 2023-10-06T03:17:07.674 INFO:teuthology.orchestra.run.smithi192.stdout: linux-image-5.4.0-164-generic linux-modules-5.4.0-164-generic 2023-10-06T03:17:07.675 INFO:teuthology.orchestra.run.smithi192.stdout: linux-modules-extra-5.4.0-164-generic 2023-10-06T03:17:07.676 INFO:teuthology.orchestra.run.smithi192.stdout:The following packages will be upgraded: 2023-10-06T03:17:07.676 INFO:teuthology.orchestra.run.smithi192.stdout: linux-generic linux-headers-generic linux-image-generic 2023-10-06T03:17:07.731 INFO:teuthology.orchestra.run.smithi192.stdout:3 upgraded, 5 newly installed, 0 to remove and 271 not upgraded. 2023-10-06T03:17:07.731 INFO:teuthology.orchestra.run.smithi192.stdout:Need to get 77.0 MB of archives. 2023-10-06T03:17:07.731 INFO:teuthology.orchestra.run.smithi192.stdout:After this operation, 380 MB of additional disk space will be used. 2023-10-06T03:17:07.732 INFO:teuthology.orchestra.run.smithi192.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-5.4.0-164-generic amd64 5.4.0-164.181 [14.9 MB] 2023-10-06T03:17:07.870 INFO:teuthology.orchestra.run.smithi110.stdout:The following additional packages will be installed: 2023-10-06T03:17:07.870 INFO:teuthology.orchestra.run.smithi110.stdout: linux-generic linux-headers-5.4.0-164 linux-headers-5.4.0-164-generic 2023-10-06T03:17:07.870 INFO:teuthology.orchestra.run.smithi110.stdout: linux-headers-generic linux-image-5.4.0-164-generic 2023-10-06T03:17:07.870 INFO:teuthology.orchestra.run.smithi110.stdout: linux-modules-5.4.0-164-generic linux-modules-extra-5.4.0-164-generic 2023-10-06T03:17:07.871 INFO:teuthology.orchestra.run.smithi110.stdout:Suggested packages: 2023-10-06T03:17:07.871 INFO:teuthology.orchestra.run.smithi110.stdout: fdutils linux-doc | linux-source-5.4.0 linux-tools 2023-10-06T03:17:07.871 INFO:teuthology.orchestra.run.smithi110.stdout:Recommended packages: 2023-10-06T03:17:07.871 INFO:teuthology.orchestra.run.smithi110.stdout: thermald 2023-10-06T03:17:07.930 INFO:teuthology.orchestra.run.smithi110.stdout:The following NEW packages will be installed: 2023-10-06T03:17:07.930 INFO:teuthology.orchestra.run.smithi110.stdout: linux-headers-5.4.0-164 linux-headers-5.4.0-164-generic 2023-10-06T03:17:07.930 INFO:teuthology.orchestra.run.smithi110.stdout: linux-image-5.4.0-164-generic linux-modules-5.4.0-164-generic 2023-10-06T03:17:07.931 INFO:teuthology.orchestra.run.smithi110.stdout: linux-modules-extra-5.4.0-164-generic 2023-10-06T03:17:07.932 INFO:teuthology.orchestra.run.smithi110.stdout:The following packages will be upgraded: 2023-10-06T03:17:07.932 INFO:teuthology.orchestra.run.smithi110.stdout: linux-generic linux-headers-generic linux-image-generic 2023-10-06T03:17:07.989 INFO:teuthology.orchestra.run.smithi110.stdout:3 upgraded, 5 newly installed, 0 to remove and 271 not upgraded. 2023-10-06T03:17:07.990 INFO:teuthology.orchestra.run.smithi110.stdout:Need to get 77.0 MB of archives. 2023-10-06T03:17:07.990 INFO:teuthology.orchestra.run.smithi110.stdout:After this operation, 380 MB of additional disk space will be used. 2023-10-06T03:17:07.990 INFO:teuthology.orchestra.run.smithi110.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-5.4.0-164-generic amd64 5.4.0-164.181 [14.9 MB] 2023-10-06T03:17:08.062 INFO:teuthology.orchestra.run.smithi192.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-5.4.0-164-generic amd64 5.4.0-164.181 [10.5 MB] 2023-10-06T03:17:08.157 INFO:teuthology.orchestra.run.smithi192.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-extra-5.4.0-164-generic amd64 5.4.0-164.181 [39.2 MB] 2023-10-06T03:17:08.336 INFO:teuthology.orchestra.run.smithi110.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-5.4.0-164-generic amd64 5.4.0-164.181 [10.5 MB] 2023-10-06T03:17:08.439 INFO:teuthology.orchestra.run.smithi110.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-extra-5.4.0-164-generic amd64 5.4.0-164.181 [39.2 MB] 2023-10-06T03:17:08.622 INFO:teuthology.orchestra.run.smithi192.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-generic amd64 5.4.0.164.161 [1,896 B] 2023-10-06T03:17:08.622 INFO:teuthology.orchestra.run.smithi192.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-generic amd64 5.4.0.164.161 [2,592 B] 2023-10-06T03:17:08.622 INFO:teuthology.orchestra.run.smithi192.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-headers-5.4.0-164 all 5.4.0-164.181 [11.0 MB] 2023-10-06T03:17:08.716 INFO:teuthology.orchestra.run.smithi192.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-headers-5.4.0-164-generic amd64 5.4.0-164.181 [1,360 kB] 2023-10-06T03:17:08.726 INFO:teuthology.orchestra.run.smithi192.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-headers-generic amd64 5.4.0.164.161 [2,468 B] 2023-10-06T03:17:09.005 INFO:teuthology.orchestra.run.smithi110.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-generic amd64 5.4.0.164.161 [1,896 B] 2023-10-06T03:17:09.006 INFO:teuthology.orchestra.run.smithi110.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-generic amd64 5.4.0.164.161 [2,592 B] 2023-10-06T03:17:09.006 INFO:teuthology.orchestra.run.smithi110.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-headers-5.4.0-164 all 5.4.0-164.181 [11.0 MB] 2023-10-06T03:17:09.327 INFO:teuthology.orchestra.run.smithi110.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-headers-5.4.0-164-generic amd64 5.4.0-164.181 [1,360 kB] 2023-10-06T03:17:09.375 INFO:teuthology.orchestra.run.smithi110.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-headers-generic amd64 5.4.0.164.161 [2,468 B] 2023-10-06T03:17:09.427 INFO:teuthology.orchestra.run.smithi192.stdout:Fetched 77.0 MB in 1s (73.8 MB/s) 2023-10-06T03:17:09.593 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package linux-modules-5.4.0-164-generic. 2023-10-06T03:17:10.076 INFO:teuthology.orchestra.run.smithi110.stdout:Fetched 77.0 MB in 1s (53.6 MB/s) 2023-10-06T03:17:10.275 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package linux-modules-5.4.0-164-generic. 2023-10-06T03:17:11.318 INFO:teuthology.orchestra.run.smithi192.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 104373 files and directories currently installed.) 2023-10-06T03:17:11.322 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../0-linux-modules-5.4.0-164-generic_5.4.0-164.181_amd64.deb ... 2023-10-06T03:17:11.431 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking linux-modules-5.4.0-164-generic (5.4.0-164.181) ... 2023-10-06T03:17:12.072 INFO:teuthology.orchestra.run.smithi110.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 104373 files and directories currently installed.) 2023-10-06T03:17:12.077 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../0-linux-modules-5.4.0-164-generic_5.4.0-164.181_amd64.deb ... 2023-10-06T03:17:12.222 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking linux-modules-5.4.0-164-generic (5.4.0-164.181) ... 2023-10-06T03:17:13.464 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package linux-image-5.4.0-164-generic. 2023-10-06T03:17:13.477 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../1-linux-image-5.4.0-164-generic_5.4.0-164.181_amd64.deb ... 2023-10-06T03:17:13.574 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking linux-image-5.4.0-164-generic (5.4.0-164.181) ... 2023-10-06T03:17:14.314 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package linux-image-5.4.0-164-generic. 2023-10-06T03:17:14.327 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../1-linux-image-5.4.0-164-generic_5.4.0-164.181_amd64.deb ... 2023-10-06T03:17:14.419 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking linux-image-5.4.0-164-generic (5.4.0-164.181) ... 2023-10-06T03:17:14.659 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package linux-modules-extra-5.4.0-164-generic. 2023-10-06T03:17:14.674 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../2-linux-modules-extra-5.4.0-164-generic_5.4.0-164.181_amd64.deb ... 2023-10-06T03:17:14.717 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking linux-modules-extra-5.4.0-164-generic (5.4.0-164.181) ... 2023-10-06T03:17:15.525 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package linux-modules-extra-5.4.0-164-generic. 2023-10-06T03:17:15.538 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../2-linux-modules-extra-5.4.0-164-generic_5.4.0-164.181_amd64.deb ... 2023-10-06T03:17:15.583 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking linux-modules-extra-5.4.0-164-generic (5.4.0-164.181) ... 2023-10-06T03:17:19.751 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../3-linux-generic_5.4.0.164.161_amd64.deb ... 2023-10-06T03:17:19.878 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking linux-generic (5.4.0.164.161) over (5.4.0.124.125) ... 2023-10-06T03:17:20.228 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../4-linux-image-generic_5.4.0.164.161_amd64.deb ... 2023-10-06T03:17:20.357 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking linux-image-generic (5.4.0.164.161) over (5.4.0.124.125) ... 2023-10-06T03:17:20.459 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../3-linux-generic_5.4.0.164.161_amd64.deb ... 2023-10-06T03:17:20.565 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking linux-generic (5.4.0.164.161) over (5.4.0.124.125) ... 2023-10-06T03:17:20.701 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package linux-headers-5.4.0-164. 2023-10-06T03:17:20.718 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../5-linux-headers-5.4.0-164_5.4.0-164.181_all.deb ... 2023-10-06T03:17:20.759 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking linux-headers-5.4.0-164 (5.4.0-164.181) ... 2023-10-06T03:17:20.918 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../4-linux-image-generic_5.4.0.164.161_amd64.deb ... 2023-10-06T03:17:21.061 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking linux-image-generic (5.4.0.164.161) over (5.4.0.124.125) ... 2023-10-06T03:17:21.414 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package linux-headers-5.4.0-164. 2023-10-06T03:17:21.430 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../5-linux-headers-5.4.0-164_5.4.0-164.181_all.deb ... 2023-10-06T03:17:21.472 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking linux-headers-5.4.0-164 (5.4.0-164.181) ... 2023-10-06T03:17:26.180 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package linux-headers-5.4.0-164-generic. 2023-10-06T03:17:26.202 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../6-linux-headers-5.4.0-164-generic_5.4.0-164.181_amd64.deb ... 2023-10-06T03:17:26.246 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking linux-headers-5.4.0-164-generic (5.4.0-164.181) ... 2023-10-06T03:17:26.946 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package linux-headers-5.4.0-164-generic. 2023-10-06T03:17:26.969 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../6-linux-headers-5.4.0-164-generic_5.4.0-164.181_amd64.deb ... 2023-10-06T03:17:27.012 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking linux-headers-5.4.0-164-generic (5.4.0-164.181) ... 2023-10-06T03:17:28.933 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../7-linux-headers-generic_5.4.0.164.161_amd64.deb ... 2023-10-06T03:17:29.081 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking linux-headers-generic (5.4.0.164.161) over (5.4.0.124.125) ... 2023-10-06T03:17:29.418 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up linux-headers-5.4.0-164 (5.4.0-164.181) ... 2023-10-06T03:17:29.559 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up linux-headers-5.4.0-164-generic (5.4.0-164.181) ... 2023-10-06T03:17:29.683 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up linux-modules-5.4.0-164-generic (5.4.0-164.181) ... 2023-10-06T03:17:30.268 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../7-linux-headers-generic_5.4.0.164.161_amd64.deb ... 2023-10-06T03:17:30.371 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking linux-headers-generic (5.4.0.164.161) over (5.4.0.124.125) ... 2023-10-06T03:17:30.713 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up linux-headers-5.4.0-164 (5.4.0-164.181) ... 2023-10-06T03:17:30.859 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up linux-headers-5.4.0-164-generic (5.4.0-164.181) ... 2023-10-06T03:17:30.902 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up linux-image-5.4.0-164-generic (5.4.0-164.181) ... 2023-10-06T03:17:30.985 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up linux-modules-5.4.0-164-generic (5.4.0-164.181) ... 2023-10-06T03:17:32.083 INFO:teuthology.orchestra.run.smithi192.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.4.0-164-generic 2023-10-06T03:17:32.084 INFO:teuthology.orchestra.run.smithi192.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.4.0-164-generic 2023-10-06T03:17:32.220 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up linux-image-5.4.0-164-generic (5.4.0-164.181) ... 2023-10-06T03:17:32.229 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up linux-headers-generic (5.4.0.164.161) ... 2023-10-06T03:17:32.372 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up linux-modules-extra-5.4.0-164-generic (5.4.0-164.181) ... 2023-10-06T03:17:33.763 INFO:teuthology.orchestra.run.smithi110.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.4.0-164-generic 2023-10-06T03:17:33.763 INFO:teuthology.orchestra.run.smithi110.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.4.0-164-generic 2023-10-06T03:17:34.172 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up linux-headers-generic (5.4.0.164.161) ... 2023-10-06T03:17:34.307 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up linux-modules-extra-5.4.0-164-generic (5.4.0-164.181) ... 2023-10-06T03:17:34.441 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up linux-image-generic (5.4.0.164.161) ... 2023-10-06T03:17:34.608 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up linux-generic (5.4.0.164.161) ... 2023-10-06T03:17:34.710 INFO:teuthology.orchestra.run.smithi192.stdout:Processing triggers for linux-image-5.4.0-164-generic (5.4.0-164.181) ... 2023-10-06T03:17:34.778 INFO:teuthology.orchestra.run.smithi192.stdout:/etc/kernel/postinst.d/initramfs-tools: 2023-10-06T03:17:34.779 INFO:teuthology.orchestra.run.smithi192.stdout:update-initramfs: Generating /boot/initrd.img-5.4.0-164-generic 2023-10-06T03:17:35.508 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up linux-image-generic (5.4.0.164.161) ... 2023-10-06T03:17:35.634 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up linux-generic (5.4.0.164.161) ... 2023-10-06T03:17:35.761 INFO:teuthology.orchestra.run.smithi110.stdout:Processing triggers for linux-image-5.4.0-164-generic (5.4.0-164.181) ... 2023-10-06T03:17:35.851 INFO:teuthology.orchestra.run.smithi110.stdout:/etc/kernel/postinst.d/initramfs-tools: 2023-10-06T03:17:35.851 INFO:teuthology.orchestra.run.smithi110.stdout:update-initramfs: Generating /boot/initrd.img-5.4.0-164-generic 2023-10-06T03:17:56.845 INFO:teuthology.orchestra.run.smithi192.stdout:/etc/kernel/postinst.d/zz-update-grub: 2023-10-06T03:17:56.846 INFO:teuthology.orchestra.run.smithi192.stdout:Sourcing file `/etc/default/grub' 2023-10-06T03:17:56.859 INFO:teuthology.orchestra.run.smithi192.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2023-10-06T03:17:56.873 INFO:teuthology.orchestra.run.smithi192.stdout:Generating grub configuration file ... 2023-10-06T03:17:57.257 INFO:teuthology.orchestra.run.smithi192.stdout:Found linux image: /boot/vmlinuz-5.4.0-164-generic 2023-10-06T03:17:57.281 INFO:teuthology.orchestra.run.smithi192.stdout:Found initrd image: /boot/initrd.img-5.4.0-164-generic 2023-10-06T03:17:57.680 INFO:teuthology.orchestra.run.smithi192.stdout:Found linux image: /boot/vmlinuz-5.4.0-124-generic 2023-10-06T03:17:57.691 INFO:teuthology.orchestra.run.smithi192.stdout:Found initrd image: /boot/initrd.img-5.4.0-124-generic 2023-10-06T03:17:58.040 INFO:teuthology.orchestra.run.smithi110.stdout:/etc/kernel/postinst.d/zz-update-grub: 2023-10-06T03:17:58.040 INFO:teuthology.orchestra.run.smithi110.stdout:Sourcing file `/etc/default/grub' 2023-10-06T03:17:58.058 INFO:teuthology.orchestra.run.smithi110.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2023-10-06T03:17:58.113 INFO:teuthology.orchestra.run.smithi110.stdout:Generating grub configuration file ... 2023-10-06T03:17:58.485 INFO:teuthology.orchestra.run.smithi110.stdout:Found linux image: /boot/vmlinuz-5.4.0-164-generic 2023-10-06T03:17:58.508 INFO:teuthology.orchestra.run.smithi110.stdout:Found initrd image: /boot/initrd.img-5.4.0-164-generic 2023-10-06T03:17:58.980 INFO:teuthology.orchestra.run.smithi110.stdout:Found linux image: /boot/vmlinuz-5.4.0-124-generic 2023-10-06T03:17:58.991 INFO:teuthology.orchestra.run.smithi110.stdout:Found initrd image: /boot/initrd.img-5.4.0-124-generic 2023-10-06T03:18:00.424 INFO:teuthology.orchestra.run.smithi192.stdout:done 2023-10-06T03:18:00.850 DEBUG:teuthology.orchestra.run.smithi192:> dpkg -s linux-image-generic 2023-10-06T03:18:00.905 INFO:teuthology.orchestra.run.smithi192.stdout:Package: linux-image-generic 2023-10-06T03:18:00.905 INFO:teuthology.orchestra.run.smithi192.stdout:Status: install ok installed 2023-10-06T03:18:00.905 INFO:teuthology.orchestra.run.smithi192.stdout:Priority: optional 2023-10-06T03:18:00.906 INFO:teuthology.orchestra.run.smithi192.stdout:Section: kernel 2023-10-06T03:18:00.906 INFO:teuthology.orchestra.run.smithi192.stdout:Installed-Size: 20 2023-10-06T03:18:00.906 INFO:teuthology.orchestra.run.smithi192.stdout:Maintainer: Ubuntu Kernel Team 2023-10-06T03:18:00.906 INFO:teuthology.orchestra.run.smithi192.stdout:Architecture: amd64 2023-10-06T03:18:00.906 INFO:teuthology.orchestra.run.smithi192.stdout:Source: linux-meta 2023-10-06T03:18:00.906 INFO:teuthology.orchestra.run.smithi192.stdout:Version: 5.4.0.164.161 2023-10-06T03:18:00.906 INFO:teuthology.orchestra.run.smithi192.stdout:Provides: virtualbox-guest-modules (= 6.1.38-dfsg-3~ubuntu1.20.04.1), wireguard-modules (= 1.0.20201112-1~20.04.1), zfs-modules (= 0.8.3-1ubuntu12.15) 2023-10-06T03:18:00.906 INFO:teuthology.orchestra.run.smithi192.stdout:Depends: linux-image-5.4.0-164-generic, linux-modules-extra-5.4.0-164-generic, linux-firmware, intel-microcode, amd64-microcode 2023-10-06T03:18:00.906 INFO:teuthology.orchestra.run.smithi192.stdout:Recommends: thermald 2023-10-06T03:18:00.906 INFO:teuthology.orchestra.run.smithi192.stdout:Description: Generic Linux kernel image 2023-10-06T03:18:00.906 INFO:teuthology.orchestra.run.smithi192.stdout: This package will always depend on the latest generic kernel image 2023-10-06T03:18:00.907 INFO:teuthology.orchestra.run.smithi192.stdout: available. 2023-10-06T03:18:00.907 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.4.0-124-generic Expected: 5.4.0-164-generic 2023-10-06T03:18:00.907 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-10-06T03:18:00.907 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-10-06T03:18:00.907 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2023-10-06T03:18:00.907 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi192.front.sepia.ceph.com, path=None, version=distro) 2023-10-06T03:18:00.907 DEBUG:teuthology.orchestra.run.smithi192:> sudo apt-get clean 2023-10-06T03:18:00.994 DEBUG:teuthology.orchestra.run.smithi192:> sudo apt-get update 2023-10-06T03:18:01.126 INFO:teuthology.orchestra.run.smithi192.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2023-10-06T03:18:01.127 INFO:teuthology.orchestra.run.smithi192.stdout:Hit:2 http://security.ubuntu.com/ubuntu focal-security InRelease 2023-10-06T03:18:01.132 INFO:teuthology.orchestra.run.smithi192.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2023-10-06T03:18:01.149 INFO:teuthology.orchestra.run.smithi192.stdout:Hit:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2023-10-06T03:18:01.740 INFO:teuthology.orchestra.run.smithi110.stdout:done 2023-10-06T03:18:02.205 DEBUG:teuthology.orchestra.run.smithi110:> dpkg -s linux-image-generic 2023-10-06T03:18:02.264 INFO:teuthology.orchestra.run.smithi110.stdout:Package: linux-image-generic 2023-10-06T03:18:02.264 INFO:teuthology.orchestra.run.smithi110.stdout:Status: install ok installed 2023-10-06T03:18:02.264 INFO:teuthology.orchestra.run.smithi110.stdout:Priority: optional 2023-10-06T03:18:02.264 INFO:teuthology.orchestra.run.smithi110.stdout:Section: kernel 2023-10-06T03:18:02.265 INFO:teuthology.orchestra.run.smithi110.stdout:Installed-Size: 20 2023-10-06T03:18:02.265 INFO:teuthology.orchestra.run.smithi110.stdout:Maintainer: Ubuntu Kernel Team 2023-10-06T03:18:02.265 INFO:teuthology.orchestra.run.smithi110.stdout:Architecture: amd64 2023-10-06T03:18:02.265 INFO:teuthology.orchestra.run.smithi110.stdout:Source: linux-meta 2023-10-06T03:18:02.265 INFO:teuthology.orchestra.run.smithi110.stdout:Version: 5.4.0.164.161 2023-10-06T03:18:02.265 INFO:teuthology.orchestra.run.smithi110.stdout:Provides: virtualbox-guest-modules (= 6.1.38-dfsg-3~ubuntu1.20.04.1), wireguard-modules (= 1.0.20201112-1~20.04.1), zfs-modules (= 0.8.3-1ubuntu12.15) 2023-10-06T03:18:02.265 INFO:teuthology.orchestra.run.smithi110.stdout:Depends: linux-image-5.4.0-164-generic, linux-modules-extra-5.4.0-164-generic, linux-firmware, intel-microcode, amd64-microcode 2023-10-06T03:18:02.265 INFO:teuthology.orchestra.run.smithi110.stdout:Recommends: thermald 2023-10-06T03:18:02.265 INFO:teuthology.orchestra.run.smithi110.stdout:Description: Generic Linux kernel image 2023-10-06T03:18:02.265 INFO:teuthology.orchestra.run.smithi110.stdout: This package will always depend on the latest generic kernel image 2023-10-06T03:18:02.265 INFO:teuthology.orchestra.run.smithi110.stdout: available. 2023-10-06T03:18:02.266 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.4.0-124-generic Expected: 5.4.0-164-generic 2023-10-06T03:18:02.266 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-10-06T03:18:02.266 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-10-06T03:18:02.266 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2023-10-06T03:18:02.266 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi110.front.sepia.ceph.com, path=None, version=distro) 2023-10-06T03:18:02.266 DEBUG:teuthology.orchestra.run.smithi110:> sudo apt-get clean 2023-10-06T03:18:02.347 DEBUG:teuthology.orchestra.run.smithi110:> sudo apt-get update 2023-10-06T03:18:02.482 INFO:teuthology.orchestra.run.smithi110.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2023-10-06T03:18:02.488 INFO:teuthology.orchestra.run.smithi110.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2023-10-06T03:18:02.505 INFO:teuthology.orchestra.run.smithi110.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2023-10-06T03:18:02.601 INFO:teuthology.orchestra.run.smithi110.stdout:Hit:4 http://security.ubuntu.com/ubuntu focal-security InRelease 2023-10-06T03:18:02.875 INFO:teuthology.orchestra.run.smithi192.stdout:Reading package lists... 2023-10-06T03:18:02.901 DEBUG:teuthology.orchestra.run.smithi192:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2023-10-06T03:18:02.967 INFO:teuthology.orchestra.run.smithi192.stdout:Reading package lists... 2023-10-06T03:18:03.167 INFO:teuthology.orchestra.run.smithi192.stdout:Building dependency tree... 2023-10-06T03:18:03.168 INFO:teuthology.orchestra.run.smithi192.stdout:Reading state information... 2023-10-06T03:18:03.386 INFO:teuthology.orchestra.run.smithi192.stdout:linux-image-generic is already the newest version (5.4.0.164.161). 2023-10-06T03:18:03.387 INFO:teuthology.orchestra.run.smithi192.stdout:0 upgraded, 0 newly installed, 0 to remove and 271 not upgraded. 2023-10-06T03:18:03.388 DEBUG:teuthology.orchestra.run.smithi192:> dpkg -s linux-image-generic 2023-10-06T03:18:03.411 INFO:teuthology.orchestra.run.smithi192.stdout:Package: linux-image-generic 2023-10-06T03:18:03.411 INFO:teuthology.orchestra.run.smithi192.stdout:Status: install ok installed 2023-10-06T03:18:03.411 INFO:teuthology.orchestra.run.smithi192.stdout:Priority: optional 2023-10-06T03:18:03.411 INFO:teuthology.orchestra.run.smithi192.stdout:Section: kernel 2023-10-06T03:18:03.411 INFO:teuthology.orchestra.run.smithi192.stdout:Installed-Size: 20 2023-10-06T03:18:03.411 INFO:teuthology.orchestra.run.smithi192.stdout:Maintainer: Ubuntu Kernel Team 2023-10-06T03:18:03.411 INFO:teuthology.orchestra.run.smithi192.stdout:Architecture: amd64 2023-10-06T03:18:03.411 INFO:teuthology.orchestra.run.smithi192.stdout:Source: linux-meta 2023-10-06T03:18:03.411 INFO:teuthology.orchestra.run.smithi192.stdout:Version: 5.4.0.164.161 2023-10-06T03:18:03.411 INFO:teuthology.orchestra.run.smithi192.stdout:Provides: virtualbox-guest-modules (= 6.1.38-dfsg-3~ubuntu1.20.04.1), wireguard-modules (= 1.0.20201112-1~20.04.1), zfs-modules (= 0.8.3-1ubuntu12.15) 2023-10-06T03:18:03.412 INFO:teuthology.orchestra.run.smithi192.stdout:Depends: linux-image-5.4.0-164-generic, linux-modules-extra-5.4.0-164-generic, linux-firmware, intel-microcode, amd64-microcode 2023-10-06T03:18:03.412 INFO:teuthology.orchestra.run.smithi192.stdout:Recommends: thermald 2023-10-06T03:18:03.412 INFO:teuthology.orchestra.run.smithi192.stdout:Description: Generic Linux kernel image 2023-10-06T03:18:03.412 INFO:teuthology.orchestra.run.smithi192.stdout: This package will always depend on the latest generic kernel image 2023-10-06T03:18:03.412 INFO:teuthology.orchestra.run.smithi192.stdout: available. 2023-10-06T03:18:03.412 DEBUG:teuthology.orchestra.run.smithi192:> mktemp 2023-10-06T03:18:03.417 INFO:teuthology.orchestra.run.smithi192.stdout:/tmp/tmp.FNqdythS2U 2023-10-06T03:18:03.417 DEBUG:teuthology.orchestra.run.smithi192:> sudo cp /boot/grub/grub.cfg /tmp/tmp.FNqdythS2U 2023-10-06T03:18:03.485 DEBUG:teuthology.orchestra.run.smithi192:> sudo chmod 0666 /tmp/tmp.FNqdythS2U 2023-10-06T03:18:03.532 DEBUG:teuthology.orchestra.remote:smithi192:/tmp/tmp.FNqdythS2U is 9KB 2023-10-06T03:18:03.545 DEBUG:teuthology.orchestra.run.smithi192:> rm -fr /tmp/tmp.FNqdythS2U 2023-10-06T03:18:03.550 DEBUG:teuthology.orchestra.run.smithi192:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2023-10-06T03:18:03.604 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2023-10-06T03:18:03.604 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd of=/etc/grub.d/01_ceph_kernel 2023-10-06T03:18:03.605 DEBUG:teuthology.orchestra.run.smithi192:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2023-10-06T03:18:03.632 INFO:teuthology.task.kernel:Distro Kernel Version: 5.4.0-164-generic 2023-10-06T03:18:03.632 DEBUG:teuthology.orchestra.run.smithi192:> sudo update-grub 2023-10-06T03:18:04.215 INFO:teuthology.orchestra.run.smithi110.stdout:Reading package lists... 2023-10-06T03:18:04.239 DEBUG:teuthology.orchestra.run.smithi110:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2023-10-06T03:18:04.309 INFO:teuthology.orchestra.run.smithi110.stdout:Reading package lists... 2023-10-06T03:18:04.487 INFO:teuthology.orchestra.run.smithi110.stdout:Building dependency tree... 2023-10-06T03:18:04.487 INFO:teuthology.orchestra.run.smithi110.stdout:Reading state information... 2023-10-06T03:18:04.692 INFO:teuthology.orchestra.run.smithi110.stdout:linux-image-generic is already the newest version (5.4.0.164.161). 2023-10-06T03:18:04.696 INFO:teuthology.orchestra.run.smithi110.stdout:0 upgraded, 0 newly installed, 0 to remove and 271 not upgraded. 2023-10-06T03:18:04.697 DEBUG:teuthology.orchestra.run.smithi110:> dpkg -s linux-image-generic 2023-10-06T03:18:04.718 INFO:teuthology.orchestra.run.smithi110.stdout:Package: linux-image-generic 2023-10-06T03:18:04.718 INFO:teuthology.orchestra.run.smithi110.stdout:Status: install ok installed 2023-10-06T03:18:04.718 INFO:teuthology.orchestra.run.smithi110.stdout:Priority: optional 2023-10-06T03:18:04.718 INFO:teuthology.orchestra.run.smithi110.stdout:Section: kernel 2023-10-06T03:18:04.718 INFO:teuthology.orchestra.run.smithi110.stdout:Installed-Size: 20 2023-10-06T03:18:04.718 INFO:teuthology.orchestra.run.smithi110.stdout:Maintainer: Ubuntu Kernel Team 2023-10-06T03:18:04.718 INFO:teuthology.orchestra.run.smithi110.stdout:Architecture: amd64 2023-10-06T03:18:04.718 INFO:teuthology.orchestra.run.smithi110.stdout:Source: linux-meta 2023-10-06T03:18:04.719 INFO:teuthology.orchestra.run.smithi110.stdout:Version: 5.4.0.164.161 2023-10-06T03:18:04.719 INFO:teuthology.orchestra.run.smithi110.stdout:Provides: virtualbox-guest-modules (= 6.1.38-dfsg-3~ubuntu1.20.04.1), wireguard-modules (= 1.0.20201112-1~20.04.1), zfs-modules (= 0.8.3-1ubuntu12.15) 2023-10-06T03:18:04.719 INFO:teuthology.orchestra.run.smithi110.stdout:Depends: linux-image-5.4.0-164-generic, linux-modules-extra-5.4.0-164-generic, linux-firmware, intel-microcode, amd64-microcode 2023-10-06T03:18:04.719 INFO:teuthology.orchestra.run.smithi110.stdout:Recommends: thermald 2023-10-06T03:18:04.719 INFO:teuthology.orchestra.run.smithi110.stdout:Description: Generic Linux kernel image 2023-10-06T03:18:04.719 INFO:teuthology.orchestra.run.smithi110.stdout: This package will always depend on the latest generic kernel image 2023-10-06T03:18:04.719 INFO:teuthology.orchestra.run.smithi110.stdout: available. 2023-10-06T03:18:04.719 DEBUG:teuthology.orchestra.run.smithi110:> mktemp 2023-10-06T03:18:04.724 INFO:teuthology.orchestra.run.smithi110.stdout:/tmp/tmp.O21XlV3Pge 2023-10-06T03:18:04.724 DEBUG:teuthology.orchestra.run.smithi110:> sudo cp /boot/grub/grub.cfg /tmp/tmp.O21XlV3Pge 2023-10-06T03:18:04.782 DEBUG:teuthology.orchestra.run.smithi110:> sudo chmod 0666 /tmp/tmp.O21XlV3Pge 2023-10-06T03:18:04.819 DEBUG:teuthology.orchestra.remote:smithi110:/tmp/tmp.O21XlV3Pge is 9KB 2023-10-06T03:18:04.833 DEBUG:teuthology.orchestra.run.smithi110:> rm -fr /tmp/tmp.O21XlV3Pge 2023-10-06T03:18:04.838 DEBUG:teuthology.orchestra.run.smithi110:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2023-10-06T03:18:04.897 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2023-10-06T03:18:04.897 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd of=/etc/grub.d/01_ceph_kernel 2023-10-06T03:18:04.897 DEBUG:teuthology.orchestra.run.smithi110:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2023-10-06T03:18:04.933 INFO:teuthology.task.kernel:Distro Kernel Version: 5.4.0-164-generic 2023-10-06T03:18:04.933 DEBUG:teuthology.orchestra.run.smithi110:> sudo update-grub 2023-10-06T03:18:04.976 INFO:teuthology.orchestra.run.smithi192.stderr:Sourcing file `/etc/default/grub' 2023-10-06T03:18:04.982 INFO:teuthology.orchestra.run.smithi192.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2023-10-06T03:18:04.990 INFO:teuthology.orchestra.run.smithi192.stderr:Generating grub configuration file ... 2023-10-06T03:18:05.247 INFO:teuthology.orchestra.run.smithi192.stderr:Found linux image: /boot/vmlinuz-5.4.0-164-generic 2023-10-06T03:18:05.259 INFO:teuthology.orchestra.run.smithi192.stderr:Found initrd image: /boot/initrd.img-5.4.0-164-generic 2023-10-06T03:18:05.641 INFO:teuthology.orchestra.run.smithi192.stderr:Found linux image: /boot/vmlinuz-5.4.0-124-generic 2023-10-06T03:18:05.653 INFO:teuthology.orchestra.run.smithi192.stderr:Found initrd image: /boot/initrd.img-5.4.0-124-generic 2023-10-06T03:18:06.283 INFO:teuthology.orchestra.run.smithi110.stderr:Sourcing file `/etc/default/grub' 2023-10-06T03:18:06.290 INFO:teuthology.orchestra.run.smithi110.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2023-10-06T03:18:06.312 INFO:teuthology.orchestra.run.smithi110.stderr:Generating grub configuration file ... 2023-10-06T03:18:06.588 INFO:teuthology.orchestra.run.smithi110.stderr:Found linux image: /boot/vmlinuz-5.4.0-164-generic 2023-10-06T03:18:06.600 INFO:teuthology.orchestra.run.smithi110.stderr:Found initrd image: /boot/initrd.img-5.4.0-164-generic 2023-10-06T03:18:07.015 INFO:teuthology.orchestra.run.smithi110.stderr:Found linux image: /boot/vmlinuz-5.4.0-124-generic 2023-10-06T03:18:07.027 INFO:teuthology.orchestra.run.smithi110.stderr:Found initrd image: /boot/initrd.img-5.4.0-124-generic 2023-10-06T03:18:07.316 INFO:teuthology.orchestra.run.smithi192.stderr:done 2023-10-06T03:18:07.318 DEBUG:teuthology.orchestra.run.smithi192:> sudo shutdown -r now 2023-10-06T03:18:08.592 INFO:teuthology.orchestra.run.smithi110.stderr:done 2023-10-06T03:18:08.593 DEBUG:teuthology.orchestra.run.smithi110:> sudo shutdown -r now 2023-10-06T03:18:37.346 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2023-10-06T03:18:37.347 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi192.front.sepia.ceph.com' 2023-10-06T03:18:37.347 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi192.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-06T03:18:38.597 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2023-10-06T03:18:38.597 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi110.front.sepia.ceph.com' 2023-10-06T03:18:38.598 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi110.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-06T03:18:55.870 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.192 2023-10-06T03:18:57.150 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.110 2023-10-06T03:19:04.878 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi192.front.sepia.ceph.com' 2023-10-06T03:19:04.879 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi192.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-06T03:19:06.153 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi110.front.sepia.ceph.com' 2023-10-06T03:19:06.153 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi110.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-06T03:19:07.930 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.192 2023-10-06T03:19:09.210 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.110 2023-10-06T03:19:19.942 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi192.front.sepia.ceph.com' 2023-10-06T03:19:19.943 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi192.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-06T03:19:21.213 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi110.front.sepia.ceph.com' 2023-10-06T03:19:21.213 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi110.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-06T03:19:39.642 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.110 2023-10-06T03:19:51.629 DEBUG:teuthology.orchestra.run.smithi192:> true 2023-10-06T03:19:52.245 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi192.front.sepia.ceph.com' 2023-10-06T03:19:52.245 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.4.0-164-generic"... 2023-10-06T03:19:52.246 DEBUG:teuthology.orchestra.run.smithi192:> uname -r 2023-10-06T03:19:52.252 INFO:teuthology.orchestra.run.smithi192.stdout:5.4.0-164-generic 2023-10-06T03:19:52.252 DEBUG:teuthology.task.kernel:current kernel version is 5.4.0-164-generic vs 5.4.0-164-generic 2023-10-06T03:19:52.252 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-10-06T03:19:52.253 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2023-10-06T03:19:53.253 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2023-10-06T03:19:53.254 DEBUG:teuthology.orchestra.run.smithi192:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-10-06T03:19:53.335 INFO:teuthology.orchestra.run.smithi192.stdout:ttyS1 2023-10-06T03:19:53.363 DEBUG:teuthology.parallel:result is None 2023-10-06T03:19:54.643 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi110.front.sepia.ceph.com' 2023-10-06T03:19:54.644 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi110.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-06T03:19:54.845 DEBUG:teuthology.orchestra.run.smithi110:> true 2023-10-06T03:19:55.450 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi110.front.sepia.ceph.com' 2023-10-06T03:19:55.451 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.4.0-164-generic"... 2023-10-06T03:19:55.451 DEBUG:teuthology.orchestra.run.smithi110:> uname -r 2023-10-06T03:19:55.498 INFO:teuthology.orchestra.run.smithi110.stdout:5.4.0-164-generic 2023-10-06T03:19:55.498 DEBUG:teuthology.task.kernel:current kernel version is 5.4.0-164-generic vs 5.4.0-164-generic 2023-10-06T03:19:55.498 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-10-06T03:19:55.499 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2023-10-06T03:19:56.499 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2023-10-06T03:19:56.499 DEBUG:teuthology.orchestra.run.smithi110:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-10-06T03:19:56.603 INFO:teuthology.orchestra.run.smithi110.stdout:ttyS1 2023-10-06T03:19:56.631 DEBUG:teuthology.parallel:result is None 2023-10-06T03:19:56.632 INFO:teuthology.run_tasks:Running task internal.base... 2023-10-06T03:19:56.639 INFO:teuthology.task.internal:Creating test directory... 2023-10-06T03:19:56.639 DEBUG:teuthology.orchestra.run.smithi110:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-10-06T03:19:56.642 DEBUG:teuthology.orchestra.run.smithi192:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-10-06T03:19:56.649 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2023-10-06T03:19:56.655 INFO:teuthology.run_tasks:Running task internal.archive... 2023-10-06T03:19:56.665 INFO:teuthology.task.internal:Creating archive directory... 2023-10-06T03:19:56.665 DEBUG:teuthology.orchestra.run.smithi110:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-10-06T03:19:56.690 DEBUG:teuthology.orchestra.run.smithi192:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-10-06T03:19:56.710 INFO:teuthology.run_tasks:Running task internal.coredump... 2023-10-06T03:19:56.717 INFO:teuthology.task.internal:Enabling coredump saving... 2023-10-06T03:19:56.717 DEBUG:teuthology.orchestra.run.smithi110:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-10-06T03:19:56.719 DEBUG:teuthology.orchestra.run.smithi192:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-10-06T03:19:56.737 INFO:teuthology.orchestra.run.smithi192.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-10-06T03:19:56.737 INFO:teuthology.orchestra.run.smithi110.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-10-06T03:19:56.746 INFO:teuthology.orchestra.run.smithi192.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-10-06T03:19:56.747 INFO:teuthology.orchestra.run.smithi110.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-10-06T03:19:56.749 INFO:teuthology.run_tasks:Running task internal.sudo... 2023-10-06T03:19:56.756 INFO:teuthology.task.internal:Configuring sudo... 2023-10-06T03:19:56.756 DEBUG:teuthology.orchestra.run.smithi110:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-10-06T03:19:56.798 DEBUG:teuthology.orchestra.run.smithi192:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-10-06T03:19:56.813 INFO:teuthology.run_tasks:Running task internal.syslog... 2023-10-06T03:19:56.822 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2023-10-06T03:19:56.822 DEBUG:teuthology.orchestra.run.smithi110:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-10-06T03:19:56.825 DEBUG:teuthology.orchestra.run.smithi192:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-10-06T03:19:56.862 DEBUG:teuthology.orchestra.run.smithi110:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-10-06T03:19:56.870 DEBUG:teuthology.orchestra.run.smithi110:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-10-06T03:19:56.919 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2023-10-06T03:19:56.919 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-10-06T03:19:56.942 DEBUG:teuthology.orchestra.run.smithi192:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-10-06T03:19:56.949 DEBUG:teuthology.orchestra.run.smithi192:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-10-06T03:19:56.957 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2023-10-06T03:19:56.957 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-10-06T03:19:57.022 DEBUG:teuthology.orchestra.run.smithi110:> sudo service rsyslog restart 2023-10-06T03:19:57.024 DEBUG:teuthology.orchestra.run.smithi192:> sudo service rsyslog restart 2023-10-06T03:19:57.073 INFO:teuthology.run_tasks:Running task internal.timer... 2023-10-06T03:19:57.080 INFO:teuthology.task.internal:Starting timer... 2023-10-06T03:19:57.080 INFO:teuthology.run_tasks:Running task pcp... 2023-10-06T03:19:57.089 INFO:teuthology.run_tasks:Running task selinux... 2023-10-06T03:19:57.097 DEBUG:teuthology.task.selinux:Excluding smithi110: OS 'ubuntu' does not support SELinux 2023-10-06T03:19:57.097 DEBUG:teuthology.task.selinux:Excluding smithi192: OS 'ubuntu' does not support SELinux 2023-10-06T03:19:57.098 DEBUG:teuthology.task.selinux:Getting current SELinux state 2023-10-06T03:19:57.098 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2023-10-06T03:19:57.098 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2023-10-06T03:19:57.098 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2023-10-06T03:19:57.107 INFO:teuthology.repo_utils:/home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main was just updated or references a specific commit; assuming it is current 2023-10-06T03:19:57.107 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2023-10-06T03:19:57.124 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2023-10-06T03:19:57.125 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi110.front.sepia.ceph.com,smithi192.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2023-10-06T03:25:39.866 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi110.front.sepia.ceph.com'), Remote(name='ubuntu@smithi192.front.sepia.ceph.com')] 2023-10-06T03:25:39.867 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi110.front.sepia.ceph.com' 2023-10-06T03:25:39.868 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi110.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-06T03:25:39.919 DEBUG:teuthology.orchestra.run.smithi110:> true 2023-10-06T03:25:40.040 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi110.front.sepia.ceph.com' 2023-10-06T03:25:40.040 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi192.front.sepia.ceph.com' 2023-10-06T03:25:40.041 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi192.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-06T03:25:40.087 DEBUG:teuthology.orchestra.run.smithi192:> true 2023-10-06T03:25:40.212 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi192.front.sepia.ceph.com' 2023-10-06T03:25:40.212 INFO:teuthology.run_tasks:Running task clock... 2023-10-06T03:25:40.225 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2023-10-06T03:25:40.226 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-10-06T03:25:40.226 DEBUG:teuthology.orchestra.run.smithi110:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-10-06T03:25:40.228 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-10-06T03:25:40.228 DEBUG:teuthology.orchestra.run.smithi192:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-10-06T03:25:40.260 INFO:teuthology.orchestra.run.smithi110.stdout: 6 Oct 03:25:40 ntpd[22814]: ntpd 4.2.8p12@1.3728-o (1): Starting 2023-10-06T03:25:40.260 INFO:teuthology.orchestra.run.smithi110.stdout: 6 Oct 03:25:40 ntpd[22814]: Command line: ntpd -gq 2023-10-06T03:25:40.261 INFO:teuthology.orchestra.run.smithi110.stdout: 6 Oct 03:25:40 ntpd[22814]: proto: precision = 0.087 usec (-23) 2023-10-06T03:25:40.261 INFO:teuthology.orchestra.run.smithi110.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2023-10-06T03:25:40.262 INFO:teuthology.orchestra.run.smithi110.stdout: 6 Oct 03:25:40 ntpd[22814]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2023-10-06T03:25:40.262 INFO:teuthology.orchestra.run.smithi110.stderr:restrict ::: KOD does nothing without LIMITED. 2023-10-06T03:25:40.263 INFO:teuthology.orchestra.run.smithi110.stdout: 6 Oct 03:25:40 ntpd[22814]: restrict ::: KOD does nothing without LIMITED. 2023-10-06T03:25:40.263 INFO:teuthology.orchestra.run.smithi110.stdout: 6 Oct 03:25:40 ntpd[22814]: Listen and drop on 0 v6wildcard [::]:123 2023-10-06T03:25:40.263 INFO:teuthology.orchestra.run.smithi110.stdout: 6 Oct 03:25:40 ntpd[22814]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2023-10-06T03:25:40.263 INFO:teuthology.orchestra.run.smithi110.stdout: 6 Oct 03:25:40 ntpd[22814]: Listen normally on 2 lo 127.0.0.1:123 2023-10-06T03:25:40.263 INFO:teuthology.orchestra.run.smithi110.stdout: 6 Oct 03:25:40 ntpd[22814]: Listen normally on 3 enp3s0f1 172.21.15.110:123 2023-10-06T03:25:40.263 INFO:teuthology.orchestra.run.smithi110.stdout: 6 Oct 03:25:40 ntpd[22814]: Listen normally on 4 lo [::1]:123 2023-10-06T03:25:40.264 INFO:teuthology.orchestra.run.smithi110.stdout: 6 Oct 03:25:40 ntpd[22814]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe8f:cc35%5]:123 2023-10-06T03:25:40.264 INFO:teuthology.orchestra.run.smithi110.stdout: 6 Oct 03:25:40 ntpd[22814]: Listening on routing socket on fd #22 for interface updates 2023-10-06T03:25:40.264 INFO:teuthology.orchestra.run.smithi192.stdout: 6 Oct 03:25:40 ntpd[22877]: ntpd 4.2.8p12@1.3728-o (1): Starting 2023-10-06T03:25:40.264 INFO:teuthology.orchestra.run.smithi192.stdout: 6 Oct 03:25:40 ntpd[22877]: Command line: ntpd -gq 2023-10-06T03:25:40.265 INFO:teuthology.orchestra.run.smithi192.stdout: 6 Oct 03:25:40 ntpd[22877]: proto: precision = 0.093 usec (-23) 2023-10-06T03:25:40.265 INFO:teuthology.orchestra.run.smithi192.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2023-10-06T03:25:40.265 INFO:teuthology.orchestra.run.smithi192.stdout: 6 Oct 03:25:40 ntpd[22877]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2023-10-06T03:25:40.265 INFO:teuthology.orchestra.run.smithi192.stderr:restrict ::: KOD does nothing without LIMITED. 2023-10-06T03:25:40.266 INFO:teuthology.orchestra.run.smithi192.stdout: 6 Oct 03:25:40 ntpd[22877]: restrict ::: KOD does nothing without LIMITED. 2023-10-06T03:25:40.267 INFO:teuthology.orchestra.run.smithi192.stdout: 6 Oct 03:25:40 ntpd[22877]: Listen and drop on 0 v6wildcard [::]:123 2023-10-06T03:25:40.267 INFO:teuthology.orchestra.run.smithi192.stdout: 6 Oct 03:25:40 ntpd[22877]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2023-10-06T03:25:40.267 INFO:teuthology.orchestra.run.smithi192.stdout: 6 Oct 03:25:40 ntpd[22877]: Listen normally on 2 lo 127.0.0.1:123 2023-10-06T03:25:40.267 INFO:teuthology.orchestra.run.smithi192.stdout: 6 Oct 03:25:40 ntpd[22877]: Listen normally on 3 enp3s0f1 172.21.15.192:123 2023-10-06T03:25:40.267 INFO:teuthology.orchestra.run.smithi192.stdout: 6 Oct 03:25:40 ntpd[22877]: Listen normally on 4 lo [::1]:123 2023-10-06T03:25:40.267 INFO:teuthology.orchestra.run.smithi192.stdout: 6 Oct 03:25:40 ntpd[22877]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe88:8049%5]:123 2023-10-06T03:25:40.267 INFO:teuthology.orchestra.run.smithi192.stdout: 6 Oct 03:25:40 ntpd[22877]: Listening on routing socket on fd #22 for interface updates 2023-10-06T03:25:41.261 INFO:teuthology.orchestra.run.smithi110.stderr: 6 Oct 03:25:41 ntpd[22814]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-06T03:25:41.261 INFO:teuthology.orchestra.run.smithi110.stderr: 6 Oct 03:25:41 ntpd[22814]: can't open /var/log/ntpstats/rawstats.20231006: Permission denied 2023-10-06T03:25:41.262 INFO:teuthology.orchestra.run.smithi110.stderr: 6 Oct 03:25:41 ntpd[22814]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-06T03:25:41.262 INFO:teuthology.orchestra.run.smithi110.stderr: 6 Oct 03:25:41 ntpd[22814]: can't open /var/log/ntpstats/rawstats.20231006: Permission denied 2023-10-06T03:25:41.262 INFO:teuthology.orchestra.run.smithi110.stderr: 6 Oct 03:25:41 ntpd[22814]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-06T03:25:41.262 INFO:teuthology.orchestra.run.smithi110.stderr: 6 Oct 03:25:41 ntpd[22814]: can't open /var/log/ntpstats/peerstats.20231006: Permission denied 2023-10-06T03:25:41.263 INFO:teuthology.orchestra.run.smithi192.stderr: 6 Oct 03:25:41 ntpd[22877]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-06T03:25:41.263 INFO:teuthology.orchestra.run.smithi192.stderr: 6 Oct 03:25:41 ntpd[22877]: can't open /var/log/ntpstats/rawstats.20231006: Permission denied 2023-10-06T03:25:41.263 INFO:teuthology.orchestra.run.smithi192.stderr: 6 Oct 03:25:41 ntpd[22877]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-06T03:25:41.263 INFO:teuthology.orchestra.run.smithi192.stderr: 6 Oct 03:25:41 ntpd[22877]: can't open /var/log/ntpstats/rawstats.20231006: Permission denied 2023-10-06T03:25:41.263 INFO:teuthology.orchestra.run.smithi192.stderr: 6 Oct 03:25:41 ntpd[22877]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-06T03:25:41.263 INFO:teuthology.orchestra.run.smithi192.stderr: 6 Oct 03:25:41 ntpd[22877]: can't open /var/log/ntpstats/rawstats.20231006: Permission denied 2023-10-06T03:25:41.263 INFO:teuthology.orchestra.run.smithi192.stderr: 6 Oct 03:25:41 ntpd[22877]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-06T03:25:41.264 INFO:teuthology.orchestra.run.smithi192.stderr: 6 Oct 03:25:41 ntpd[22877]: can't open /var/log/ntpstats/peerstats.20231006: Permission denied 2023-10-06T03:25:42.261 INFO:teuthology.orchestra.run.smithi110.stderr: 6 Oct 03:25:42 ntpd[22814]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-06T03:25:42.261 INFO:teuthology.orchestra.run.smithi110.stderr: 6 Oct 03:25:42 ntpd[22814]: can't open /var/log/ntpstats/rawstats.20231006: Permission denied 2023-10-06T03:25:42.261 INFO:teuthology.orchestra.run.smithi110.stderr: 6 Oct 03:25:42 ntpd[22814]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-06T03:25:42.261 INFO:teuthology.orchestra.run.smithi110.stderr: 6 Oct 03:25:42 ntpd[22814]: can't open /var/log/ntpstats/rawstats.20231006: Permission denied 2023-10-06T03:25:42.262 INFO:teuthology.orchestra.run.smithi110.stderr: 6 Oct 03:25:42 ntpd[22814]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-06T03:25:42.262 INFO:teuthology.orchestra.run.smithi110.stderr: 6 Oct 03:25:42 ntpd[22814]: can't open /var/log/ntpstats/peerstats.20231006: Permission denied 2023-10-06T03:25:42.263 INFO:teuthology.orchestra.run.smithi192.stderr: 6 Oct 03:25:42 ntpd[22877]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-06T03:25:42.263 INFO:teuthology.orchestra.run.smithi192.stderr: 6 Oct 03:25:42 ntpd[22877]: can't open /var/log/ntpstats/rawstats.20231006: Permission denied 2023-10-06T03:25:42.263 INFO:teuthology.orchestra.run.smithi192.stderr: 6 Oct 03:25:42 ntpd[22877]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-06T03:25:42.263 INFO:teuthology.orchestra.run.smithi192.stderr: 6 Oct 03:25:42 ntpd[22877]: can't open /var/log/ntpstats/rawstats.20231006: Permission denied 2023-10-06T03:25:42.263 INFO:teuthology.orchestra.run.smithi192.stderr: 6 Oct 03:25:42 ntpd[22877]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-06T03:25:42.263 INFO:teuthology.orchestra.run.smithi192.stderr: 6 Oct 03:25:42 ntpd[22877]: can't open /var/log/ntpstats/peerstats.20231006: Permission denied 2023-10-06T03:25:43.261 INFO:teuthology.orchestra.run.smithi110.stderr: 6 Oct 03:25:43 ntpd[22814]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-06T03:25:43.261 INFO:teuthology.orchestra.run.smithi110.stderr: 6 Oct 03:25:43 ntpd[22814]: can't open /var/log/ntpstats/rawstats.20231006: Permission denied 2023-10-06T03:25:43.262 INFO:teuthology.orchestra.run.smithi110.stderr: 6 Oct 03:25:43 ntpd[22814]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-06T03:25:43.262 INFO:teuthology.orchestra.run.smithi110.stderr: 6 Oct 03:25:43 ntpd[22814]: can't open /var/log/ntpstats/rawstats.20231006: Permission denied 2023-10-06T03:25:43.262 INFO:teuthology.orchestra.run.smithi110.stderr: 6 Oct 03:25:43 ntpd[22814]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-06T03:25:43.262 INFO:teuthology.orchestra.run.smithi110.stderr: 6 Oct 03:25:43 ntpd[22814]: can't open /var/log/ntpstats/rawstats.20231006: Permission denied 2023-10-06T03:25:43.262 INFO:teuthology.orchestra.run.smithi110.stderr: 6 Oct 03:25:43 ntpd[22814]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-06T03:25:43.262 INFO:teuthology.orchestra.run.smithi110.stderr: 6 Oct 03:25:43 ntpd[22814]: can't open /var/log/ntpstats/peerstats.20231006: Permission denied 2023-10-06T03:25:43.263 INFO:teuthology.orchestra.run.smithi192.stderr: 6 Oct 03:25:43 ntpd[22877]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-06T03:25:43.263 INFO:teuthology.orchestra.run.smithi192.stderr: 6 Oct 03:25:43 ntpd[22877]: can't open /var/log/ntpstats/rawstats.20231006: Permission denied 2023-10-06T03:25:43.264 INFO:teuthology.orchestra.run.smithi192.stderr: 6 Oct 03:25:43 ntpd[22877]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-06T03:25:43.264 INFO:teuthology.orchestra.run.smithi192.stderr: 6 Oct 03:25:43 ntpd[22877]: can't open /var/log/ntpstats/rawstats.20231006: Permission denied 2023-10-06T03:25:43.264 INFO:teuthology.orchestra.run.smithi192.stderr: 6 Oct 03:25:43 ntpd[22877]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-06T03:25:43.264 INFO:teuthology.orchestra.run.smithi192.stderr: 6 Oct 03:25:43 ntpd[22877]: can't open /var/log/ntpstats/rawstats.20231006: Permission denied 2023-10-06T03:25:43.264 INFO:teuthology.orchestra.run.smithi192.stderr: 6 Oct 03:25:43 ntpd[22877]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-06T03:25:43.264 INFO:teuthology.orchestra.run.smithi192.stderr: 6 Oct 03:25:43 ntpd[22877]: can't open /var/log/ntpstats/peerstats.20231006: Permission denied 2023-10-06T03:25:43.264 INFO:teuthology.orchestra.run.smithi192.stderr: 6 Oct 03:25:43 ntpd[22877]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-06T03:25:43.264 INFO:teuthology.orchestra.run.smithi192.stderr: 6 Oct 03:25:43 ntpd[22877]: can't open /var/log/ntpstats/rawstats.20231006: Permission denied 2023-10-06T03:25:43.264 INFO:teuthology.orchestra.run.smithi192.stderr: 6 Oct 03:25:43 ntpd[22877]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-06T03:25:43.264 INFO:teuthology.orchestra.run.smithi192.stderr: 6 Oct 03:25:43 ntpd[22877]: can't open /var/log/ntpstats/peerstats.20231006: Permission denied 2023-10-06T03:25:44.261 INFO:teuthology.orchestra.run.smithi110.stderr: 6 Oct 03:25:44 ntpd[22814]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-06T03:25:44.261 INFO:teuthology.orchestra.run.smithi110.stderr: 6 Oct 03:25:44 ntpd[22814]: can't open /var/log/ntpstats/rawstats.20231006: Permission denied 2023-10-06T03:25:44.261 INFO:teuthology.orchestra.run.smithi110.stderr: 6 Oct 03:25:44 ntpd[22814]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-06T03:25:44.261 INFO:teuthology.orchestra.run.smithi110.stderr: 6 Oct 03:25:44 ntpd[22814]: can't open /var/log/ntpstats/rawstats.20231006: Permission denied 2023-10-06T03:25:44.262 INFO:teuthology.orchestra.run.smithi110.stderr: 6 Oct 03:25:44 ntpd[22814]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-06T03:25:44.262 INFO:teuthology.orchestra.run.smithi110.stderr: 6 Oct 03:25:44 ntpd[22814]: can't open /var/log/ntpstats/peerstats.20231006: Permission denied 2023-10-06T03:25:44.263 INFO:teuthology.orchestra.run.smithi192.stderr: 6 Oct 03:25:44 ntpd[22877]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-06T03:25:44.263 INFO:teuthology.orchestra.run.smithi192.stderr: 6 Oct 03:25:44 ntpd[22877]: can't open /var/log/ntpstats/rawstats.20231006: Permission denied 2023-10-06T03:25:44.263 INFO:teuthology.orchestra.run.smithi192.stderr: 6 Oct 03:25:44 ntpd[22877]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-06T03:25:44.263 INFO:teuthology.orchestra.run.smithi192.stderr: 6 Oct 03:25:44 ntpd[22877]: can't open /var/log/ntpstats/rawstats.20231006: Permission denied 2023-10-06T03:25:44.263 INFO:teuthology.orchestra.run.smithi192.stderr: 6 Oct 03:25:44 ntpd[22877]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-06T03:25:44.263 INFO:teuthology.orchestra.run.smithi192.stderr: 6 Oct 03:25:44 ntpd[22877]: can't open /var/log/ntpstats/peerstats.20231006: Permission denied 2023-10-06T03:25:45.263 INFO:teuthology.orchestra.run.smithi110.stderr: 6 Oct 03:25:45 ntpd[22814]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-06T03:25:45.263 INFO:teuthology.orchestra.run.smithi110.stderr: 6 Oct 03:25:45 ntpd[22814]: can't open /var/log/ntpstats/rawstats.20231006: Permission denied 2023-10-06T03:25:45.263 INFO:teuthology.orchestra.run.smithi110.stderr: 6 Oct 03:25:45 ntpd[22814]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-06T03:25:45.263 INFO:teuthology.orchestra.run.smithi110.stderr: 6 Oct 03:25:45 ntpd[22814]: can't open /var/log/ntpstats/rawstats.20231006: Permission denied 2023-10-06T03:25:45.263 INFO:teuthology.orchestra.run.smithi110.stderr: 6 Oct 03:25:45 ntpd[22814]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-06T03:25:45.263 INFO:teuthology.orchestra.run.smithi110.stderr: 6 Oct 03:25:45 ntpd[22814]: can't open /var/log/ntpstats/rawstats.20231006: Permission denied 2023-10-06T03:25:45.264 INFO:teuthology.orchestra.run.smithi110.stderr: 6 Oct 03:25:45 ntpd[22814]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-06T03:25:45.264 INFO:teuthology.orchestra.run.smithi110.stderr: 6 Oct 03:25:45 ntpd[22814]: can't open /var/log/ntpstats/peerstats.20231006: Permission denied 2023-10-06T03:25:45.264 INFO:teuthology.orchestra.run.smithi110.stderr: 6 Oct 03:25:45 ntpd[22814]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-06T03:25:45.264 INFO:teuthology.orchestra.run.smithi110.stderr: 6 Oct 03:25:45 ntpd[22814]: can't open /var/log/ntpstats/rawstats.20231006: Permission denied 2023-10-06T03:25:45.264 INFO:teuthology.orchestra.run.smithi110.stderr: 6 Oct 03:25:45 ntpd[22814]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-06T03:25:45.264 INFO:teuthology.orchestra.run.smithi110.stderr: 6 Oct 03:25:45 ntpd[22814]: can't open /var/log/ntpstats/peerstats.20231006: Permission denied 2023-10-06T03:25:45.265 INFO:teuthology.orchestra.run.smithi192.stderr: 6 Oct 03:25:45 ntpd[22877]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-06T03:25:45.266 INFO:teuthology.orchestra.run.smithi192.stderr: 6 Oct 03:25:45 ntpd[22877]: can't open /var/log/ntpstats/rawstats.20231006: Permission denied 2023-10-06T03:25:45.266 INFO:teuthology.orchestra.run.smithi192.stderr: 6 Oct 03:25:45 ntpd[22877]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-06T03:25:45.266 INFO:teuthology.orchestra.run.smithi192.stderr: 6 Oct 03:25:45 ntpd[22877]: can't open /var/log/ntpstats/rawstats.20231006: Permission denied 2023-10-06T03:25:45.266 INFO:teuthology.orchestra.run.smithi192.stderr: 6 Oct 03:25:45 ntpd[22877]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-06T03:25:45.266 INFO:teuthology.orchestra.run.smithi192.stderr: 6 Oct 03:25:45 ntpd[22877]: can't open /var/log/ntpstats/rawstats.20231006: Permission denied 2023-10-06T03:25:45.266 INFO:teuthology.orchestra.run.smithi192.stderr: 6 Oct 03:25:45 ntpd[22877]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-06T03:25:45.267 INFO:teuthology.orchestra.run.smithi192.stderr: 6 Oct 03:25:45 ntpd[22877]: can't open /var/log/ntpstats/peerstats.20231006: Permission denied 2023-10-06T03:25:45.267 INFO:teuthology.orchestra.run.smithi192.stderr: 6 Oct 03:25:45 ntpd[22877]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-06T03:25:45.267 INFO:teuthology.orchestra.run.smithi192.stderr: 6 Oct 03:25:45 ntpd[22877]: can't open /var/log/ntpstats/rawstats.20231006: Permission denied 2023-10-06T03:25:45.267 INFO:teuthology.orchestra.run.smithi192.stderr: 6 Oct 03:25:45 ntpd[22877]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-06T03:25:45.267 INFO:teuthology.orchestra.run.smithi192.stderr: 6 Oct 03:25:45 ntpd[22877]: can't open /var/log/ntpstats/peerstats.20231006: Permission denied 2023-10-06T03:25:46.261 INFO:teuthology.orchestra.run.smithi110.stderr: 6 Oct 03:25:46 ntpd[22814]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-06T03:25:46.261 INFO:teuthology.orchestra.run.smithi110.stderr: 6 Oct 03:25:46 ntpd[22814]: can't open /var/log/ntpstats/rawstats.20231006: Permission denied 2023-10-06T03:25:46.261 INFO:teuthology.orchestra.run.smithi110.stderr: 6 Oct 03:25:46 ntpd[22814]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-06T03:25:46.262 INFO:teuthology.orchestra.run.smithi110.stderr: 6 Oct 03:25:46 ntpd[22814]: can't open /var/log/ntpstats/rawstats.20231006: Permission denied 2023-10-06T03:25:46.262 INFO:teuthology.orchestra.run.smithi110.stderr: 6 Oct 03:25:46 ntpd[22814]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-06T03:25:46.262 INFO:teuthology.orchestra.run.smithi110.stderr: 6 Oct 03:25:46 ntpd[22814]: can't open /var/log/ntpstats/peerstats.20231006: Permission denied 2023-10-06T03:25:46.263 INFO:teuthology.orchestra.run.smithi192.stderr: 6 Oct 03:25:46 ntpd[22877]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-06T03:25:46.263 INFO:teuthology.orchestra.run.smithi192.stderr: 6 Oct 03:25:46 ntpd[22877]: can't open /var/log/ntpstats/rawstats.20231006: Permission denied 2023-10-06T03:25:46.263 INFO:teuthology.orchestra.run.smithi192.stderr: 6 Oct 03:25:46 ntpd[22877]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-06T03:25:46.263 INFO:teuthology.orchestra.run.smithi192.stderr: 6 Oct 03:25:46 ntpd[22877]: can't open /var/log/ntpstats/rawstats.20231006: Permission denied 2023-10-06T03:25:46.263 INFO:teuthology.orchestra.run.smithi192.stderr: 6 Oct 03:25:46 ntpd[22877]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-06T03:25:46.263 INFO:teuthology.orchestra.run.smithi192.stderr: 6 Oct 03:25:46 ntpd[22877]: can't open /var/log/ntpstats/peerstats.20231006: Permission denied 2023-10-06T03:25:47.261 INFO:teuthology.orchestra.run.smithi110.stderr: 6 Oct 03:25:47 ntpd[22814]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-06T03:25:47.262 INFO:teuthology.orchestra.run.smithi110.stderr: 6 Oct 03:25:47 ntpd[22814]: can't open /var/log/ntpstats/rawstats.20231006: Permission denied 2023-10-06T03:25:47.262 INFO:teuthology.orchestra.run.smithi110.stderr: 6 Oct 03:25:47 ntpd[22814]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-06T03:25:47.262 INFO:teuthology.orchestra.run.smithi110.stderr: 6 Oct 03:25:47 ntpd[22814]: can't open /var/log/ntpstats/rawstats.20231006: Permission denied 2023-10-06T03:25:47.262 INFO:teuthology.orchestra.run.smithi110.stderr: 6 Oct 03:25:47 ntpd[22814]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-06T03:25:47.262 INFO:teuthology.orchestra.run.smithi110.stderr: 6 Oct 03:25:47 ntpd[22814]: can't open /var/log/ntpstats/rawstats.20231006: Permission denied 2023-10-06T03:25:47.262 INFO:teuthology.orchestra.run.smithi110.stderr: 6 Oct 03:25:47 ntpd[22814]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-06T03:25:47.262 INFO:teuthology.orchestra.run.smithi110.stderr: 6 Oct 03:25:47 ntpd[22814]: can't open /var/log/ntpstats/peerstats.20231006: Permission denied 2023-10-06T03:25:47.263 INFO:teuthology.orchestra.run.smithi110.stdout: 6 Oct 03:25:47 ntpd[22814]: ntpd: time slew +0.000926 s 2023-10-06T03:25:47.263 INFO:teuthology.orchestra.run.smithi110.stdout:ntpd: time slew +0.000926s 2023-10-06T03:25:47.263 INFO:teuthology.orchestra.run.smithi110.stderr: 6 Oct 03:25:47 ntpd[22814]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2023-10-06T03:25:47.263 INFO:teuthology.orchestra.run.smithi110.stderr: 6 Oct 03:25:47 ntpd[22814]: can't open /var/log/ntpstats/loopstats.20231006: Permission denied 2023-10-06T03:25:47.264 INFO:teuthology.orchestra.run.smithi192.stderr: 6 Oct 03:25:47 ntpd[22877]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-06T03:25:47.264 INFO:teuthology.orchestra.run.smithi192.stderr: 6 Oct 03:25:47 ntpd[22877]: can't open /var/log/ntpstats/rawstats.20231006: Permission denied 2023-10-06T03:25:47.265 INFO:teuthology.orchestra.run.smithi192.stderr: 6 Oct 03:25:47 ntpd[22877]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-06T03:25:47.265 INFO:teuthology.orchestra.run.smithi192.stderr: 6 Oct 03:25:47 ntpd[22877]: can't open /var/log/ntpstats/rawstats.20231006: Permission denied 2023-10-06T03:25:47.265 INFO:teuthology.orchestra.run.smithi192.stderr: 6 Oct 03:25:47 ntpd[22877]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-06T03:25:47.265 INFO:teuthology.orchestra.run.smithi192.stderr: 6 Oct 03:25:47 ntpd[22877]: can't open /var/log/ntpstats/rawstats.20231006: Permission denied 2023-10-06T03:25:47.265 INFO:teuthology.orchestra.run.smithi192.stderr: 6 Oct 03:25:47 ntpd[22877]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-06T03:25:47.265 INFO:teuthology.orchestra.run.smithi192.stderr: 6 Oct 03:25:47 ntpd[22877]: can't open /var/log/ntpstats/peerstats.20231006: Permission denied 2023-10-06T03:25:47.266 INFO:teuthology.orchestra.run.smithi192.stdout: 6 Oct 03:25:47 ntpd[22877]: ntpd: time slew +0.000640 s 2023-10-06T03:25:47.266 INFO:teuthology.orchestra.run.smithi192.stdout:ntpd: time slew +0.000640s 2023-10-06T03:25:47.266 INFO:teuthology.orchestra.run.smithi192.stderr: 6 Oct 03:25:47 ntpd[22877]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2023-10-06T03:25:47.266 INFO:teuthology.orchestra.run.smithi192.stderr: 6 Oct 03:25:47 ntpd[22877]: can't open /var/log/ntpstats/loopstats.20231006: Permission denied 2023-10-06T03:25:47.318 INFO:teuthology.orchestra.run.smithi110.stdout: remote refid st t when poll reach delay offset jitter 2023-10-06T03:25:47.318 INFO:teuthology.orchestra.run.smithi110.stdout:============================================================================== 2023-10-06T03:25:47.318 INFO:teuthology.orchestra.run.smithi110.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2023-10-06T03:25:47.318 INFO:teuthology.orchestra.run.smithi110.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2023-10-06T03:25:47.318 INFO:teuthology.orchestra.run.smithi110.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2023-10-06T03:25:47.318 INFO:teuthology.orchestra.run.smithi110.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2023-10-06T03:25:47.324 INFO:teuthology.orchestra.run.smithi192.stdout: remote refid st t when poll reach delay offset jitter 2023-10-06T03:25:47.324 INFO:teuthology.orchestra.run.smithi192.stdout:============================================================================== 2023-10-06T03:25:47.324 INFO:teuthology.orchestra.run.smithi192.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2023-10-06T03:25:47.325 INFO:teuthology.orchestra.run.smithi192.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2023-10-06T03:25:47.325 INFO:teuthology.orchestra.run.smithi192.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2023-10-06T03:25:47.325 INFO:teuthology.orchestra.run.smithi192.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2023-10-06T03:25:47.326 INFO:teuthology.run_tasks:Running task install... 2023-10-06T03:25:47.334 DEBUG:teuthology.task.install:project ceph 2023-10-06T03:25:47.335 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '736ac5ba72cb47af7d6f6666b1e100f7e20a20e2'}} 2023-10-06T03:25:47.335 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel']}, 'flavor': 'default', 'sha1': '736ac5ba72cb47af7d6f6666b1e100f7e20a20e2'} 2023-10-06T03:25:47.335 INFO:teuthology.task.install:Using flavor: default 2023-10-06T03:25:47.340 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2023-10-06T03:25:47.341 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']} 2023-10-06T03:25:47.341 DEBUG:teuthology.orchestra.run.smithi110:> sudo apt-key list | grep Ceph 2023-10-06T03:25:47.342 DEBUG:teuthology.orchestra.run.smithi192:> sudo apt-key list | grep Ceph 2023-10-06T03:25:47.443 INFO:teuthology.orchestra.run.smithi110.stderr:Warning: apt-key output should not be parsed (stdout is not a terminal) 2023-10-06T03:25:47.476 INFO:teuthology.orchestra.run.smithi192.stderr:Warning: apt-key output should not be parsed (stdout is not a terminal) 2023-10-06T03:25:47.504 INFO:teuthology.orchestra.run.smithi110.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2023-10-06T03:25:47.505 INFO:teuthology.orchestra.run.smithi110.stdout:uid [ unknown] Ceph.com (release key) 2023-10-06T03:25:47.506 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror on remote deb x86_64 2023-10-06T03:25:47.506 INFO:teuthology.task.install.deb:Installing system (non-project) packages: bison, flex, libelf-dev, libssl-dev, network-manager, iproute2, util-linux, dump, indent, libaio-dev, libtool-bin, uuid-dev, xfslibs-dev on remote deb x86_64 2023-10-06T03:25:47.506 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&sha1=736ac5ba72cb47af7d6f6666b1e100f7e20a20e2 2023-10-06T03:25:47.538 INFO:teuthology.orchestra.run.smithi192.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2023-10-06T03:25:47.538 INFO:teuthology.orchestra.run.smithi192.stdout:uid [ unknown] Ceph.com (release key) 2023-10-06T03:25:47.538 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror on remote deb x86_64 2023-10-06T03:25:47.539 INFO:teuthology.task.install.deb:Installing system (non-project) packages: bison, flex, libelf-dev, libssl-dev, network-manager, iproute2, util-linux, dump, indent, libaio-dev, libtool-bin, uuid-dev, xfslibs-dev on remote deb x86_64 2023-10-06T03:25:47.539 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&sha1=736ac5ba72cb47af7d6f6666b1e100f7e20a20e2 2023-10-06T03:25:47.649 INFO:teuthology.task.install.deb:Pulling from https://3.chacra.ceph.com/r/ceph/wip-yuri-testing-2023-10-05-0730-quincy/736ac5ba72cb47af7d6f6666b1e100f7e20a20e2/ubuntu/focal/flavors/default/ 2023-10-06T03:25:47.649 INFO:teuthology.task.install.deb:Package version is 17.2.6-1296-g736ac5ba-1focal 2023-10-06T03:25:47.749 INFO:teuthology.task.install.deb:Pulling from https://3.chacra.ceph.com/r/ceph/wip-yuri-testing-2023-10-05-0730-quincy/736ac5ba72cb47af7d6f6666b1e100f7e20a20e2/ubuntu/focal/flavors/default/ 2023-10-06T03:25:47.750 INFO:teuthology.task.install.deb:Package version is 17.2.6-1296-g736ac5ba-1focal 2023-10-06T03:25:47.764 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2023-10-06T03:25:47.764 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2023-10-06T03:25:47.780 DEBUG:teuthology.orchestra.run.smithi110:> sudo apt-get update 2023-10-06T03:25:47.920 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2023-10-06T03:25:47.921 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2023-10-06T03:25:47.938 DEBUG:teuthology.orchestra.run.smithi192:> sudo apt-get update 2023-10-06T03:25:47.942 INFO:teuthology.orchestra.run.smithi110.stdout:Hit:1 http://security.ubuntu.com/ubuntu focal-security InRelease 2023-10-06T03:25:47.977 INFO:teuthology.orchestra.run.smithi110.stdout:Ign:2 https://3.chacra.ceph.com/r/ceph/wip-yuri-testing-2023-10-05-0730-quincy/736ac5ba72cb47af7d6f6666b1e100f7e20a20e2/ubuntu/focal/flavors/default focal InRelease 2023-10-06T03:25:48.000 INFO:teuthology.orchestra.run.smithi110.stdout:Get:3 https://3.chacra.ceph.com/r/ceph/wip-yuri-testing-2023-10-05-0730-quincy/736ac5ba72cb47af7d6f6666b1e100f7e20a20e2/ubuntu/focal/flavors/default focal Release [7,662 B] 2023-10-06T03:25:48.022 INFO:teuthology.orchestra.run.smithi110.stdout:Ign:4 https://3.chacra.ceph.com/r/ceph/wip-yuri-testing-2023-10-05-0730-quincy/736ac5ba72cb47af7d6f6666b1e100f7e20a20e2/ubuntu/focal/flavors/default focal Release.gpg 2023-10-06T03:25:48.037 INFO:teuthology.orchestra.run.smithi110.stdout:Hit:5 http://archive.ubuntu.com/ubuntu focal InRelease 2023-10-06T03:25:48.044 INFO:teuthology.orchestra.run.smithi110.stdout:Get:6 https://3.chacra.ceph.com/r/ceph/wip-yuri-testing-2023-10-05-0730-quincy/736ac5ba72cb47af7d6f6666b1e100f7e20a20e2/ubuntu/focal/flavors/default focal/main amd64 Packages [17.4 kB] 2023-10-06T03:25:48.075 INFO:teuthology.orchestra.run.smithi192.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2023-10-06T03:25:48.075 INFO:teuthology.orchestra.run.smithi192.stdout:Hit:2 http://security.ubuntu.com/ubuntu focal-security InRelease 2023-10-06T03:25:48.083 INFO:teuthology.orchestra.run.smithi192.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2023-10-06T03:25:48.100 INFO:teuthology.orchestra.run.smithi192.stdout:Hit:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2023-10-06T03:25:48.100 INFO:teuthology.orchestra.run.smithi110.stdout:Get:7 https://3.chacra.ceph.com/r/ceph/wip-yuri-testing-2023-10-05-0730-quincy/736ac5ba72cb47af7d6f6666b1e100f7e20a20e2/ubuntu/focal/flavors/default focal/main i386 Packages [3,380 B] 2023-10-06T03:25:48.119 INFO:teuthology.orchestra.run.smithi110.stdout:Hit:8 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2023-10-06T03:25:48.195 INFO:teuthology.orchestra.run.smithi192.stdout:Ign:5 https://3.chacra.ceph.com/r/ceph/wip-yuri-testing-2023-10-05-0730-quincy/736ac5ba72cb47af7d6f6666b1e100f7e20a20e2/ubuntu/focal/flavors/default focal InRelease 2023-10-06T03:25:48.201 INFO:teuthology.orchestra.run.smithi110.stdout:Hit:9 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2023-10-06T03:25:48.231 INFO:teuthology.orchestra.run.smithi192.stdout:Get:6 https://3.chacra.ceph.com/r/ceph/wip-yuri-testing-2023-10-05-0730-quincy/736ac5ba72cb47af7d6f6666b1e100f7e20a20e2/ubuntu/focal/flavors/default focal Release [7,662 B] 2023-10-06T03:25:48.269 INFO:teuthology.orchestra.run.smithi192.stdout:Ign:7 https://3.chacra.ceph.com/r/ceph/wip-yuri-testing-2023-10-05-0730-quincy/736ac5ba72cb47af7d6f6666b1e100f7e20a20e2/ubuntu/focal/flavors/default focal Release.gpg 2023-10-06T03:25:48.305 INFO:teuthology.orchestra.run.smithi192.stdout:Get:8 https://3.chacra.ceph.com/r/ceph/wip-yuri-testing-2023-10-05-0730-quincy/736ac5ba72cb47af7d6f6666b1e100f7e20a20e2/ubuntu/focal/flavors/default focal/main i386 Packages [3,380 B] 2023-10-06T03:25:48.363 INFO:teuthology.orchestra.run.smithi192.stdout:Get:9 https://3.chacra.ceph.com/r/ceph/wip-yuri-testing-2023-10-05-0730-quincy/736ac5ba72cb47af7d6f6666b1e100f7e20a20e2/ubuntu/focal/flavors/default focal/main amd64 Packages [17.4 kB] 2023-10-06T03:25:48.590 INFO:teuthology.orchestra.run.smithi110.stdout:Fetched 28.4 kB in 1s (43.7 kB/s) 2023-10-06T03:25:48.729 INFO:teuthology.orchestra.run.smithi192.stdout:Fetched 28.4 kB in 1s (44.9 kB/s) 2023-10-06T03:25:49.625 INFO:teuthology.orchestra.run.smithi110.stdout:Reading package lists... 2023-10-06T03:25:49.651 DEBUG:teuthology.orchestra.run.smithi110:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=17.2.6-1296-g736ac5ba-1focal cephadm=17.2.6-1296-g736ac5ba-1focal ceph-mds=17.2.6-1296-g736ac5ba-1focal ceph-mgr=17.2.6-1296-g736ac5ba-1focal ceph-common=17.2.6-1296-g736ac5ba-1focal ceph-fuse=17.2.6-1296-g736ac5ba-1focal ceph-test=17.2.6-1296-g736ac5ba-1focal ceph-volume=17.2.6-1296-g736ac5ba-1focal radosgw=17.2.6-1296-g736ac5ba-1focal python3-rados=17.2.6-1296-g736ac5ba-1focal python3-rgw=17.2.6-1296-g736ac5ba-1focal python3-cephfs=17.2.6-1296-g736ac5ba-1focal python3-rbd=17.2.6-1296-g736ac5ba-1focal libcephfs2=17.2.6-1296-g736ac5ba-1focal libcephfs-dev=17.2.6-1296-g736ac5ba-1focal librados2=17.2.6-1296-g736ac5ba-1focal librbd1=17.2.6-1296-g736ac5ba-1focal rbd-fuse=17.2.6-1296-g736ac5ba-1focal python3-cephfs=17.2.6-1296-g736ac5ba-1focal cephfs-shell=17.2.6-1296-g736ac5ba-1focal cephfs-top=17.2.6-1296-g736ac5ba-1focal cephfs-mirror=17.2.6-1296-g736ac5ba-1focal 2023-10-06T03:25:49.719 INFO:teuthology.orchestra.run.smithi110.stdout:Reading package lists... 2023-10-06T03:25:49.771 INFO:teuthology.orchestra.run.smithi192.stdout:Reading package lists... 2023-10-06T03:25:49.797 DEBUG:teuthology.orchestra.run.smithi192:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=17.2.6-1296-g736ac5ba-1focal cephadm=17.2.6-1296-g736ac5ba-1focal ceph-mds=17.2.6-1296-g736ac5ba-1focal ceph-mgr=17.2.6-1296-g736ac5ba-1focal ceph-common=17.2.6-1296-g736ac5ba-1focal ceph-fuse=17.2.6-1296-g736ac5ba-1focal ceph-test=17.2.6-1296-g736ac5ba-1focal ceph-volume=17.2.6-1296-g736ac5ba-1focal radosgw=17.2.6-1296-g736ac5ba-1focal python3-rados=17.2.6-1296-g736ac5ba-1focal python3-rgw=17.2.6-1296-g736ac5ba-1focal python3-cephfs=17.2.6-1296-g736ac5ba-1focal python3-rbd=17.2.6-1296-g736ac5ba-1focal libcephfs2=17.2.6-1296-g736ac5ba-1focal libcephfs-dev=17.2.6-1296-g736ac5ba-1focal librados2=17.2.6-1296-g736ac5ba-1focal librbd1=17.2.6-1296-g736ac5ba-1focal rbd-fuse=17.2.6-1296-g736ac5ba-1focal python3-cephfs=17.2.6-1296-g736ac5ba-1focal cephfs-shell=17.2.6-1296-g736ac5ba-1focal cephfs-top=17.2.6-1296-g736ac5ba-1focal cephfs-mirror=17.2.6-1296-g736ac5ba-1focal 2023-10-06T03:25:49.866 INFO:teuthology.orchestra.run.smithi192.stdout:Reading package lists... 2023-10-06T03:25:49.920 INFO:teuthology.orchestra.run.smithi110.stdout:Building dependency tree... 2023-10-06T03:25:49.920 INFO:teuthology.orchestra.run.smithi110.stdout:Reading state information... 2023-10-06T03:25:50.061 INFO:teuthology.orchestra.run.smithi192.stdout:Building dependency tree... 2023-10-06T03:25:50.062 INFO:teuthology.orchestra.run.smithi192.stdout:Reading state information... 2023-10-06T03:25:50.109 INFO:teuthology.orchestra.run.smithi110.stdout:The following packages were automatically installed and are no longer required: 2023-10-06T03:25:50.110 INFO:teuthology.orchestra.run.smithi110.stdout: libboost-iostreams1.71.0 libboost-thread1.71.0 2023-10-06T03:25:50.110 INFO:teuthology.orchestra.run.smithi110.stdout:Use 'sudo apt autoremove' to remove them. 2023-10-06T03:25:50.110 INFO:teuthology.orchestra.run.smithi110.stdout:The following additional packages will be installed: 2023-10-06T03:25:50.110 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2023-10-06T03:25:50.110 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd fonts-mathjax jq 2023-10-06T03:25:50.110 INFO:teuthology.orchestra.run.smithi110.stdout: libdouble-conversion3 libjq1 libjs-mathjax libjs-sphinxdoc libjs-underscore 2023-10-06T03:25:50.110 INFO:teuthology.orchestra.run.smithi110.stdout: liblbfgsb0 liblttng-ust0 liblua5.3-0 liboath0 libonig5 libpcre2-16-0 2023-10-06T03:25:50.111 INFO:teuthology.orchestra.run.smithi110.stdout: libqt5core5a libqt5dbus5 libqt5network5 librabbitmq4 libradosstriper1 2023-10-06T03:25:50.111 INFO:teuthology.orchestra.run.smithi110.stdout: librdkafka1 librgw2 libsqlite3-mod-ceph libthrift-0.13.0 nvme-cli 2023-10-06T03:25:50.111 INFO:teuthology.orchestra.run.smithi110.stdout: python-natsort-doc python-pastedeploy-tpl python3-atomicwrites python3-attr 2023-10-06T03:25:50.111 INFO:teuthology.orchestra.run.smithi110.stdout: python3-bcrypt python3-blinker python3-bs4 python3-cachetools 2023-10-06T03:25:50.111 INFO:teuthology.orchestra.run.smithi110.stdout: python3-ceph-argparse python3-ceph-common python3-cherrypy3 python3-cmd2 2023-10-06T03:25:50.112 INFO:teuthology.orchestra.run.smithi110.stdout: python3-colorama python3-cryptography python3-decorator python3-google-auth 2023-10-06T03:25:50.112 INFO:teuthology.orchestra.run.smithi110.stdout: python3-html5lib python3-jinja2 python3-joblib python3-jwt 2023-10-06T03:25:50.112 INFO:teuthology.orchestra.run.smithi110.stdout: python3-kubernetes python3-logutils python3-lxml python3-mako 2023-10-06T03:25:50.112 INFO:teuthology.orchestra.run.smithi110.stdout: python3-markupsafe python3-natsort python3-oauthlib python3-openssl 2023-10-06T03:25:50.112 INFO:teuthology.orchestra.run.smithi110.stdout: python3-packaging python3-paste python3-pastedeploy python3-pastescript 2023-10-06T03:25:50.112 INFO:teuthology.orchestra.run.smithi110.stdout: python3-pecan python3-pluggy python3-prettytable python3-psutil python3-py 2023-10-06T03:25:50.112 INFO:teuthology.orchestra.run.smithi110.stdout: python3-pyasn1 python3-pyasn1-modules python3-pyinotify python3-pyperclip 2023-10-06T03:25:50.112 INFO:teuthology.orchestra.run.smithi110.stdout: python3-pytest python3-repoze.lru python3-requests-oauthlib python3-routes 2023-10-06T03:25:50.112 INFO:teuthology.orchestra.run.smithi110.stdout: python3-rsa python3-scipy python3-simplegeneric python3-simplejson 2023-10-06T03:25:50.112 INFO:teuthology.orchestra.run.smithi110.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib python3-soupsieve 2023-10-06T03:25:50.112 INFO:teuthology.orchestra.run.smithi110.stdout: python3-tempita python3-waitress python3-wcwidth python3-webencodings 2023-10-06T03:25:50.112 INFO:teuthology.orchestra.run.smithi110.stdout: python3-webob python3-websocket python3-webtest python3-werkzeug 2023-10-06T03:25:50.113 INFO:teuthology.orchestra.run.smithi110.stdout: qttranslations5-l10n socat xmlstarlet 2023-10-06T03:25:50.113 INFO:teuthology.orchestra.run.smithi110.stdout:Suggested packages: 2023-10-06T03:25:50.113 INFO:teuthology.orchestra.run.smithi110.stdout: python3-influxdb fonts-mathjax-extras fonts-stix libjs-mathjax-doc 2023-10-06T03:25:50.113 INFO:teuthology.orchestra.run.smithi110.stdout: python-attr-doc python-blinker-doc python-cryptography-doc 2023-10-06T03:25:50.113 INFO:teuthology.orchestra.run.smithi110.stdout: python3-cryptography-vectors python3-genshi python-jinja2-doc python3-crypto 2023-10-06T03:25:50.113 INFO:teuthology.orchestra.run.smithi110.stdout: python3-lxml-dbg python-lxml-doc python3-beaker python-mako-doc 2023-10-06T03:25:50.113 INFO:teuthology.orchestra.run.smithi110.stdout: python-openssl-doc python3-openssl-dbg httpd-wsgi libapache2-mod-python 2023-10-06T03:25:50.113 INFO:teuthology.orchestra.run.smithi110.stdout: libapache2-mod-scgi libjs-mochikit python-pecan-doc python-psutil-doc 2023-10-06T03:25:50.114 INFO:teuthology.orchestra.run.smithi110.stdout: subversion python-pyinotify-doc python-scipy-doc python3-dap 2023-10-06T03:25:50.114 INFO:teuthology.orchestra.run.smithi110.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2023-10-06T03:25:50.114 INFO:teuthology.orchestra.run.smithi110.stdout: python-webtest-doc python-werkzeug-doc python3-termcolor python3-watchdog 2023-10-06T03:25:50.114 INFO:teuthology.orchestra.run.smithi110.stdout:Recommended packages: 2023-10-06T03:25:50.114 INFO:teuthology.orchestra.run.smithi110.stdout: btrfs-tools 2023-10-06T03:25:50.231 INFO:teuthology.orchestra.run.smithi110.stdout:The following NEW packages will be installed: 2023-10-06T03:25:50.231 INFO:teuthology.orchestra.run.smithi110.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2023-10-06T03:25:50.231 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2023-10-06T03:25:50.231 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm 2023-10-06T03:25:50.231 INFO:teuthology.orchestra.run.smithi110.stdout: cephfs-mirror cephfs-shell cephfs-top fonts-mathjax jq libcephfs-dev 2023-10-06T03:25:50.231 INFO:teuthology.orchestra.run.smithi110.stdout: libcephfs2 libdouble-conversion3 libjq1 libjs-mathjax libjs-sphinxdoc 2023-10-06T03:25:50.231 INFO:teuthology.orchestra.run.smithi110.stdout: libjs-underscore liblbfgsb0 liblttng-ust0 liblua5.3-0 liboath0 libonig5 2023-10-06T03:25:50.232 INFO:teuthology.orchestra.run.smithi110.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 librabbitmq4 2023-10-06T03:25:50.232 INFO:teuthology.orchestra.run.smithi110.stdout: libradosstriper1 librdkafka1 librgw2 libsqlite3-mod-ceph libthrift-0.13.0 2023-10-06T03:25:50.232 INFO:teuthology.orchestra.run.smithi110.stdout: nvme-cli python-natsort-doc python-pastedeploy-tpl python3-atomicwrites 2023-10-06T03:25:50.232 INFO:teuthology.orchestra.run.smithi110.stdout: python3-attr python3-bcrypt python3-blinker python3-bs4 python3-cachetools 2023-10-06T03:25:50.232 INFO:teuthology.orchestra.run.smithi110.stdout: python3-ceph-argparse python3-ceph-common python3-cephfs python3-cherrypy3 2023-10-06T03:25:50.232 INFO:teuthology.orchestra.run.smithi110.stdout: python3-cmd2 python3-colorama python3-cryptography python3-decorator 2023-10-06T03:25:50.232 INFO:teuthology.orchestra.run.smithi110.stdout: python3-google-auth python3-html5lib python3-jinja2 python3-joblib 2023-10-06T03:25:50.232 INFO:teuthology.orchestra.run.smithi110.stdout: python3-jwt python3-kubernetes python3-logutils python3-lxml python3-mako 2023-10-06T03:25:50.232 INFO:teuthology.orchestra.run.smithi110.stdout: python3-markupsafe python3-natsort python3-oauthlib python3-openssl 2023-10-06T03:25:50.233 INFO:teuthology.orchestra.run.smithi110.stdout: python3-packaging python3-paste python3-pastedeploy python3-pastescript 2023-10-06T03:25:50.233 INFO:teuthology.orchestra.run.smithi110.stdout: python3-pecan python3-pluggy python3-prettytable python3-psutil python3-py 2023-10-06T03:25:50.233 INFO:teuthology.orchestra.run.smithi110.stdout: python3-pyasn1 python3-pyasn1-modules python3-pyinotify python3-pyperclip 2023-10-06T03:25:50.233 INFO:teuthology.orchestra.run.smithi110.stdout: python3-pytest python3-rados python3-rbd python3-repoze.lru 2023-10-06T03:25:50.233 INFO:teuthology.orchestra.run.smithi110.stdout: python3-requests-oauthlib python3-rgw python3-routes python3-rsa 2023-10-06T03:25:50.233 INFO:teuthology.orchestra.run.smithi110.stdout: python3-scipy python3-simplegeneric python3-simplejson 2023-10-06T03:25:50.233 INFO:teuthology.orchestra.run.smithi110.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib python3-soupsieve 2023-10-06T03:25:50.233 INFO:teuthology.orchestra.run.smithi110.stdout: python3-tempita python3-waitress python3-wcwidth python3-webencodings 2023-10-06T03:25:50.233 INFO:teuthology.orchestra.run.smithi110.stdout: python3-webob python3-websocket python3-webtest python3-werkzeug 2023-10-06T03:25:50.233 INFO:teuthology.orchestra.run.smithi110.stdout: qttranslations5-l10n radosgw rbd-fuse socat xmlstarlet 2023-10-06T03:25:50.234 INFO:teuthology.orchestra.run.smithi110.stdout:The following packages will be upgraded: 2023-10-06T03:25:50.235 INFO:teuthology.orchestra.run.smithi110.stdout: librados2 librbd1 2023-10-06T03:25:50.235 INFO:teuthology.orchestra.run.smithi192.stdout:The following packages were automatically installed and are no longer required: 2023-10-06T03:25:50.236 INFO:teuthology.orchestra.run.smithi192.stdout: libboost-iostreams1.71.0 libboost-thread1.71.0 2023-10-06T03:25:50.236 INFO:teuthology.orchestra.run.smithi192.stdout:Use 'sudo apt autoremove' to remove them. 2023-10-06T03:25:50.236 INFO:teuthology.orchestra.run.smithi192.stdout:The following additional packages will be installed: 2023-10-06T03:25:50.236 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2023-10-06T03:25:50.236 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd fonts-mathjax jq 2023-10-06T03:25:50.236 INFO:teuthology.orchestra.run.smithi192.stdout: libdouble-conversion3 libjq1 libjs-mathjax libjs-sphinxdoc libjs-underscore 2023-10-06T03:25:50.236 INFO:teuthology.orchestra.run.smithi192.stdout: liblbfgsb0 liblttng-ust0 liblua5.3-0 liboath0 libonig5 libpcre2-16-0 2023-10-06T03:25:50.237 INFO:teuthology.orchestra.run.smithi192.stdout: libqt5core5a libqt5dbus5 libqt5network5 librabbitmq4 libradosstriper1 2023-10-06T03:25:50.237 INFO:teuthology.orchestra.run.smithi192.stdout: librdkafka1 librgw2 libsqlite3-mod-ceph libthrift-0.13.0 nvme-cli 2023-10-06T03:25:50.237 INFO:teuthology.orchestra.run.smithi192.stdout: python-natsort-doc python-pastedeploy-tpl python3-atomicwrites python3-attr 2023-10-06T03:25:50.237 INFO:teuthology.orchestra.run.smithi192.stdout: python3-bcrypt python3-blinker python3-bs4 python3-cachetools 2023-10-06T03:25:50.237 INFO:teuthology.orchestra.run.smithi192.stdout: python3-ceph-argparse python3-ceph-common python3-cherrypy3 python3-cmd2 2023-10-06T03:25:50.237 INFO:teuthology.orchestra.run.smithi192.stdout: python3-colorama python3-cryptography python3-decorator python3-google-auth 2023-10-06T03:25:50.237 INFO:teuthology.orchestra.run.smithi192.stdout: python3-html5lib python3-jinja2 python3-joblib python3-jwt 2023-10-06T03:25:50.237 INFO:teuthology.orchestra.run.smithi192.stdout: python3-kubernetes python3-logutils python3-lxml python3-mako 2023-10-06T03:25:50.237 INFO:teuthology.orchestra.run.smithi192.stdout: python3-markupsafe python3-natsort python3-oauthlib python3-openssl 2023-10-06T03:25:50.237 INFO:teuthology.orchestra.run.smithi192.stdout: python3-packaging python3-paste python3-pastedeploy python3-pastescript 2023-10-06T03:25:50.238 INFO:teuthology.orchestra.run.smithi192.stdout: python3-pecan python3-pluggy python3-prettytable python3-psutil python3-py 2023-10-06T03:25:50.238 INFO:teuthology.orchestra.run.smithi192.stdout: python3-pyasn1 python3-pyasn1-modules python3-pyinotify python3-pyperclip 2023-10-06T03:25:50.238 INFO:teuthology.orchestra.run.smithi192.stdout: python3-pytest python3-repoze.lru python3-requests-oauthlib python3-routes 2023-10-06T03:25:50.238 INFO:teuthology.orchestra.run.smithi192.stdout: python3-rsa python3-scipy python3-simplegeneric python3-simplejson 2023-10-06T03:25:50.238 INFO:teuthology.orchestra.run.smithi192.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib python3-soupsieve 2023-10-06T03:25:50.238 INFO:teuthology.orchestra.run.smithi192.stdout: python3-tempita python3-waitress python3-wcwidth python3-webencodings 2023-10-06T03:25:50.238 INFO:teuthology.orchestra.run.smithi192.stdout: python3-webob python3-websocket python3-webtest python3-werkzeug 2023-10-06T03:25:50.238 INFO:teuthology.orchestra.run.smithi192.stdout: qttranslations5-l10n socat xmlstarlet 2023-10-06T03:25:50.238 INFO:teuthology.orchestra.run.smithi192.stdout:Suggested packages: 2023-10-06T03:25:50.238 INFO:teuthology.orchestra.run.smithi192.stdout: python3-influxdb fonts-mathjax-extras fonts-stix libjs-mathjax-doc 2023-10-06T03:25:50.239 INFO:teuthology.orchestra.run.smithi192.stdout: python-attr-doc python-blinker-doc python-cryptography-doc 2023-10-06T03:25:50.239 INFO:teuthology.orchestra.run.smithi192.stdout: python3-cryptography-vectors python3-genshi python-jinja2-doc python3-crypto 2023-10-06T03:25:50.239 INFO:teuthology.orchestra.run.smithi192.stdout: python3-lxml-dbg python-lxml-doc python3-beaker python-mako-doc 2023-10-06T03:25:50.239 INFO:teuthology.orchestra.run.smithi192.stdout: python-openssl-doc python3-openssl-dbg httpd-wsgi libapache2-mod-python 2023-10-06T03:25:50.239 INFO:teuthology.orchestra.run.smithi192.stdout: libapache2-mod-scgi libjs-mochikit python-pecan-doc python-psutil-doc 2023-10-06T03:25:50.239 INFO:teuthology.orchestra.run.smithi192.stdout: subversion python-pyinotify-doc python-scipy-doc python3-dap 2023-10-06T03:25:50.239 INFO:teuthology.orchestra.run.smithi192.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2023-10-06T03:25:50.239 INFO:teuthology.orchestra.run.smithi192.stdout: python-webtest-doc python-werkzeug-doc python3-termcolor python3-watchdog 2023-10-06T03:25:50.239 INFO:teuthology.orchestra.run.smithi192.stdout:Recommended packages: 2023-10-06T03:25:50.239 INFO:teuthology.orchestra.run.smithi192.stdout: btrfs-tools 2023-10-06T03:25:50.353 INFO:teuthology.orchestra.run.smithi192.stdout:The following NEW packages will be installed: 2023-10-06T03:25:50.353 INFO:teuthology.orchestra.run.smithi192.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2023-10-06T03:25:50.354 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2023-10-06T03:25:50.354 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm 2023-10-06T03:25:50.354 INFO:teuthology.orchestra.run.smithi192.stdout: cephfs-mirror cephfs-shell cephfs-top fonts-mathjax jq libcephfs-dev 2023-10-06T03:25:50.354 INFO:teuthology.orchestra.run.smithi192.stdout: libcephfs2 libdouble-conversion3 libjq1 libjs-mathjax libjs-sphinxdoc 2023-10-06T03:25:50.354 INFO:teuthology.orchestra.run.smithi192.stdout: libjs-underscore liblbfgsb0 liblttng-ust0 liblua5.3-0 liboath0 libonig5 2023-10-06T03:25:50.354 INFO:teuthology.orchestra.run.smithi192.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 librabbitmq4 2023-10-06T03:25:50.355 INFO:teuthology.orchestra.run.smithi192.stdout: libradosstriper1 librdkafka1 librgw2 libsqlite3-mod-ceph libthrift-0.13.0 2023-10-06T03:25:50.355 INFO:teuthology.orchestra.run.smithi192.stdout: nvme-cli python-natsort-doc python-pastedeploy-tpl python3-atomicwrites 2023-10-06T03:25:50.356 INFO:teuthology.orchestra.run.smithi192.stdout: python3-attr python3-bcrypt python3-blinker python3-bs4 python3-cachetools 2023-10-06T03:25:50.356 INFO:teuthology.orchestra.run.smithi192.stdout: python3-ceph-argparse python3-ceph-common python3-cephfs python3-cherrypy3 2023-10-06T03:25:50.356 INFO:teuthology.orchestra.run.smithi192.stdout: python3-cmd2 python3-colorama python3-cryptography python3-decorator 2023-10-06T03:25:50.356 INFO:teuthology.orchestra.run.smithi192.stdout: python3-google-auth python3-html5lib python3-jinja2 python3-joblib 2023-10-06T03:25:50.356 INFO:teuthology.orchestra.run.smithi192.stdout: python3-jwt python3-kubernetes python3-logutils python3-lxml python3-mako 2023-10-06T03:25:50.356 INFO:teuthology.orchestra.run.smithi192.stdout: python3-markupsafe python3-natsort python3-oauthlib python3-openssl 2023-10-06T03:25:50.356 INFO:teuthology.orchestra.run.smithi192.stdout: python3-packaging python3-paste python3-pastedeploy python3-pastescript 2023-10-06T03:25:50.356 INFO:teuthology.orchestra.run.smithi192.stdout: python3-pecan python3-pluggy python3-prettytable python3-psutil python3-py 2023-10-06T03:25:50.356 INFO:teuthology.orchestra.run.smithi192.stdout: python3-pyasn1 python3-pyasn1-modules python3-pyinotify python3-pyperclip 2023-10-06T03:25:50.356 INFO:teuthology.orchestra.run.smithi192.stdout: python3-pytest python3-rados python3-rbd python3-repoze.lru 2023-10-06T03:25:50.357 INFO:teuthology.orchestra.run.smithi192.stdout: python3-requests-oauthlib python3-rgw python3-routes python3-rsa 2023-10-06T03:25:50.357 INFO:teuthology.orchestra.run.smithi192.stdout: python3-scipy python3-simplegeneric python3-simplejson 2023-10-06T03:25:50.357 INFO:teuthology.orchestra.run.smithi192.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib python3-soupsieve 2023-10-06T03:25:50.357 INFO:teuthology.orchestra.run.smithi192.stdout: python3-tempita python3-waitress python3-wcwidth python3-webencodings 2023-10-06T03:25:50.357 INFO:teuthology.orchestra.run.smithi192.stdout: python3-webob python3-websocket python3-webtest python3-werkzeug 2023-10-06T03:25:50.357 INFO:teuthology.orchestra.run.smithi192.stdout: qttranslations5-l10n radosgw rbd-fuse socat xmlstarlet 2023-10-06T03:25:50.357 INFO:teuthology.orchestra.run.smithi192.stdout:The following packages will be upgraded: 2023-10-06T03:25:50.357 INFO:teuthology.orchestra.run.smithi192.stdout: librados2 librbd1 2023-10-06T03:25:50.383 INFO:teuthology.orchestra.run.smithi110.stdout:2 upgraded, 114 newly installed, 0 to remove and 253 not upgraded. 2023-10-06T03:25:50.383 INFO:teuthology.orchestra.run.smithi110.stdout:Need to get 210 MB of archives. 2023-10-06T03:25:50.383 INFO:teuthology.orchestra.run.smithi110.stdout:After this operation, 981 MB of additional disk space will be used. 2023-10-06T03:25:50.383 INFO:teuthology.orchestra.run.smithi110.stdout:Get:1 https://3.chacra.ceph.com/r/ceph/wip-yuri-testing-2023-10-05-0730-quincy/736ac5ba72cb47af7d6f6666b1e100f7e20a20e2/ubuntu/focal/flavors/default focal/main amd64 librbd1 amd64 17.2.6-1296-g736ac5ba-1focal [3,501 kB] 2023-10-06T03:25:50.418 INFO:teuthology.orchestra.run.smithi192.stdout:2 upgraded, 114 newly installed, 0 to remove and 253 not upgraded. 2023-10-06T03:25:50.418 INFO:teuthology.orchestra.run.smithi192.stdout:Need to get 210 MB of archives. 2023-10-06T03:25:50.418 INFO:teuthology.orchestra.run.smithi192.stdout:After this operation, 981 MB of additional disk space will be used. 2023-10-06T03:25:50.418 INFO:teuthology.orchestra.run.smithi192.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyperclip all 1.7.0-1 [10.5 kB] 2023-10-06T03:25:50.424 INFO:teuthology.orchestra.run.smithi110.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyperclip all 1.7.0-1 [10.5 kB] 2023-10-06T03:25:50.436 INFO:teuthology.orchestra.run.smithi192.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-wcwidth all 0.1.8+dfsg1-3 [17.4 kB] 2023-10-06T03:25:50.444 INFO:teuthology.orchestra.run.smithi192.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-cmd2 all 0.8.5-2.1 [49.6 kB] 2023-10-06T03:25:50.469 INFO:teuthology.orchestra.run.smithi192.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal/main amd64 liblttng-ust0 amd64 2.11.0-1 [160 kB] 2023-10-06T03:25:50.498 INFO:teuthology.orchestra.run.smithi192.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-prettytable all 0.7.2-5 [20.1 kB] 2023-10-06T03:25:50.500 INFO:teuthology.orchestra.run.smithi192.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal/main amd64 liblua5.3-0 amd64 5.3.3-1.1ubuntu2 [116 kB] 2023-10-06T03:25:50.501 INFO:teuthology.orchestra.run.smithi192.stdout:Get:7 https://3.chacra.ceph.com/r/ceph/wip-yuri-testing-2023-10-05-0730-quincy/736ac5ba72cb47af7d6f6666b1e100f7e20a20e2/ubuntu/focal/flavors/default focal/main amd64 librbd1 amd64 17.2.6-1296-g736ac5ba-1focal [3,501 kB] 2023-10-06T03:25:50.508 INFO:teuthology.orchestra.run.smithi110.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-wcwidth all 0.1.8+dfsg1-3 [17.4 kB] 2023-10-06T03:25:50.509 INFO:teuthology.orchestra.run.smithi192.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal/main amd64 librabbitmq4 amd64 0.10.0-1 [35.8 kB] 2023-10-06T03:25:50.512 INFO:teuthology.orchestra.run.smithi192.stdout:Get:9 http://archive.ubuntu.com/ubuntu focal/universe amd64 librdkafka1 amd64 1.2.1-1ubuntu1 [410 kB] 2023-10-06T03:25:50.529 INFO:teuthology.orchestra.run.smithi192.stdout:Get:10 http://archive.ubuntu.com/ubuntu focal/universe amd64 libdouble-conversion3 amd64 3.1.5-4ubuntu1 [37.9 kB] 2023-10-06T03:25:50.531 INFO:teuthology.orchestra.run.smithi192.stdout:Get:11 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libpcre2-16-0 amd64 10.34-7ubuntu0.1 [181 kB] 2023-10-06T03:25:50.535 INFO:teuthology.orchestra.run.smithi192.stdout:Get:12 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libqt5core5a amd64 5.12.8+dfsg-0ubuntu2.1 [2,006 kB] 2023-10-06T03:25:50.542 INFO:teuthology.orchestra.run.smithi110.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-cmd2 all 0.8.5-2.1 [49.6 kB] 2023-10-06T03:25:50.562 INFO:teuthology.orchestra.run.smithi192.stdout:Get:13 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libqt5dbus5 amd64 5.12.8+dfsg-0ubuntu2.1 [208 kB] 2023-10-06T03:25:50.564 INFO:teuthology.orchestra.run.smithi192.stdout:Get:14 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libqt5network5 amd64 5.12.8+dfsg-0ubuntu2.1 [673 kB] 2023-10-06T03:25:50.570 INFO:teuthology.orchestra.run.smithi192.stdout:Get:15 http://archive.ubuntu.com/ubuntu focal/universe amd64 libthrift-0.13.0 amd64 0.13.0-2build2 [249 kB] 2023-10-06T03:25:50.571 INFO:teuthology.orchestra.run.smithi192.stdout:Get:16 http://archive.ubuntu.com/ubuntu focal/main amd64 liboath0 amd64 2.6.1-1.3 [44.7 kB] 2023-10-06T03:25:50.572 INFO:teuthology.orchestra.run.smithi192.stdout:Get:17 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-cryptography amd64 2.8-3ubuntu0.1 [211 kB] 2023-10-06T03:25:50.574 INFO:teuthology.orchestra.run.smithi192.stdout:Get:18 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-openssl all 19.0.0-1build1 [43.3 kB] 2023-10-06T03:25:50.574 INFO:teuthology.orchestra.run.smithi192.stdout:Get:19 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-bcrypt amd64 3.1.7-2ubuntu1 [30.4 kB] 2023-10-06T03:25:50.575 INFO:teuthology.orchestra.run.smithi192.stdout:Get:20 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-cherrypy3 all 8.9.1-6 [160 kB] 2023-10-06T03:25:50.576 INFO:teuthology.orchestra.run.smithi192.stdout:Get:21 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-natsort all 7.0.1-1 [29.2 kB] 2023-10-06T03:25:50.585 INFO:teuthology.orchestra.run.smithi192.stdout:Get:22 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-jwt all 1.7.1-2ubuntu2.1 [18.0 kB] 2023-10-06T03:25:50.585 INFO:teuthology.orchestra.run.smithi192.stdout:Get:23 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-logutils all 0.3.3-7 [17.3 kB] 2023-10-06T03:25:50.591 INFO:teuthology.orchestra.run.smithi110.stdout:Get:5 https://3.chacra.ceph.com/r/ceph/wip-yuri-testing-2023-10-05-0730-quincy/736ac5ba72cb47af7d6f6666b1e100f7e20a20e2/ubuntu/focal/flavors/default focal/main amd64 librados2 amd64 17.2.6-1296-g736ac5ba-1focal [3,554 kB] 2023-10-06T03:25:50.601 INFO:teuthology.orchestra.run.smithi192.stdout:Get:24 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-markupsafe amd64 1.1.0-1build2 [13.9 kB] 2023-10-06T03:25:50.602 INFO:teuthology.orchestra.run.smithi192.stdout:Get:25 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-mako all 1.1.0+ds1-1ubuntu2.1 [59.2 kB] 2023-10-06T03:25:50.602 INFO:teuthology.orchestra.run.smithi192.stdout:Get:26 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2023-10-06T03:25:50.603 INFO:teuthology.orchestra.run.smithi192.stdout:Get:27 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-singledispatch all 3.4.0.3-2build1 [7,128 B] 2023-10-06T03:25:50.603 INFO:teuthology.orchestra.run.smithi192.stdout:Get:28 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webob all 1:1.8.5-2 [84.4 kB] 2023-10-06T03:25:50.605 INFO:teuthology.orchestra.run.smithi192.stdout:Get:29 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-soupsieve all 1.9.5+dfsg-1 [29.1 kB] 2023-10-06T03:25:50.605 INFO:teuthology.orchestra.run.smithi192.stdout:Get:30 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-bs4 all 4.8.2-1 [83.0 kB] 2023-10-06T03:25:50.606 INFO:teuthology.orchestra.run.smithi192.stdout:Get:31 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-waitress all 1.4.1-1ubuntu0.1 [81.4 kB] 2023-10-06T03:25:50.618 INFO:teuthology.orchestra.run.smithi192.stdout:Get:32 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-tempita all 0.5.2-6 [14.8 kB] 2023-10-06T03:25:50.619 INFO:teuthology.orchestra.run.smithi192.stdout:Get:33 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-paste all 3.4.0+dfsg1-1 [456 kB] 2023-10-06T03:25:50.635 INFO:teuthology.orchestra.run.smithi192.stdout:Get:34 http://archive.ubuntu.com/ubuntu focal/main amd64 python-pastedeploy-tpl all 2.1.0-1 [5,016 B] 2023-10-06T03:25:50.637 INFO:teuthology.orchestra.run.smithi192.stdout:Get:35 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pastedeploy all 2.1.0-1 [26.5 kB] 2023-10-06T03:25:50.638 INFO:teuthology.orchestra.run.smithi192.stdout:Get:36 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webtest all 2.0.32-2 [28.2 kB] 2023-10-06T03:25:50.638 INFO:teuthology.orchestra.run.smithi192.stdout:Get:37 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pecan all 1.3.3-3ubuntu1 [85.8 kB] 2023-10-06T03:25:50.638 INFO:teuthology.orchestra.run.smithi192.stdout:Get:38 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-werkzeug all 0.16.1+dfsg1-2ubuntu0.1 [183 kB] 2023-10-06T03:25:50.640 INFO:teuthology.orchestra.run.smithi192.stdout:Get:39 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-jinja2 all 2.10.1-2 [95.5 kB] 2023-10-06T03:25:50.641 INFO:teuthology.orchestra.run.smithi192.stdout:Get:40 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2023-10-06T03:25:50.642 INFO:teuthology.orchestra.run.smithi192.stdout:Get:41 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libjs-underscore all 1.9.1~dfsg-1ubuntu0.20.04.1 [99.5 kB] 2023-10-06T03:25:50.652 INFO:teuthology.orchestra.run.smithi192.stdout:Get:42 http://archive.ubuntu.com/ubuntu focal/main amd64 libjs-sphinxdoc all 1.8.5-7ubuntu3 [97.1 kB] 2023-10-06T03:25:50.653 INFO:teuthology.orchestra.run.smithi192.stdout:Get:43 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-routes all 2.4.1-2 [88.1 kB] 2023-10-06T03:25:50.666 INFO:teuthology.orchestra.run.smithi110.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal/main amd64 liblttng-ust0 amd64 2.11.0-1 [160 kB] 2023-10-06T03:25:50.669 INFO:teuthology.orchestra.run.smithi192.stdout:Get:44 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-decorator all 4.4.2-0ubuntu1 [10.3 kB] 2023-10-06T03:25:50.669 INFO:teuthology.orchestra.run.smithi192.stdout:Get:45 http://archive.ubuntu.com/ubuntu focal/universe amd64 liblbfgsb0 amd64 3.0+dfsg.3-7build1 [29.5 kB] 2023-10-06T03:25:50.670 INFO:teuthology.orchestra.run.smithi192.stdout:Get:46 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-scipy amd64 1.3.3-3build1 [11.1 MB] 2023-10-06T03:25:50.751 INFO:teuthology.orchestra.run.smithi192.stdout:Get:47 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-sklearn-lib amd64 0.22.2.post1+dfsg-5 [2,358 kB] 2023-10-06T03:25:50.768 INFO:teuthology.orchestra.run.smithi192.stdout:Get:48 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-joblib all 0.14.0-3 [192 kB] 2023-10-06T03:25:50.805 INFO:teuthology.orchestra.run.smithi110.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-prettytable all 0.7.2-5 [20.1 kB] 2023-10-06T03:25:50.814 INFO:teuthology.orchestra.run.smithi110.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal/main amd64 liblua5.3-0 amd64 5.3.3-1.1ubuntu2 [116 kB] 2023-10-06T03:25:50.827 INFO:teuthology.orchestra.run.smithi192.stdout:Get:49 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-sklearn all 0.22.2.post1+dfsg-5 [1,773 kB] 2023-10-06T03:25:50.847 INFO:teuthology.orchestra.run.smithi192.stdout:Get:50 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-cachetools all 4.0.0-1 [9,820 B] 2023-10-06T03:25:50.847 INFO:teuthology.orchestra.run.smithi192.stdout:Get:51 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyasn1 all 0.4.2-3build1 [46.7 kB] 2023-10-06T03:25:50.848 INFO:teuthology.orchestra.run.smithi110.stdout:Get:9 https://3.chacra.ceph.com/r/ceph/wip-yuri-testing-2023-10-05-0730-quincy/736ac5ba72cb47af7d6f6666b1e100f7e20a20e2/ubuntu/focal/flavors/default focal/main amd64 libcephfs2 amd64 17.2.6-1296-g736ac5ba-1focal [760 kB] 2023-10-06T03:25:50.848 INFO:teuthology.orchestra.run.smithi192.stdout:Get:52 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyasn1-modules all 0.2.1-0.2build1 [32.9 kB] 2023-10-06T03:25:50.848 INFO:teuthology.orchestra.run.smithi192.stdout:Get:53 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-rsa all 4.0-3 [24.7 kB] 2023-10-06T03:25:50.848 INFO:teuthology.orchestra.run.smithi192.stdout:Get:54 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-google-auth all 1.5.1-2 [35.6 kB] 2023-10-06T03:25:50.849 INFO:teuthology.orchestra.run.smithi192.stdout:Get:55 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-blinker all 1.4+dfsg1-0.3ubuntu1 [13.2 kB] 2023-10-06T03:25:50.849 INFO:teuthology.orchestra.run.smithi192.stdout:Get:56 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-oauthlib all 3.1.0-1ubuntu2 [84.8 kB] 2023-10-06T03:25:50.850 INFO:teuthology.orchestra.run.smithi192.stdout:Get:57 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-requests-oauthlib all 1.0.0-1.1build1 [16.8 kB] 2023-10-06T03:25:50.859 INFO:teuthology.orchestra.run.smithi192.stdout:Get:58 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-websocket all 0.53.0-2ubuntu1 [32.3 kB] 2023-10-06T03:25:50.860 INFO:teuthology.orchestra.run.smithi110.stdout:Get:10 http://archive.ubuntu.com/ubuntu focal/main amd64 librabbitmq4 amd64 0.10.0-1 [35.8 kB] 2023-10-06T03:25:50.865 INFO:teuthology.orchestra.run.smithi192.stdout:Get:59 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-kubernetes all 7.0.0~a1-2 [271 kB] 2023-10-06T03:25:50.868 INFO:teuthology.orchestra.run.smithi192.stdout:Get:60 http://archive.ubuntu.com/ubuntu focal/universe amd64 libonig5 amd64 6.9.4-1 [142 kB] 2023-10-06T03:25:50.870 INFO:teuthology.orchestra.run.smithi110.stdout:Get:11 http://archive.ubuntu.com/ubuntu focal/universe amd64 librdkafka1 amd64 1.2.1-1ubuntu1 [410 kB] 2023-10-06T03:25:50.871 INFO:teuthology.orchestra.run.smithi110.stdout:Get:12 https://3.chacra.ceph.com/r/ceph/wip-yuri-testing-2023-10-05-0730-quincy/736ac5ba72cb47af7d6f6666b1e100f7e20a20e2/ubuntu/focal/flavors/default focal/main amd64 python3-rados amd64 17.2.6-1296-g736ac5ba-1focal [413 kB] 2023-10-06T03:25:50.881 INFO:teuthology.orchestra.run.smithi192.stdout:Get:61 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libjq1 amd64 1.6-1ubuntu0.20.04.1 [121 kB] 2023-10-06T03:25:50.884 INFO:teuthology.orchestra.run.smithi192.stdout:Get:62 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 jq amd64 1.6-1ubuntu0.20.04.1 [50.2 kB] 2023-10-06T03:25:50.885 INFO:teuthology.orchestra.run.smithi192.stdout:Get:63 http://archive.ubuntu.com/ubuntu focal/main amd64 socat amd64 1.7.3.3-2 [323 kB] 2023-10-06T03:25:50.885 INFO:teuthology.orchestra.run.smithi110.stdout:Get:13 https://3.chacra.ceph.com/r/ceph/wip-yuri-testing-2023-10-05-0730-quincy/736ac5ba72cb47af7d6f6666b1e100f7e20a20e2/ubuntu/focal/flavors/default focal/main amd64 python3-ceph-argparse all 17.2.6-1296-g736ac5ba-1focal [22.5 kB] 2023-10-06T03:25:50.886 INFO:teuthology.orchestra.run.smithi110.stdout:Get:14 https://3.chacra.ceph.com/r/ceph/wip-yuri-testing-2023-10-05-0730-quincy/736ac5ba72cb47af7d6f6666b1e100f7e20a20e2/ubuntu/focal/flavors/default focal/main amd64 python3-cephfs amd64 17.2.6-1296-g736ac5ba-1focal [200 kB] 2023-10-06T03:25:50.889 INFO:teuthology.orchestra.run.smithi192.stdout:Get:64 http://archive.ubuntu.com/ubuntu focal/universe amd64 xmlstarlet amd64 1.6.1-2build1 [267 kB] 2023-10-06T03:25:50.892 INFO:teuthology.orchestra.run.smithi192.stdout:Get:65 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-colorama all 0.4.3-1build1 [23.9 kB] 2023-10-06T03:25:50.892 INFO:teuthology.orchestra.run.smithi192.stdout:Get:66 http://archive.ubuntu.com/ubuntu focal/universe amd64 fonts-mathjax all 2.7.4+dfsg-1 [2,208 kB] 2023-10-06T03:25:50.896 INFO:teuthology.orchestra.run.smithi110.stdout:Get:15 https://3.chacra.ceph.com/r/ceph/wip-yuri-testing-2023-10-05-0730-quincy/736ac5ba72cb47af7d6f6666b1e100f7e20a20e2/ubuntu/focal/flavors/default focal/main amd64 python3-ceph-common all 17.2.6-1296-g736ac5ba-1focal [47.1 kB] 2023-10-06T03:25:50.899 INFO:teuthology.orchestra.run.smithi110.stdout:Get:16 https://3.chacra.ceph.com/r/ceph/wip-yuri-testing-2023-10-05-0730-quincy/736ac5ba72cb47af7d6f6666b1e100f7e20a20e2/ubuntu/focal/flavors/default focal/main amd64 python3-rbd amd64 17.2.6-1296-g736ac5ba-1focal [386 kB] 2023-10-06T03:25:50.908 INFO:teuthology.orchestra.run.smithi192.stdout:Get:67 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 nvme-cli amd64 1.9-1ubuntu0.1 [283 kB] 2023-10-06T03:25:50.910 INFO:teuthology.orchestra.run.smithi110.stdout:Get:17 https://3.chacra.ceph.com/r/ceph/wip-yuri-testing-2023-10-05-0730-quincy/736ac5ba72cb47af7d6f6666b1e100f7e20a20e2/ubuntu/focal/flavors/default focal/main amd64 librgw2 amd64 17.2.6-1296-g736ac5ba-1focal [5,570 kB] 2023-10-06T03:25:50.911 INFO:teuthology.orchestra.run.smithi192.stdout:Get:68 http://archive.ubuntu.com/ubuntu focal/universe amd64 libjs-mathjax all 2.7.4+dfsg-1 [5,654 kB] 2023-10-06T03:25:50.947 INFO:teuthology.orchestra.run.smithi192.stdout:Get:69 http://archive.ubuntu.com/ubuntu focal/universe amd64 python-natsort-doc all 7.0.1-1 [103 kB] 2023-10-06T03:25:50.948 INFO:teuthology.orchestra.run.smithi192.stdout:Get:70 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-atomicwrites all 1.1.5-2build1 [7,016 B] 2023-10-06T03:25:50.948 INFO:teuthology.orchestra.run.smithi192.stdout:Get:71 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-attr all 19.3.0-2 [33.9 kB] 2023-10-06T03:25:50.948 INFO:teuthology.orchestra.run.smithi192.stdout:Get:72 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webencodings all 0.5.1-1ubuntu1 [11.0 kB] 2023-10-06T03:25:50.948 INFO:teuthology.orchestra.run.smithi192.stdout:Get:73 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-html5lib all 1.0.1-2 [84.3 kB] 2023-10-06T03:25:50.949 INFO:teuthology.orchestra.run.smithi192.stdout:Get:74 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-lxml amd64 4.5.0-1ubuntu0.5 [1,384 kB] 2023-10-06T03:25:50.953 INFO:teuthology.orchestra.run.smithi110.stdout:Get:18 http://archive.ubuntu.com/ubuntu focal/universe amd64 libdouble-conversion3 amd64 3.1.5-4ubuntu1 [37.9 kB] 2023-10-06T03:25:50.958 INFO:teuthology.orchestra.run.smithi110.stdout:Get:19 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libpcre2-16-0 amd64 10.34-7ubuntu0.1 [181 kB] 2023-10-06T03:25:50.961 INFO:teuthology.orchestra.run.smithi192.stdout:Get:75 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-packaging all 20.3-1 [26.8 kB] 2023-10-06T03:25:50.961 INFO:teuthology.orchestra.run.smithi192.stdout:Get:76 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2023-10-06T03:25:50.961 INFO:teuthology.orchestra.run.smithi192.stdout:Get:77 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-pluggy all 0.13.0-2 [18.4 kB] 2023-10-06T03:25:50.964 INFO:teuthology.orchestra.run.smithi192.stdout:Get:78 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-psutil amd64 5.5.1-1ubuntu4 [141 kB] 2023-10-06T03:25:50.981 INFO:teuthology.orchestra.run.smithi192.stdout:Get:79 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 python3-py all 1.8.1-1ubuntu0.1 [65.6 kB] 2023-10-06T03:25:50.982 INFO:teuthology.orchestra.run.smithi110.stdout:Get:20 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libqt5core5a amd64 5.12.8+dfsg-0ubuntu2.1 [2,006 kB] 2023-10-06T03:25:50.982 INFO:teuthology.orchestra.run.smithi192.stdout:Get:80 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyinotify all 0.9.6-1.2ubuntu1 [24.8 kB] 2023-10-06T03:25:50.983 INFO:teuthology.orchestra.run.smithi192.stdout:Get:81 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-pytest all 4.6.9-1 [178 kB] 2023-10-06T03:25:50.984 INFO:teuthology.orchestra.run.smithi192.stdout:Get:82 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-simplejson amd64 3.16.0-2ubuntu2 [50.8 kB] 2023-10-06T03:25:50.986 INFO:teuthology.orchestra.run.smithi192.stdout:Get:83 http://archive.ubuntu.com/ubuntu focal/universe amd64 qttranslations5-l10n all 5.12.8-0ubuntu1 [1,486 kB] 2023-10-06T03:25:51.107 INFO:teuthology.orchestra.run.smithi110.stdout:Get:21 https://3.chacra.ceph.com/r/ceph/wip-yuri-testing-2023-10-05-0730-quincy/736ac5ba72cb47af7d6f6666b1e100f7e20a20e2/ubuntu/focal/flavors/default focal/main amd64 python3-rgw amd64 17.2.6-1296-g736ac5ba-1focal [116 kB] 2023-10-06T03:25:51.109 INFO:teuthology.orchestra.run.smithi110.stdout:Get:22 https://3.chacra.ceph.com/r/ceph/wip-yuri-testing-2023-10-05-0730-quincy/736ac5ba72cb47af7d6f6666b1e100f7e20a20e2/ubuntu/focal/flavors/default focal/main amd64 libradosstriper1 amd64 17.2.6-1296-g736ac5ba-1focal [479 kB] 2023-10-06T03:25:51.125 INFO:teuthology.orchestra.run.smithi110.stdout:Get:23 https://3.chacra.ceph.com/r/ceph/wip-yuri-testing-2023-10-05-0730-quincy/736ac5ba72cb47af7d6f6666b1e100f7e20a20e2/ubuntu/focal/flavors/default focal/main amd64 ceph-common amd64 17.2.6-1296-g736ac5ba-1focal [27.8 MB] 2023-10-06T03:25:51.275 INFO:teuthology.orchestra.run.smithi110.stdout:Get:24 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libqt5dbus5 amd64 5.12.8+dfsg-0ubuntu2.1 [208 kB] 2023-10-06T03:25:51.707 INFO:teuthology.orchestra.run.smithi110.stdout:Get:25 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libqt5network5 amd64 5.12.8+dfsg-0ubuntu2.1 [673 kB] 2023-10-06T03:25:51.889 INFO:teuthology.orchestra.run.smithi110.stdout:Get:26 http://archive.ubuntu.com/ubuntu focal/universe amd64 libthrift-0.13.0 amd64 0.13.0-2build2 [249 kB] 2023-10-06T03:25:51.921 INFO:teuthology.orchestra.run.smithi110.stdout:Get:27 http://archive.ubuntu.com/ubuntu focal/main amd64 liboath0 amd64 2.6.1-1.3 [44.7 kB] 2023-10-06T03:25:51.926 INFO:teuthology.orchestra.run.smithi110.stdout:Get:28 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-cryptography amd64 2.8-3ubuntu0.1 [211 kB] 2023-10-06T03:25:51.951 INFO:teuthology.orchestra.run.smithi110.stdout:Get:29 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-openssl all 19.0.0-1build1 [43.3 kB] 2023-10-06T03:25:51.956 INFO:teuthology.orchestra.run.smithi110.stdout:Get:30 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-bcrypt amd64 3.1.7-2ubuntu1 [30.4 kB] 2023-10-06T03:25:51.959 INFO:teuthology.orchestra.run.smithi110.stdout:Get:31 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-cherrypy3 all 8.9.1-6 [160 kB] 2023-10-06T03:25:51.971 INFO:teuthology.orchestra.run.smithi110.stdout:Get:32 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-natsort all 7.0.1-1 [29.2 kB] 2023-10-06T03:25:51.974 INFO:teuthology.orchestra.run.smithi110.stdout:Get:33 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-jwt all 1.7.1-2ubuntu2.1 [18.0 kB] 2023-10-06T03:25:51.975 INFO:teuthology.orchestra.run.smithi110.stdout:Get:34 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-logutils all 0.3.3-7 [17.3 kB] 2023-10-06T03:25:52.133 INFO:teuthology.orchestra.run.smithi192.stdout:Get:84 https://3.chacra.ceph.com/r/ceph/wip-yuri-testing-2023-10-05-0730-quincy/736ac5ba72cb47af7d6f6666b1e100f7e20a20e2/ubuntu/focal/flavors/default focal/main amd64 librados2 amd64 17.2.6-1296-g736ac5ba-1focal [3,554 kB] 2023-10-06T03:25:52.140 INFO:teuthology.orchestra.run.smithi110.stdout:Get:35 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-markupsafe amd64 1.1.0-1build2 [13.9 kB] 2023-10-06T03:25:52.245 INFO:teuthology.orchestra.run.smithi110.stdout:Get:36 https://3.chacra.ceph.com/r/ceph/wip-yuri-testing-2023-10-05-0730-quincy/736ac5ba72cb47af7d6f6666b1e100f7e20a20e2/ubuntu/focal/flavors/default focal/main amd64 ceph-base amd64 17.2.6-1296-g736ac5ba-1focal [6,453 kB] 2023-10-06T03:25:52.328 INFO:teuthology.orchestra.run.smithi110.stdout:Get:37 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-mako all 1.1.0+ds1-1ubuntu2.1 [59.2 kB] 2023-10-06T03:25:52.349 INFO:teuthology.orchestra.run.smithi110.stdout:Get:38 https://3.chacra.ceph.com/r/ceph/wip-yuri-testing-2023-10-05-0730-quincy/736ac5ba72cb47af7d6f6666b1e100f7e20a20e2/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-modules-core all 17.2.6-1296-g736ac5ba-1focal [221 kB] 2023-10-06T03:25:52.356 INFO:teuthology.orchestra.run.smithi110.stdout:Get:39 https://3.chacra.ceph.com/r/ceph/wip-yuri-testing-2023-10-05-0730-quincy/736ac5ba72cb47af7d6f6666b1e100f7e20a20e2/ubuntu/focal/flavors/default focal/main amd64 libsqlite3-mod-ceph amd64 17.2.6-1296-g736ac5ba-1focal [152 kB] 2023-10-06T03:25:52.361 INFO:teuthology.orchestra.run.smithi110.stdout:Get:40 https://3.chacra.ceph.com/r/ceph/wip-yuri-testing-2023-10-05-0730-quincy/736ac5ba72cb47af7d6f6666b1e100f7e20a20e2/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr amd64 17.2.6-1296-g736ac5ba-1focal [1,591 kB] 2023-10-06T03:25:52.417 INFO:teuthology.orchestra.run.smithi110.stdout:Get:41 https://3.chacra.ceph.com/r/ceph/wip-yuri-testing-2023-10-05-0730-quincy/736ac5ba72cb47af7d6f6666b1e100f7e20a20e2/ubuntu/focal/flavors/default focal/main amd64 ceph-mon amd64 17.2.6-1296-g736ac5ba-1focal [7,846 kB] 2023-10-06T03:25:52.468 INFO:teuthology.orchestra.run.smithi110.stdout:Get:42 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2023-10-06T03:25:52.487 INFO:teuthology.orchestra.run.smithi110.stdout:Get:43 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-singledispatch all 3.4.0.3-2build1 [7,128 B] 2023-10-06T03:25:52.498 INFO:teuthology.orchestra.run.smithi110.stdout:Get:44 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webob all 1:1.8.5-2 [84.4 kB] 2023-10-06T03:25:52.575 INFO:teuthology.orchestra.run.smithi110.stdout:Get:45 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-soupsieve all 1.9.5+dfsg-1 [29.1 kB] 2023-10-06T03:25:52.597 INFO:teuthology.orchestra.run.smithi110.stdout:Get:46 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-bs4 all 4.8.2-1 [83.0 kB] 2023-10-06T03:25:52.637 INFO:teuthology.orchestra.run.smithi110.stdout:Get:47 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-waitress all 1.4.1-1ubuntu0.1 [81.4 kB] 2023-10-06T03:25:52.667 INFO:teuthology.orchestra.run.smithi110.stdout:Get:48 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-tempita all 0.5.2-6 [14.8 kB] 2023-10-06T03:25:52.671 INFO:teuthology.orchestra.run.smithi110.stdout:Get:49 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-paste all 3.4.0+dfsg1-1 [456 kB] 2023-10-06T03:25:52.693 INFO:teuthology.orchestra.run.smithi110.stdout:Get:50 https://3.chacra.ceph.com/r/ceph/wip-yuri-testing-2023-10-05-0730-quincy/736ac5ba72cb47af7d6f6666b1e100f7e20a20e2/ubuntu/focal/flavors/default focal/main amd64 ceph-osd amd64 17.2.6-1296-g736ac5ba-1focal [28.7 MB] 2023-10-06T03:25:52.766 INFO:teuthology.orchestra.run.smithi110.stdout:Get:51 http://archive.ubuntu.com/ubuntu focal/main amd64 python-pastedeploy-tpl all 2.1.0-1 [5,016 B] 2023-10-06T03:25:52.930 INFO:teuthology.orchestra.run.smithi110.stdout:Get:52 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pastedeploy all 2.1.0-1 [26.5 kB] 2023-10-06T03:25:53.115 INFO:teuthology.orchestra.run.smithi110.stdout:Get:53 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webtest all 2.0.32-2 [28.2 kB] 2023-10-06T03:25:53.164 INFO:teuthology.orchestra.run.smithi110.stdout:Get:54 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pecan all 1.3.3-3ubuntu1 [85.8 kB] 2023-10-06T03:25:53.295 INFO:teuthology.orchestra.run.smithi110.stdout:Get:55 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-werkzeug all 0.16.1+dfsg1-2ubuntu0.1 [183 kB] 2023-10-06T03:25:53.399 INFO:teuthology.orchestra.run.smithi110.stdout:Get:56 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-jinja2 all 2.10.1-2 [95.5 kB] 2023-10-06T03:25:53.428 INFO:teuthology.orchestra.run.smithi110.stdout:Get:57 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2023-10-06T03:25:53.431 INFO:teuthology.orchestra.run.smithi110.stdout:Get:58 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libjs-underscore all 1.9.1~dfsg-1ubuntu0.20.04.1 [99.5 kB] 2023-10-06T03:25:53.455 INFO:teuthology.orchestra.run.smithi110.stdout:Get:59 http://archive.ubuntu.com/ubuntu focal/main amd64 libjs-sphinxdoc all 1.8.5-7ubuntu3 [97.1 kB] 2023-10-06T03:25:53.476 INFO:teuthology.orchestra.run.smithi110.stdout:Get:60 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-routes all 2.4.1-2 [88.1 kB] 2023-10-06T03:25:53.492 INFO:teuthology.orchestra.run.smithi110.stdout:Get:61 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-decorator all 4.4.2-0ubuntu1 [10.3 kB] 2023-10-06T03:25:53.494 INFO:teuthology.orchestra.run.smithi110.stdout:Get:62 http://archive.ubuntu.com/ubuntu focal/universe amd64 liblbfgsb0 amd64 3.0+dfsg.3-7build1 [29.5 kB] 2023-10-06T03:25:53.635 INFO:teuthology.orchestra.run.smithi192.stdout:Get:85 https://3.chacra.ceph.com/r/ceph/wip-yuri-testing-2023-10-05-0730-quincy/736ac5ba72cb47af7d6f6666b1e100f7e20a20e2/ubuntu/focal/flavors/default focal/main amd64 libcephfs2 amd64 17.2.6-1296-g736ac5ba-1focal [760 kB] 2023-10-06T03:25:53.663 INFO:teuthology.orchestra.run.smithi110.stdout:Get:63 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-scipy amd64 1.3.3-3build1 [11.1 MB] 2023-10-06T03:25:53.903 INFO:teuthology.orchestra.run.smithi110.stdout:Get:64 https://3.chacra.ceph.com/r/ceph/wip-yuri-testing-2023-10-05-0730-quincy/736ac5ba72cb47af7d6f6666b1e100f7e20a20e2/ubuntu/focal/flavors/default focal/main amd64 ceph amd64 17.2.6-1296-g736ac5ba-1focal [3,920 B] 2023-10-06T03:25:53.976 INFO:teuthology.orchestra.run.smithi110.stdout:Get:65 https://3.chacra.ceph.com/r/ceph/wip-yuri-testing-2023-10-05-0730-quincy/736ac5ba72cb47af7d6f6666b1e100f7e20a20e2/ubuntu/focal/flavors/default focal/main amd64 ceph-fuse amd64 17.2.6-1296-g736ac5ba-1focal [869 kB] 2023-10-06T03:25:53.989 INFO:teuthology.orchestra.run.smithi110.stdout:Get:66 https://3.chacra.ceph.com/r/ceph/wip-yuri-testing-2023-10-05-0730-quincy/736ac5ba72cb47af7d6f6666b1e100f7e20a20e2/ubuntu/focal/flavors/default focal/main amd64 ceph-mds amd64 17.2.6-1296-g736ac5ba-1focal [2,408 kB] 2023-10-06T03:25:54.006 INFO:teuthology.orchestra.run.smithi110.stdout:Get:67 https://3.chacra.ceph.com/r/ceph/wip-yuri-testing-2023-10-05-0730-quincy/736ac5ba72cb47af7d6f6666b1e100f7e20a20e2/ubuntu/focal/flavors/default focal/main amd64 cephadm amd64 17.2.6-1296-g736ac5ba-1focal [79.1 kB] 2023-10-06T03:25:54.007 INFO:teuthology.orchestra.run.smithi110.stdout:Get:68 https://3.chacra.ceph.com/r/ceph/wip-yuri-testing-2023-10-05-0730-quincy/736ac5ba72cb47af7d6f6666b1e100f7e20a20e2/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-cephadm all 17.2.6-1296-g736ac5ba-1focal [117 kB] 2023-10-06T03:25:54.007 INFO:teuthology.orchestra.run.smithi110.stdout:Get:69 https://3.chacra.ceph.com/r/ceph/wip-yuri-testing-2023-10-05-0730-quincy/736ac5ba72cb47af7d6f6666b1e100f7e20a20e2/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-dashboard all 17.2.6-1296-g736ac5ba-1focal [1,760 kB] 2023-10-06T03:25:54.058 INFO:teuthology.orchestra.run.smithi110.stdout:Get:70 https://3.chacra.ceph.com/r/ceph/wip-yuri-testing-2023-10-05-0730-quincy/736ac5ba72cb47af7d6f6666b1e100f7e20a20e2/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-diskprediction-local all 17.2.6-1296-g736ac5ba-1focal [7,888 kB] 2023-10-06T03:25:54.316 INFO:teuthology.orchestra.run.smithi110.stdout:Get:71 https://3.chacra.ceph.com/r/ceph/wip-yuri-testing-2023-10-05-0730-quincy/736ac5ba72cb47af7d6f6666b1e100f7e20a20e2/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-k8sevents all 17.2.6-1296-g736ac5ba-1focal [4,032 B] 2023-10-06T03:25:54.317 INFO:teuthology.orchestra.run.smithi110.stdout:Get:72 https://3.chacra.ceph.com/r/ceph/wip-yuri-testing-2023-10-05-0730-quincy/736ac5ba72cb47af7d6f6666b1e100f7e20a20e2/ubuntu/focal/flavors/default focal/main amd64 ceph-test amd64 17.2.6-1296-g736ac5ba-1focal [56.2 MB] 2023-10-06T03:25:54.698 INFO:teuthology.orchestra.run.smithi110.stdout:Get:73 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-sklearn-lib amd64 0.22.2.post1+dfsg-5 [2,358 kB] 2023-10-06T03:25:54.721 INFO:teuthology.orchestra.run.smithi110.stdout:Get:74 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-joblib all 0.14.0-3 [192 kB] 2023-10-06T03:25:54.723 INFO:teuthology.orchestra.run.smithi110.stdout:Get:75 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-sklearn all 0.22.2.post1+dfsg-5 [1,773 kB] 2023-10-06T03:25:54.795 INFO:teuthology.orchestra.run.smithi192.stdout:Get:86 https://3.chacra.ceph.com/r/ceph/wip-yuri-testing-2023-10-05-0730-quincy/736ac5ba72cb47af7d6f6666b1e100f7e20a20e2/ubuntu/focal/flavors/default focal/main amd64 python3-rados amd64 17.2.6-1296-g736ac5ba-1focal [413 kB] 2023-10-06T03:25:54.797 INFO:teuthology.orchestra.run.smithi110.stdout:Get:76 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-cachetools all 4.0.0-1 [9,820 B] 2023-10-06T03:25:54.797 INFO:teuthology.orchestra.run.smithi110.stdout:Get:77 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyasn1 all 0.4.2-3build1 [46.7 kB] 2023-10-06T03:25:54.798 INFO:teuthology.orchestra.run.smithi110.stdout:Get:78 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyasn1-modules all 0.2.1-0.2build1 [32.9 kB] 2023-10-06T03:25:54.798 INFO:teuthology.orchestra.run.smithi110.stdout:Get:79 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-rsa all 4.0-3 [24.7 kB] 2023-10-06T03:25:54.798 INFO:teuthology.orchestra.run.smithi110.stdout:Get:80 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-google-auth all 1.5.1-2 [35.6 kB] 2023-10-06T03:25:54.799 INFO:teuthology.orchestra.run.smithi110.stdout:Get:81 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-blinker all 1.4+dfsg1-0.3ubuntu1 [13.2 kB] 2023-10-06T03:25:54.800 INFO:teuthology.orchestra.run.smithi110.stdout:Get:82 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-oauthlib all 3.1.0-1ubuntu2 [84.8 kB] 2023-10-06T03:25:54.964 INFO:teuthology.orchestra.run.smithi110.stdout:Get:83 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-requests-oauthlib all 1.0.0-1.1build1 [16.8 kB] 2023-10-06T03:25:55.088 INFO:teuthology.orchestra.run.smithi192.stdout:Get:87 https://3.chacra.ceph.com/r/ceph/wip-yuri-testing-2023-10-05-0730-quincy/736ac5ba72cb47af7d6f6666b1e100f7e20a20e2/ubuntu/focal/flavors/default focal/main amd64 python3-ceph-argparse all 17.2.6-1296-g736ac5ba-1focal [22.5 kB] 2023-10-06T03:25:55.089 INFO:teuthology.orchestra.run.smithi192.stdout:Get:88 https://3.chacra.ceph.com/r/ceph/wip-yuri-testing-2023-10-05-0730-quincy/736ac5ba72cb47af7d6f6666b1e100f7e20a20e2/ubuntu/focal/flavors/default focal/main amd64 python3-cephfs amd64 17.2.6-1296-g736ac5ba-1focal [200 kB] 2023-10-06T03:25:55.131 INFO:teuthology.orchestra.run.smithi110.stdout:Get:84 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-websocket all 0.53.0-2ubuntu1 [32.3 kB] 2023-10-06T03:25:55.166 INFO:teuthology.orchestra.run.smithi192.stdout:Get:89 https://3.chacra.ceph.com/r/ceph/wip-yuri-testing-2023-10-05-0730-quincy/736ac5ba72cb47af7d6f6666b1e100f7e20a20e2/ubuntu/focal/flavors/default focal/main amd64 python3-ceph-common all 17.2.6-1296-g736ac5ba-1focal [47.1 kB] 2023-10-06T03:25:55.167 INFO:teuthology.orchestra.run.smithi192.stdout:Get:90 https://3.chacra.ceph.com/r/ceph/wip-yuri-testing-2023-10-05-0730-quincy/736ac5ba72cb47af7d6f6666b1e100f7e20a20e2/ubuntu/focal/flavors/default focal/main amd64 python3-rbd amd64 17.2.6-1296-g736ac5ba-1focal [386 kB] 2023-10-06T03:25:55.213 INFO:teuthology.orchestra.run.smithi110.stdout:Get:85 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-kubernetes all 7.0.0~a1-2 [271 kB] 2023-10-06T03:25:55.391 INFO:teuthology.orchestra.run.smithi192.stdout:Get:91 https://3.chacra.ceph.com/r/ceph/wip-yuri-testing-2023-10-05-0730-quincy/736ac5ba72cb47af7d6f6666b1e100f7e20a20e2/ubuntu/focal/flavors/default focal/main amd64 librgw2 amd64 17.2.6-1296-g736ac5ba-1focal [5,570 kB] 2023-10-06T03:25:55.444 INFO:teuthology.orchestra.run.smithi110.stdout:Get:86 http://archive.ubuntu.com/ubuntu focal/universe amd64 libonig5 amd64 6.9.4-1 [142 kB] 2023-10-06T03:25:55.488 INFO:teuthology.orchestra.run.smithi110.stdout:Get:87 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libjq1 amd64 1.6-1ubuntu0.20.04.1 [121 kB] 2023-10-06T03:25:55.517 INFO:teuthology.orchestra.run.smithi110.stdout:Get:88 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 jq amd64 1.6-1ubuntu0.20.04.1 [50.2 kB] 2023-10-06T03:25:55.527 INFO:teuthology.orchestra.run.smithi110.stdout:Get:89 http://archive.ubuntu.com/ubuntu focal/main amd64 socat amd64 1.7.3.3-2 [323 kB] 2023-10-06T03:25:55.577 INFO:teuthology.orchestra.run.smithi110.stdout:Get:90 http://archive.ubuntu.com/ubuntu focal/universe amd64 xmlstarlet amd64 1.6.1-2build1 [267 kB] 2023-10-06T03:25:55.603 INFO:teuthology.orchestra.run.smithi110.stdout:Get:91 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-colorama all 0.4.3-1build1 [23.9 kB] 2023-10-06T03:25:55.606 INFO:teuthology.orchestra.run.smithi110.stdout:Get:92 http://archive.ubuntu.com/ubuntu focal/universe amd64 fonts-mathjax all 2.7.4+dfsg-1 [2,208 kB] 2023-10-06T03:25:55.726 INFO:teuthology.orchestra.run.smithi110.stdout:Get:93 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 nvme-cli amd64 1.9-1ubuntu0.1 [283 kB] 2023-10-06T03:25:55.898 INFO:teuthology.orchestra.run.smithi110.stdout:Get:94 http://archive.ubuntu.com/ubuntu focal/universe amd64 libjs-mathjax all 2.7.4+dfsg-1 [5,654 kB] 2023-10-06T03:25:56.562 INFO:teuthology.orchestra.run.smithi110.stdout:Get:95 https://3.chacra.ceph.com/r/ceph/wip-yuri-testing-2023-10-05-0730-quincy/736ac5ba72cb47af7d6f6666b1e100f7e20a20e2/ubuntu/focal/flavors/default focal/main amd64 ceph-volume all 17.2.6-1296-g736ac5ba-1focal [229 kB] 2023-10-06T03:25:56.567 INFO:teuthology.orchestra.run.smithi110.stdout:Get:96 https://3.chacra.ceph.com/r/ceph/wip-yuri-testing-2023-10-05-0730-quincy/736ac5ba72cb47af7d6f6666b1e100f7e20a20e2/ubuntu/focal/flavors/default focal/main amd64 cephfs-mirror amd64 17.2.6-1296-g736ac5ba-1focal [240 kB] 2023-10-06T03:25:56.572 INFO:teuthology.orchestra.run.smithi110.stdout:Get:97 https://3.chacra.ceph.com/r/ceph/wip-yuri-testing-2023-10-05-0730-quincy/736ac5ba72cb47af7d6f6666b1e100f7e20a20e2/ubuntu/focal/flavors/default focal/main amd64 cephfs-shell all 17.2.6-1296-g736ac5ba-1focal [22.0 kB] 2023-10-06T03:25:56.597 INFO:teuthology.orchestra.run.smithi110.stdout:Get:98 https://3.chacra.ceph.com/r/ceph/wip-yuri-testing-2023-10-05-0730-quincy/736ac5ba72cb47af7d6f6666b1e100f7e20a20e2/ubuntu/focal/flavors/default focal/main amd64 cephfs-top all 17.2.6-1296-g736ac5ba-1focal [13.1 kB] 2023-10-06T03:25:56.597 INFO:teuthology.orchestra.run.smithi110.stdout:Get:99 https://3.chacra.ceph.com/r/ceph/wip-yuri-testing-2023-10-05-0730-quincy/736ac5ba72cb47af7d6f6666b1e100f7e20a20e2/ubuntu/focal/flavors/default focal/main amd64 libcephfs-dev amd64 17.2.6-1296-g736ac5ba-1focal [21.9 kB] 2023-10-06T03:25:56.597 INFO:teuthology.orchestra.run.smithi110.stdout:Get:100 https://3.chacra.ceph.com/r/ceph/wip-yuri-testing-2023-10-05-0730-quincy/736ac5ba72cb47af7d6f6666b1e100f7e20a20e2/ubuntu/focal/flavors/default focal/main amd64 radosgw amd64 17.2.6-1296-g736ac5ba-1focal [17.0 MB] 2023-10-06T03:25:56.723 INFO:teuthology.orchestra.run.smithi110.stdout:Get:101 http://archive.ubuntu.com/ubuntu focal/universe amd64 python-natsort-doc all 7.0.1-1 [103 kB] 2023-10-06T03:25:56.725 INFO:teuthology.orchestra.run.smithi110.stdout:Get:102 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-atomicwrites all 1.1.5-2build1 [7,016 B] 2023-10-06T03:25:56.725 INFO:teuthology.orchestra.run.smithi110.stdout:Get:103 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-attr all 19.3.0-2 [33.9 kB] 2023-10-06T03:25:56.726 INFO:teuthology.orchestra.run.smithi110.stdout:Get:104 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webencodings all 0.5.1-1ubuntu1 [11.0 kB] 2023-10-06T03:25:56.727 INFO:teuthology.orchestra.run.smithi110.stdout:Get:105 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-html5lib all 1.0.1-2 [84.3 kB] 2023-10-06T03:25:56.729 INFO:teuthology.orchestra.run.smithi110.stdout:Get:106 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-lxml amd64 4.5.0-1ubuntu0.5 [1,384 kB] 2023-10-06T03:25:56.746 INFO:teuthology.orchestra.run.smithi110.stdout:Get:107 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-packaging all 20.3-1 [26.8 kB] 2023-10-06T03:25:56.746 INFO:teuthology.orchestra.run.smithi110.stdout:Get:108 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2023-10-06T03:25:56.747 INFO:teuthology.orchestra.run.smithi110.stdout:Get:109 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-pluggy all 0.13.0-2 [18.4 kB] 2023-10-06T03:25:56.804 INFO:teuthology.orchestra.run.smithi110.stdout:Get:110 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-psutil amd64 5.5.1-1ubuntu4 [141 kB] 2023-10-06T03:25:56.971 INFO:teuthology.orchestra.run.smithi110.stdout:Get:111 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 python3-py all 1.8.1-1ubuntu0.1 [65.6 kB] 2023-10-06T03:25:57.140 INFO:teuthology.orchestra.run.smithi110.stdout:Get:112 https://3.chacra.ceph.com/r/ceph/wip-yuri-testing-2023-10-05-0730-quincy/736ac5ba72cb47af7d6f6666b1e100f7e20a20e2/ubuntu/focal/flavors/default focal/main amd64 rbd-fuse amd64 17.2.6-1296-g736ac5ba-1focal [79.5 kB] 2023-10-06T03:25:57.200 INFO:teuthology.orchestra.run.smithi192.stdout:Get:92 https://3.chacra.ceph.com/r/ceph/wip-yuri-testing-2023-10-05-0730-quincy/736ac5ba72cb47af7d6f6666b1e100f7e20a20e2/ubuntu/focal/flavors/default focal/main amd64 python3-rgw amd64 17.2.6-1296-g736ac5ba-1focal [116 kB] 2023-10-06T03:25:57.206 INFO:teuthology.orchestra.run.smithi192.stdout:Get:93 https://3.chacra.ceph.com/r/ceph/wip-yuri-testing-2023-10-05-0730-quincy/736ac5ba72cb47af7d6f6666b1e100f7e20a20e2/ubuntu/focal/flavors/default focal/main amd64 libradosstriper1 amd64 17.2.6-1296-g736ac5ba-1focal [479 kB] 2023-10-06T03:25:57.256 INFO:teuthology.orchestra.run.smithi110.stdout:Get:113 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyinotify all 0.9.6-1.2ubuntu1 [24.8 kB] 2023-10-06T03:25:57.278 INFO:teuthology.orchestra.run.smithi110.stdout:Get:114 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-pytest all 4.6.9-1 [178 kB] 2023-10-06T03:25:57.287 INFO:teuthology.orchestra.run.smithi192.stdout:Get:94 https://3.chacra.ceph.com/r/ceph/wip-yuri-testing-2023-10-05-0730-quincy/736ac5ba72cb47af7d6f6666b1e100f7e20a20e2/ubuntu/focal/flavors/default focal/main amd64 ceph-common amd64 17.2.6-1296-g736ac5ba-1focal [27.8 MB] 2023-10-06T03:25:57.428 INFO:teuthology.orchestra.run.smithi110.stdout:Get:115 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-simplejson amd64 3.16.0-2ubuntu2 [50.8 kB] 2023-10-06T03:25:57.445 INFO:teuthology.orchestra.run.smithi110.stdout:Get:116 http://archive.ubuntu.com/ubuntu focal/universe amd64 qttranslations5-l10n all 5.12.8-0ubuntu1 [1,486 kB] 2023-10-06T03:25:58.286 INFO:teuthology.orchestra.run.smithi110.stdout:Fetched 210 MB in 7s (28.3 MB/s) 2023-10-06T03:25:58.465 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-pyperclip. 2023-10-06T03:25:58.522 INFO:teuthology.orchestra.run.smithi110.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 141736 files and directories currently installed.) 2023-10-06T03:25:58.526 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../000-python3-pyperclip_1.7.0-1_all.deb ... 2023-10-06T03:25:58.571 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-pyperclip (1.7.0-1) ... 2023-10-06T03:25:58.874 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-wcwidth. 2023-10-06T03:25:58.893 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../001-python3-wcwidth_0.1.8+dfsg1-3_all.deb ... 2023-10-06T03:25:58.933 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-wcwidth (0.1.8+dfsg1-3) ... 2023-10-06T03:25:59.235 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-cmd2. 2023-10-06T03:25:59.255 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../002-python3-cmd2_0.8.5-2.1_all.deb ... 2023-10-06T03:25:59.310 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-cmd2 (0.8.5-2.1) ... 2023-10-06T03:25:59.646 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package liblttng-ust0:amd64. 2023-10-06T03:25:59.665 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../003-liblttng-ust0_2.11.0-1_amd64.deb ... 2023-10-06T03:25:59.747 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking liblttng-ust0:amd64 (2.11.0-1) ... 2023-10-06T03:26:00.075 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../004-librbd1_17.2.6-1296-g736ac5ba-1focal_amd64.deb ... 2023-10-06T03:26:00.234 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking librbd1 (17.2.6-1296-g736ac5ba-1focal) over (15.2.17-0ubuntu0.20.04.4) ... 2023-10-06T03:26:00.508 INFO:teuthology.orchestra.run.smithi192.stdout:Get:95 https://3.chacra.ceph.com/r/ceph/wip-yuri-testing-2023-10-05-0730-quincy/736ac5ba72cb47af7d6f6666b1e100f7e20a20e2/ubuntu/focal/flavors/default focal/main amd64 ceph-base amd64 17.2.6-1296-g736ac5ba-1focal [6,453 kB] 2023-10-06T03:26:00.897 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../005-librados2_17.2.6-1296-g736ac5ba-1focal_amd64.deb ... 2023-10-06T03:26:01.007 INFO:teuthology.orchestra.run.smithi192.stdout:Get:96 https://3.chacra.ceph.com/r/ceph/wip-yuri-testing-2023-10-05-0730-quincy/736ac5ba72cb47af7d6f6666b1e100f7e20a20e2/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-modules-core all 17.2.6-1296-g736ac5ba-1focal [221 kB] 2023-10-06T03:26:01.022 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking librados2 (17.2.6-1296-g736ac5ba-1focal) over (15.2.17-0ubuntu0.20.04.4) ... 2023-10-06T03:26:01.024 INFO:teuthology.orchestra.run.smithi192.stdout:Get:97 https://3.chacra.ceph.com/r/ceph/wip-yuri-testing-2023-10-05-0730-quincy/736ac5ba72cb47af7d6f6666b1e100f7e20a20e2/ubuntu/focal/flavors/default focal/main amd64 libsqlite3-mod-ceph amd64 17.2.6-1296-g736ac5ba-1focal [152 kB] 2023-10-06T03:26:01.044 INFO:teuthology.orchestra.run.smithi192.stdout:Get:98 https://3.chacra.ceph.com/r/ceph/wip-yuri-testing-2023-10-05-0730-quincy/736ac5ba72cb47af7d6f6666b1e100f7e20a20e2/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr amd64 17.2.6-1296-g736ac5ba-1focal [1,591 kB] 2023-10-06T03:26:01.168 INFO:teuthology.orchestra.run.smithi192.stdout:Get:99 https://3.chacra.ceph.com/r/ceph/wip-yuri-testing-2023-10-05-0730-quincy/736ac5ba72cb47af7d6f6666b1e100f7e20a20e2/ubuntu/focal/flavors/default focal/main amd64 ceph-mon amd64 17.2.6-1296-g736ac5ba-1focal [7,846 kB] 2023-10-06T03:26:01.642 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package libcephfs2. 2023-10-06T03:26:01.660 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../006-libcephfs2_17.2.6-1296-g736ac5ba-1focal_amd64.deb ... 2023-10-06T03:26:01.692 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking libcephfs2 (17.2.6-1296-g736ac5ba-1focal) ... 2023-10-06T03:26:01.765 INFO:teuthology.orchestra.run.smithi192.stdout:Get:100 https://3.chacra.ceph.com/r/ceph/wip-yuri-testing-2023-10-05-0730-quincy/736ac5ba72cb47af7d6f6666b1e100f7e20a20e2/ubuntu/focal/flavors/default focal/main amd64 ceph-osd amd64 17.2.6-1296-g736ac5ba-1focal [28.7 MB] 2023-10-06T03:26:02.053 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-rados. 2023-10-06T03:26:02.072 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../007-python3-rados_17.2.6-1296-g736ac5ba-1focal_amd64.deb ... 2023-10-06T03:26:02.111 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-rados (17.2.6-1296-g736ac5ba-1focal) ... 2023-10-06T03:26:02.441 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-ceph-argparse. 2023-10-06T03:26:02.460 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../008-python3-ceph-argparse_17.2.6-1296-g736ac5ba-1focal_all.deb ... 2023-10-06T03:26:02.499 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-ceph-argparse (17.2.6-1296-g736ac5ba-1focal) ... 2023-10-06T03:26:02.685 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-cephfs. 2023-10-06T03:26:02.701 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../009-python3-cephfs_17.2.6-1296-g736ac5ba-1focal_amd64.deb ... 2023-10-06T03:26:02.734 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-cephfs (17.2.6-1296-g736ac5ba-1focal) ... 2023-10-06T03:26:03.088 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-ceph-common. 2023-10-06T03:26:03.107 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../010-python3-ceph-common_17.2.6-1296-g736ac5ba-1focal_all.deb ... 2023-10-06T03:26:03.146 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-ceph-common (17.2.6-1296-g736ac5ba-1focal) ... 2023-10-06T03:26:03.466 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-prettytable. 2023-10-06T03:26:03.484 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../011-python3-prettytable_0.7.2-5_all.deb ... 2023-10-06T03:26:03.524 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-prettytable (0.7.2-5) ... 2023-10-06T03:26:03.744 INFO:teuthology.orchestra.run.smithi192.stdout:Get:101 https://3.chacra.ceph.com/r/ceph/wip-yuri-testing-2023-10-05-0730-quincy/736ac5ba72cb47af7d6f6666b1e100f7e20a20e2/ubuntu/focal/flavors/default focal/main amd64 ceph amd64 17.2.6-1296-g736ac5ba-1focal [3,920 B] 2023-10-06T03:26:03.762 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-rbd. 2023-10-06T03:26:03.779 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../012-python3-rbd_17.2.6-1296-g736ac5ba-1focal_amd64.deb ... 2023-10-06T03:26:03.822 INFO:teuthology.orchestra.run.smithi192.stdout:Get:102 https://3.chacra.ceph.com/r/ceph/wip-yuri-testing-2023-10-05-0730-quincy/736ac5ba72cb47af7d6f6666b1e100f7e20a20e2/ubuntu/focal/flavors/default focal/main amd64 ceph-fuse amd64 17.2.6-1296-g736ac5ba-1focal [869 kB] 2023-10-06T03:26:03.839 INFO:teuthology.orchestra.run.smithi192.stdout:Get:103 https://3.chacra.ceph.com/r/ceph/wip-yuri-testing-2023-10-05-0730-quincy/736ac5ba72cb47af7d6f6666b1e100f7e20a20e2/ubuntu/focal/flavors/default focal/main amd64 ceph-mds amd64 17.2.6-1296-g736ac5ba-1focal [2,408 kB] 2023-10-06T03:26:03.845 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-rbd (17.2.6-1296-g736ac5ba-1focal) ... 2023-10-06T03:26:03.951 INFO:teuthology.orchestra.run.smithi192.stdout:Get:104 https://3.chacra.ceph.com/r/ceph/wip-yuri-testing-2023-10-05-0730-quincy/736ac5ba72cb47af7d6f6666b1e100f7e20a20e2/ubuntu/focal/flavors/default focal/main amd64 cephadm amd64 17.2.6-1296-g736ac5ba-1focal [79.1 kB] 2023-10-06T03:26:03.953 INFO:teuthology.orchestra.run.smithi192.stdout:Get:105 https://3.chacra.ceph.com/r/ceph/wip-yuri-testing-2023-10-05-0730-quincy/736ac5ba72cb47af7d6f6666b1e100f7e20a20e2/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-cephadm all 17.2.6-1296-g736ac5ba-1focal [117 kB] 2023-10-06T03:26:03.956 INFO:teuthology.orchestra.run.smithi192.stdout:Get:106 https://3.chacra.ceph.com/r/ceph/wip-yuri-testing-2023-10-05-0730-quincy/736ac5ba72cb47af7d6f6666b1e100f7e20a20e2/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-dashboard all 17.2.6-1296-g736ac5ba-1focal [1,760 kB] 2023-10-06T03:26:04.057 INFO:teuthology.orchestra.run.smithi192.stdout:Get:107 https://3.chacra.ceph.com/r/ceph/wip-yuri-testing-2023-10-05-0730-quincy/736ac5ba72cb47af7d6f6666b1e100f7e20a20e2/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-diskprediction-local all 17.2.6-1296-g736ac5ba-1focal [7,888 kB] 2023-10-06T03:26:04.199 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package liblua5.3-0:amd64. 2023-10-06T03:26:04.216 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../013-liblua5.3-0_5.3.3-1.1ubuntu2_amd64.deb ... 2023-10-06T03:26:04.257 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking liblua5.3-0:amd64 (5.3.3-1.1ubuntu2) ... 2023-10-06T03:26:04.509 INFO:teuthology.orchestra.run.smithi192.stdout:Get:108 https://3.chacra.ceph.com/r/ceph/wip-yuri-testing-2023-10-05-0730-quincy/736ac5ba72cb47af7d6f6666b1e100f7e20a20e2/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-k8sevents all 17.2.6-1296-g736ac5ba-1focal [4,032 B] 2023-10-06T03:26:04.509 INFO:teuthology.orchestra.run.smithi192.stdout:Get:109 https://3.chacra.ceph.com/r/ceph/wip-yuri-testing-2023-10-05-0730-quincy/736ac5ba72cb47af7d6f6666b1e100f7e20a20e2/ubuntu/focal/flavors/default focal/main amd64 ceph-test amd64 17.2.6-1296-g736ac5ba-1focal [56.2 MB] 2023-10-06T03:26:04.610 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package librabbitmq4:amd64. 2023-10-06T03:26:04.628 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../014-librabbitmq4_0.10.0-1_amd64.deb ... 2023-10-06T03:26:04.661 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking librabbitmq4:amd64 (0.10.0-1) ... 2023-10-06T03:26:04.986 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package librdkafka1:amd64. 2023-10-06T03:26:05.002 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../015-librdkafka1_1.2.1-1ubuntu1_amd64.deb ... 2023-10-06T03:26:05.036 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking librdkafka1:amd64 (1.2.1-1ubuntu1) ... 2023-10-06T03:26:05.339 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2023-10-06T03:26:05.358 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../016-libdouble-conversion3_3.1.5-4ubuntu1_amd64.deb ... 2023-10-06T03:26:05.397 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking libdouble-conversion3:amd64 (3.1.5-4ubuntu1) ... 2023-10-06T03:26:05.776 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2023-10-06T03:26:05.796 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../017-libpcre2-16-0_10.34-7ubuntu0.1_amd64.deb ... 2023-10-06T03:26:05.834 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking libpcre2-16-0:amd64 (10.34-7ubuntu0.1) ... 2023-10-06T03:26:06.229 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package libqt5core5a:amd64. 2023-10-06T03:26:06.248 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../018-libqt5core5a_5.12.8+dfsg-0ubuntu2.1_amd64.deb ... 2023-10-06T03:26:06.289 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking libqt5core5a:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2023-10-06T03:26:06.799 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2023-10-06T03:26:06.819 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../019-libqt5dbus5_5.12.8+dfsg-0ubuntu2.1_amd64.deb ... 2023-10-06T03:26:06.858 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking libqt5dbus5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2023-10-06T03:26:07.211 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package libqt5network5:amd64. 2023-10-06T03:26:07.231 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../020-libqt5network5_5.12.8+dfsg-0ubuntu2.1_amd64.deb ... 2023-10-06T03:26:07.270 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking libqt5network5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2023-10-06T03:26:07.510 INFO:teuthology.orchestra.run.smithi192.stdout:Get:110 https://3.chacra.ceph.com/r/ceph/wip-yuri-testing-2023-10-05-0730-quincy/736ac5ba72cb47af7d6f6666b1e100f7e20a20e2/ubuntu/focal/flavors/default focal/main amd64 ceph-volume all 17.2.6-1296-g736ac5ba-1focal [229 kB] 2023-10-06T03:26:07.515 INFO:teuthology.orchestra.run.smithi192.stdout:Get:111 https://3.chacra.ceph.com/r/ceph/wip-yuri-testing-2023-10-05-0730-quincy/736ac5ba72cb47af7d6f6666b1e100f7e20a20e2/ubuntu/focal/flavors/default focal/main amd64 cephfs-mirror amd64 17.2.6-1296-g736ac5ba-1focal [240 kB] 2023-10-06T03:26:07.520 INFO:teuthology.orchestra.run.smithi192.stdout:Get:112 https://3.chacra.ceph.com/r/ceph/wip-yuri-testing-2023-10-05-0730-quincy/736ac5ba72cb47af7d6f6666b1e100f7e20a20e2/ubuntu/focal/flavors/default focal/main amd64 cephfs-shell all 17.2.6-1296-g736ac5ba-1focal [22.0 kB] 2023-10-06T03:26:07.610 INFO:teuthology.orchestra.run.smithi192.stdout:Get:113 https://3.chacra.ceph.com/r/ceph/wip-yuri-testing-2023-10-05-0730-quincy/736ac5ba72cb47af7d6f6666b1e100f7e20a20e2/ubuntu/focal/flavors/default focal/main amd64 cephfs-top all 17.2.6-1296-g736ac5ba-1focal [13.1 kB] 2023-10-06T03:26:07.610 INFO:teuthology.orchestra.run.smithi192.stdout:Get:114 https://3.chacra.ceph.com/r/ceph/wip-yuri-testing-2023-10-05-0730-quincy/736ac5ba72cb47af7d6f6666b1e100f7e20a20e2/ubuntu/focal/flavors/default focal/main amd64 libcephfs-dev amd64 17.2.6-1296-g736ac5ba-1focal [21.9 kB] 2023-10-06T03:26:07.611 INFO:teuthology.orchestra.run.smithi192.stdout:Get:115 https://3.chacra.ceph.com/r/ceph/wip-yuri-testing-2023-10-05-0730-quincy/736ac5ba72cb47af7d6f6666b1e100f7e20a20e2/ubuntu/focal/flavors/default focal/main amd64 radosgw amd64 17.2.6-1296-g736ac5ba-1focal [17.0 MB] 2023-10-06T03:26:07.614 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package libthrift-0.13.0. 2023-10-06T03:26:07.630 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../021-libthrift-0.13.0_0.13.0-2build2_amd64.deb ... 2023-10-06T03:26:07.700 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking libthrift-0.13.0 (0.13.0-2build2) ... 2023-10-06T03:26:08.168 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package librgw2. 2023-10-06T03:26:08.187 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../022-librgw2_17.2.6-1296-g736ac5ba-1focal_amd64.deb ... 2023-10-06T03:26:08.226 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking librgw2 (17.2.6-1296-g736ac5ba-1focal) ... 2023-10-06T03:26:08.292 INFO:teuthology.orchestra.run.smithi192.stdout:Get:116 https://3.chacra.ceph.com/r/ceph/wip-yuri-testing-2023-10-05-0730-quincy/736ac5ba72cb47af7d6f6666b1e100f7e20a20e2/ubuntu/focal/flavors/default focal/main amd64 rbd-fuse amd64 17.2.6-1296-g736ac5ba-1focal [79.5 kB] 2023-10-06T03:26:08.907 INFO:teuthology.orchestra.run.smithi192.stdout:Fetched 210 MB in 18s (11.7 MB/s) 2023-10-06T03:26:09.004 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-rgw. 2023-10-06T03:26:09.021 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../023-python3-rgw_17.2.6-1296-g736ac5ba-1focal_amd64.deb ... 2023-10-06T03:26:09.056 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package python3-pyperclip. 2023-10-06T03:26:09.062 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-rgw (17.2.6-1296-g736ac5ba-1focal) ... 2023-10-06T03:26:09.112 INFO:teuthology.orchestra.run.smithi192.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 141736 files and directories currently installed.) 2023-10-06T03:26:09.116 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../000-python3-pyperclip_1.7.0-1_all.deb ... 2023-10-06T03:26:09.163 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking python3-pyperclip (1.7.0-1) ... 2023-10-06T03:26:09.448 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package liboath0. 2023-10-06T03:26:09.448 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package python3-wcwidth. 2023-10-06T03:26:09.465 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../024-liboath0_2.6.1-1.3_amd64.deb ... 2023-10-06T03:26:09.465 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../001-python3-wcwidth_0.1.8+dfsg1-3_all.deb ... 2023-10-06T03:26:09.499 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking python3-wcwidth (0.1.8+dfsg1-3) ... 2023-10-06T03:26:09.506 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking liboath0 (2.6.1-1.3) ... 2023-10-06T03:26:09.793 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package libradosstriper1. 2023-10-06T03:26:09.801 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package python3-cmd2. 2023-10-06T03:26:09.809 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../025-libradosstriper1_17.2.6-1296-g736ac5ba-1focal_amd64.deb ... 2023-10-06T03:26:09.817 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../002-python3-cmd2_0.8.5-2.1_all.deb ... 2023-10-06T03:26:09.842 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking libradosstriper1 (17.2.6-1296-g736ac5ba-1focal) ... 2023-10-06T03:26:09.860 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking python3-cmd2 (0.8.5-2.1) ... 2023-10-06T03:26:10.237 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package liblttng-ust0:amd64. 2023-10-06T03:26:10.253 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../003-liblttng-ust0_2.11.0-1_amd64.deb ... 2023-10-06T03:26:10.313 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package ceph-common. 2023-10-06T03:26:10.329 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../026-ceph-common_17.2.6-1296-g736ac5ba-1focal_amd64.deb ... 2023-10-06T03:26:10.363 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking ceph-common (17.2.6-1296-g736ac5ba-1focal) ... 2023-10-06T03:26:10.363 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking liblttng-ust0:amd64 (2.11.0-1) ... 2023-10-06T03:26:10.732 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../004-librbd1_17.2.6-1296-g736ac5ba-1focal_amd64.deb ... 2023-10-06T03:26:10.858 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking librbd1 (17.2.6-1296-g736ac5ba-1focal) over (15.2.17-0ubuntu0.20.04.4) ... 2023-10-06T03:26:11.594 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../005-librados2_17.2.6-1296-g736ac5ba-1focal_amd64.deb ... 2023-10-06T03:26:11.704 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking librados2 (17.2.6-1296-g736ac5ba-1focal) over (15.2.17-0ubuntu0.20.04.4) ... 2023-10-06T03:26:12.383 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package libcephfs2. 2023-10-06T03:26:12.398 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../006-libcephfs2_17.2.6-1296-g736ac5ba-1focal_amd64.deb ... 2023-10-06T03:26:12.466 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking libcephfs2 (17.2.6-1296-g736ac5ba-1focal) ... 2023-10-06T03:26:12.860 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package python3-rados. 2023-10-06T03:26:12.879 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../007-python3-rados_17.2.6-1296-g736ac5ba-1focal_amd64.deb ... 2023-10-06T03:26:12.910 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking python3-rados (17.2.6-1296-g736ac5ba-1focal) ... 2023-10-06T03:26:13.061 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package ceph-base. 2023-10-06T03:26:13.080 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../027-ceph-base_17.2.6-1296-g736ac5ba-1focal_amd64.deb ... 2023-10-06T03:26:13.130 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking ceph-base (17.2.6-1296-g736ac5ba-1focal) ... 2023-10-06T03:26:13.246 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package python3-ceph-argparse. 2023-10-06T03:26:13.264 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../008-python3-ceph-argparse_17.2.6-1296-g736ac5ba-1focal_all.deb ... 2023-10-06T03:26:13.296 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking python3-ceph-argparse (17.2.6-1296-g736ac5ba-1focal) ... 2023-10-06T03:26:13.532 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package python3-cephfs. 2023-10-06T03:26:13.549 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../009-python3-cephfs_17.2.6-1296-g736ac5ba-1focal_amd64.deb ... 2023-10-06T03:26:13.591 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking python3-cephfs (17.2.6-1296-g736ac5ba-1focal) ... 2023-10-06T03:26:13.944 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package python3-ceph-common. 2023-10-06T03:26:13.961 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../010-python3-ceph-common_17.2.6-1296-g736ac5ba-1focal_all.deb ... 2023-10-06T03:26:14.002 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking python3-ceph-common (17.2.6-1296-g736ac5ba-1focal) ... 2023-10-06T03:26:14.116 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-cryptography. 2023-10-06T03:26:14.133 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../028-python3-cryptography_2.8-3ubuntu0.1_amd64.deb ... 2023-10-06T03:26:14.174 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-cryptography (2.8-3ubuntu0.1) ... 2023-10-06T03:26:14.339 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package python3-prettytable. 2023-10-06T03:26:14.356 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../011-python3-prettytable_0.7.2-5_all.deb ... 2023-10-06T03:26:14.398 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking python3-prettytable (0.7.2-5) ... 2023-10-06T03:26:14.562 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-openssl. 2023-10-06T03:26:14.577 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../029-python3-openssl_19.0.0-1build1_all.deb ... 2023-10-06T03:26:14.619 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-openssl (19.0.0-1build1) ... 2023-10-06T03:26:14.672 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package python3-rbd. 2023-10-06T03:26:14.688 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../012-python3-rbd_17.2.6-1296-g736ac5ba-1focal_amd64.deb ... 2023-10-06T03:26:14.730 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking python3-rbd (17.2.6-1296-g736ac5ba-1focal) ... 2023-10-06T03:26:14.880 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2023-10-06T03:26:14.899 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../030-ceph-mgr-modules-core_17.2.6-1296-g736ac5ba-1focal_all.deb ... 2023-10-06T03:26:14.930 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking ceph-mgr-modules-core (17.2.6-1296-g736ac5ba-1focal) ... 2023-10-06T03:26:15.082 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package liblua5.3-0:amd64. 2023-10-06T03:26:15.098 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../013-liblua5.3-0_5.3.3-1.1ubuntu2_amd64.deb ... 2023-10-06T03:26:15.132 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking liblua5.3-0:amd64 (5.3.3-1.1ubuntu2) ... 2023-10-06T03:26:15.294 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-bcrypt. 2023-10-06T03:26:15.310 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../031-python3-bcrypt_3.1.7-2ubuntu1_amd64.deb ... 2023-10-06T03:26:15.353 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-bcrypt (3.1.7-2ubuntu1) ... 2023-10-06T03:26:15.469 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package librabbitmq4:amd64. 2023-10-06T03:26:15.487 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../014-librabbitmq4_0.10.0-1_amd64.deb ... 2023-10-06T03:26:15.527 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking librabbitmq4:amd64 (0.10.0-1) ... 2023-10-06T03:26:15.655 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-cherrypy3. 2023-10-06T03:26:15.674 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../032-python3-cherrypy3_8.9.1-6_all.deb ... 2023-10-06T03:26:15.713 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-cherrypy3 (8.9.1-6) ... 2023-10-06T03:26:15.872 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package librdkafka1:amd64. 2023-10-06T03:26:15.891 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../015-librdkafka1_1.2.1-1ubuntu1_amd64.deb ... 2023-10-06T03:26:15.930 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking librdkafka1:amd64 (1.2.1-1ubuntu1) ... 2023-10-06T03:26:16.076 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-natsort. 2023-10-06T03:26:16.093 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../033-python3-natsort_7.0.1-1_all.deb ... 2023-10-06T03:26:16.134 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-natsort (7.0.1-1) ... 2023-10-06T03:26:16.283 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2023-10-06T03:26:16.300 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../016-libdouble-conversion3_3.1.5-4ubuntu1_amd64.deb ... 2023-10-06T03:26:16.358 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking libdouble-conversion3:amd64 (3.1.5-4ubuntu1) ... 2023-10-06T03:26:16.446 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-jwt. 2023-10-06T03:26:16.463 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../034-python3-jwt_1.7.1-2ubuntu2.1_all.deb ... 2023-10-06T03:26:16.504 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-jwt (1.7.1-2ubuntu2.1) ... 2023-10-06T03:26:16.619 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2023-10-06T03:26:16.636 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../017-libpcre2-16-0_10.34-7ubuntu0.1_amd64.deb ... 2023-10-06T03:26:16.711 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking libpcre2-16-0:amd64 (10.34-7ubuntu0.1) ... 2023-10-06T03:26:16.832 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-logutils. 2023-10-06T03:26:16.852 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../035-python3-logutils_0.3.3-7_all.deb ... 2023-10-06T03:26:16.890 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-logutils (0.3.3-7) ... 2023-10-06T03:26:17.193 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-markupsafe. 2023-10-06T03:26:17.206 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package libqt5core5a:amd64. 2023-10-06T03:26:17.212 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../036-python3-markupsafe_1.1.0-1build2_amd64.deb ... 2023-10-06T03:26:17.226 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../018-libqt5core5a_5.12.8+dfsg-0ubuntu2.1_amd64.deb ... 2023-10-06T03:26:17.252 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-markupsafe (1.1.0-1build2) ... 2023-10-06T03:26:17.275 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking libqt5core5a:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2023-10-06T03:26:17.575 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-mako. 2023-10-06T03:26:17.592 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../037-python3-mako_1.1.0+ds1-1ubuntu2.1_all.deb ... 2023-10-06T03:26:17.629 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-mako (1.1.0+ds1-1ubuntu2.1) ... 2023-10-06T03:26:17.776 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2023-10-06T03:26:17.792 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../019-libqt5dbus5_5.12.8+dfsg-0ubuntu2.1_amd64.deb ... 2023-10-06T03:26:17.825 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking libqt5dbus5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2023-10-06T03:26:17.950 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-simplegeneric. 2023-10-06T03:26:17.966 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../038-python3-simplegeneric_0.8.1-3_all.deb ... 2023-10-06T03:26:18.008 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2023-10-06T03:26:18.195 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package libqt5network5:amd64. 2023-10-06T03:26:18.212 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../020-libqt5network5_5.12.8+dfsg-0ubuntu2.1_amd64.deb ... 2023-10-06T03:26:18.254 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking libqt5network5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2023-10-06T03:26:18.328 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-singledispatch. 2023-10-06T03:26:18.347 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../039-python3-singledispatch_3.4.0.3-2build1_all.deb ... 2023-10-06T03:26:18.386 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-singledispatch (3.4.0.3-2build1) ... 2023-10-06T03:26:18.657 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package libthrift-0.13.0. 2023-10-06T03:26:18.672 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-webob. 2023-10-06T03:26:18.673 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../021-libthrift-0.13.0_0.13.0-2build2_amd64.deb ... 2023-10-06T03:26:18.691 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../040-python3-webob_1%3a1.8.5-2_all.deb ... 2023-10-06T03:26:18.715 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking libthrift-0.13.0 (0.13.0-2build2) ... 2023-10-06T03:26:18.722 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-webob (1:1.8.5-2) ... 2023-10-06T03:26:19.033 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-soupsieve. 2023-10-06T03:26:19.049 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../041-python3-soupsieve_1.9.5+dfsg-1_all.deb ... 2023-10-06T03:26:19.092 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-soupsieve (1.9.5+dfsg-1) ... 2023-10-06T03:26:19.126 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package librgw2. 2023-10-06T03:26:19.141 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../022-librgw2_17.2.6-1296-g736ac5ba-1focal_amd64.deb ... 2023-10-06T03:26:19.209 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking librgw2 (17.2.6-1296-g736ac5ba-1focal) ... 2023-10-06T03:26:19.428 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-bs4. 2023-10-06T03:26:19.443 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../042-python3-bs4_4.8.2-1_all.deb ... 2023-10-06T03:26:19.486 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-bs4 (4.8.2-1) ... 2023-10-06T03:26:19.839 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-waitress. 2023-10-06T03:26:19.857 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../043-python3-waitress_1.4.1-1ubuntu0.1_all.deb ... 2023-10-06T03:26:19.901 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-waitress (1.4.1-1ubuntu0.1) ... 2023-10-06T03:26:20.003 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package python3-rgw. 2023-10-06T03:26:20.020 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../023-python3-rgw_17.2.6-1296-g736ac5ba-1focal_amd64.deb ... 2023-10-06T03:26:20.062 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking python3-rgw (17.2.6-1296-g736ac5ba-1focal) ... 2023-10-06T03:26:20.234 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-tempita. 2023-10-06T03:26:20.252 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../044-python3-tempita_0.5.2-6_all.deb ... 2023-10-06T03:26:20.292 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-tempita (0.5.2-6) ... 2023-10-06T03:26:20.390 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package liboath0. 2023-10-06T03:26:20.406 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../024-liboath0_2.6.1-1.3_amd64.deb ... 2023-10-06T03:26:20.449 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking liboath0 (2.6.1-1.3) ... 2023-10-06T03:26:20.612 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-paste. 2023-10-06T03:26:20.629 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../045-python3-paste_3.4.0+dfsg1-1_all.deb ... 2023-10-06T03:26:20.670 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-paste (3.4.0+dfsg1-1) ... 2023-10-06T03:26:20.785 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package libradosstriper1. 2023-10-06T03:26:20.801 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../025-libradosstriper1_17.2.6-1296-g736ac5ba-1focal_amd64.deb ... 2023-10-06T03:26:20.843 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking libradosstriper1 (17.2.6-1296-g736ac5ba-1focal) ... 2023-10-06T03:26:20.982 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2023-10-06T03:26:20.998 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../046-python-pastedeploy-tpl_2.1.0-1_all.deb ... 2023-10-06T03:26:21.040 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python-pastedeploy-tpl (2.1.0-1) ... 2023-10-06T03:26:21.352 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-pastedeploy. 2023-10-06T03:26:21.371 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../047-python3-pastedeploy_2.1.0-1_all.deb ... 2023-10-06T03:26:21.388 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package ceph-common. 2023-10-06T03:26:21.405 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../026-ceph-common_17.2.6-1296-g736ac5ba-1focal_amd64.deb ... 2023-10-06T03:26:21.410 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-pastedeploy (2.1.0-1) ... 2023-10-06T03:26:21.446 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking ceph-common (17.2.6-1296-g736ac5ba-1focal) ... 2023-10-06T03:26:21.688 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-webtest. 2023-10-06T03:26:21.707 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../048-python3-webtest_2.0.32-2_all.deb ... 2023-10-06T03:26:21.746 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-webtest (2.0.32-2) ... 2023-10-06T03:26:22.057 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-pecan. 2023-10-06T03:26:22.073 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../049-python3-pecan_1.3.3-3ubuntu1_all.deb ... 2023-10-06T03:26:22.115 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-pecan (1.3.3-3ubuntu1) ... 2023-10-06T03:26:22.487 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-werkzeug. 2023-10-06T03:26:22.503 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../050-python3-werkzeug_0.16.1+dfsg1-2ubuntu0.1_all.deb ... 2023-10-06T03:26:22.545 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-werkzeug (0.16.1+dfsg1-2ubuntu0.1) ... 2023-10-06T03:26:22.899 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2023-10-06T03:26:22.915 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../051-libsqlite3-mod-ceph_17.2.6-1296-g736ac5ba-1focal_amd64.deb ... 2023-10-06T03:26:22.957 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking libsqlite3-mod-ceph (17.2.6-1296-g736ac5ba-1focal) ... 2023-10-06T03:26:23.927 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package ceph-mgr. 2023-10-06T03:26:23.947 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../052-ceph-mgr_17.2.6-1296-g736ac5ba-1focal_amd64.deb ... 2023-10-06T03:26:24.069 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking ceph-mgr (17.2.6-1296-g736ac5ba-1focal) ... 2023-10-06T03:26:24.555 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package ceph-mon. 2023-10-06T03:26:24.571 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../053-ceph-mon_17.2.6-1296-g736ac5ba-1focal_amd64.deb ... 2023-10-06T03:26:24.613 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking ceph-mon (17.2.6-1296-g736ac5ba-1focal) ... 2023-10-06T03:26:24.927 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package ceph-base. 2023-10-06T03:26:24.943 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../027-ceph-base_17.2.6-1296-g736ac5ba-1focal_amd64.deb ... 2023-10-06T03:26:24.996 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking ceph-base (17.2.6-1296-g736ac5ba-1focal) ... 2023-10-06T03:26:25.734 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package ceph-osd. 2023-10-06T03:26:25.750 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../054-ceph-osd_17.2.6-1296-g736ac5ba-1focal_amd64.deb ... 2023-10-06T03:26:25.817 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking ceph-osd (17.2.6-1296-g736ac5ba-1focal) ... 2023-10-06T03:26:26.006 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package python3-cryptography. 2023-10-06T03:26:26.024 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../028-python3-cryptography_2.8-3ubuntu0.1_amd64.deb ... 2023-10-06T03:26:26.056 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking python3-cryptography (2.8-3ubuntu0.1) ... 2023-10-06T03:26:26.426 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package python3-openssl. 2023-10-06T03:26:26.444 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../029-python3-openssl_19.0.0-1build1_all.deb ... 2023-10-06T03:26:26.476 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking python3-openssl (19.0.0-1build1) ... 2023-10-06T03:26:26.729 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2023-10-06T03:26:26.746 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../030-ceph-mgr-modules-core_17.2.6-1296-g736ac5ba-1focal_all.deb ... 2023-10-06T03:26:26.787 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking ceph-mgr-modules-core (17.2.6-1296-g736ac5ba-1focal) ... 2023-10-06T03:26:27.183 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package python3-bcrypt. 2023-10-06T03:26:27.203 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../031-python3-bcrypt_3.1.7-2ubuntu1_amd64.deb ... 2023-10-06T03:26:27.241 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking python3-bcrypt (3.1.7-2ubuntu1) ... 2023-10-06T03:26:27.578 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package python3-cherrypy3. 2023-10-06T03:26:27.594 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../032-python3-cherrypy3_8.9.1-6_all.deb ... 2023-10-06T03:26:27.661 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking python3-cherrypy3 (8.9.1-6) ... 2023-10-06T03:26:28.023 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package python3-natsort. 2023-10-06T03:26:28.038 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../033-python3-natsort_7.0.1-1_all.deb ... 2023-10-06T03:26:28.081 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking python3-natsort (7.0.1-1) ... 2023-10-06T03:26:28.400 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package python3-jwt. 2023-10-06T03:26:28.419 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../034-python3-jwt_1.7.1-2ubuntu2.1_all.deb ... 2023-10-06T03:26:28.458 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking python3-jwt (1.7.1-2ubuntu2.1) ... 2023-10-06T03:26:28.645 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package ceph. 2023-10-06T03:26:28.664 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../055-ceph_17.2.6-1296-g736ac5ba-1focal_amd64.deb ... 2023-10-06T03:26:28.703 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking ceph (17.2.6-1296-g736ac5ba-1focal) ... 2023-10-06T03:26:28.778 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package python3-logutils. 2023-10-06T03:26:28.794 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../035-python3-logutils_0.3.3-7_all.deb ... 2023-10-06T03:26:28.836 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking python3-logutils (0.3.3-7) ... 2023-10-06T03:26:29.022 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package ceph-fuse. 2023-10-06T03:26:29.038 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../056-ceph-fuse_17.2.6-1296-g736ac5ba-1focal_amd64.deb ... 2023-10-06T03:26:29.081 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking ceph-fuse (17.2.6-1296-g736ac5ba-1focal) ... 2023-10-06T03:26:29.173 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package python3-markupsafe. 2023-10-06T03:26:29.190 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../036-python3-markupsafe_1.1.0-1build2_amd64.deb ... 2023-10-06T03:26:29.232 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking python3-markupsafe (1.1.0-1build2) ... 2023-10-06T03:26:29.568 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package python3-mako. 2023-10-06T03:26:29.576 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package ceph-mds. 2023-10-06T03:26:29.585 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../037-python3-mako_1.1.0+ds1-1ubuntu2.1_all.deb ... 2023-10-06T03:26:29.593 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../057-ceph-mds_17.2.6-1296-g736ac5ba-1focal_amd64.deb ... 2023-10-06T03:26:29.625 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking python3-mako (1.1.0+ds1-1ubuntu2.1) ... 2023-10-06T03:26:29.634 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking ceph-mds (17.2.6-1296-g736ac5ba-1focal) ... 2023-10-06T03:26:29.987 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package python3-simplegeneric. 2023-10-06T03:26:30.004 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../038-python3-simplegeneric_0.8.1-3_all.deb ... 2023-10-06T03:26:30.045 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2023-10-06T03:26:30.137 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package cephadm. 2023-10-06T03:26:30.156 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../058-cephadm_17.2.6-1296-g736ac5ba-1focal_amd64.deb ... 2023-10-06T03:26:30.195 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking cephadm (17.2.6-1296-g736ac5ba-1focal) ... 2023-10-06T03:26:30.373 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package python3-singledispatch. 2023-10-06T03:26:30.389 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../039-python3-singledispatch_3.4.0.3-2build1_all.deb ... 2023-10-06T03:26:30.456 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking python3-singledispatch (3.4.0.3-2build1) ... 2023-10-06T03:26:30.531 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-jinja2. 2023-10-06T03:26:30.549 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../059-python3-jinja2_2.10.1-2_all.deb ... 2023-10-06T03:26:30.606 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-jinja2 (2.10.1-2) ... 2023-10-06T03:26:30.758 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package python3-webob. 2023-10-06T03:26:30.776 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../040-python3-webob_1%3a1.8.5-2_all.deb ... 2023-10-06T03:26:30.825 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking python3-webob (1:1.8.5-2) ... 2023-10-06T03:26:30.934 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2023-10-06T03:26:30.950 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../060-ceph-mgr-cephadm_17.2.6-1296-g736ac5ba-1focal_all.deb ... 2023-10-06T03:26:30.984 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking ceph-mgr-cephadm (17.2.6-1296-g736ac5ba-1focal) ... 2023-10-06T03:26:31.195 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package python3-soupsieve. 2023-10-06T03:26:31.210 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../041-python3-soupsieve_1.9.5+dfsg-1_all.deb ... 2023-10-06T03:26:31.253 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking python3-soupsieve (1.9.5+dfsg-1) ... 2023-10-06T03:26:31.321 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-repoze.lru. 2023-10-06T03:26:31.340 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../061-python3-repoze.lru_0.7-2_all.deb ... 2023-10-06T03:26:31.371 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2023-10-06T03:26:31.581 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package python3-bs4. 2023-10-06T03:26:31.597 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../042-python3-bs4_4.8.2-1_all.deb ... 2023-10-06T03:26:31.623 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package libjs-underscore. 2023-10-06T03:26:31.639 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking python3-bs4 (4.8.2-1) ... 2023-10-06T03:26:31.641 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../062-libjs-underscore_1.9.1~dfsg-1ubuntu0.20.04.1_all.deb ... 2023-10-06T03:26:31.682 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking libjs-underscore (1.9.1~dfsg-1ubuntu0.20.04.1) ... 2023-10-06T03:26:31.959 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package libjs-sphinxdoc. 2023-10-06T03:26:31.979 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../063-libjs-sphinxdoc_1.8.5-7ubuntu3_all.deb ... 2023-10-06T03:26:32.000 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package python3-waitress. 2023-10-06T03:26:32.017 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking libjs-sphinxdoc (1.8.5-7ubuntu3) ... 2023-10-06T03:26:32.019 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../043-python3-waitress_1.4.1-1ubuntu0.1_all.deb ... 2023-10-06T03:26:32.061 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking python3-waitress (1.4.1-1ubuntu0.1) ... 2023-10-06T03:26:32.362 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-routes. 2023-10-06T03:26:32.378 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../064-python3-routes_2.4.1-2_all.deb ... 2023-10-06T03:26:32.411 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package python3-tempita. 2023-10-06T03:26:32.420 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-routes (2.4.1-2) ... 2023-10-06T03:26:32.431 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../044-python3-tempita_0.5.2-6_all.deb ... 2023-10-06T03:26:32.469 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking python3-tempita (0.5.2-6) ... 2023-10-06T03:26:32.748 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2023-10-06T03:26:32.765 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../065-ceph-mgr-dashboard_17.2.6-1296-g736ac5ba-1focal_all.deb ... 2023-10-06T03:26:32.797 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package python3-paste. 2023-10-06T03:26:32.806 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking ceph-mgr-dashboard (17.2.6-1296-g736ac5ba-1focal) ... 2023-10-06T03:26:32.813 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../045-python3-paste_3.4.0+dfsg1-1_all.deb ... 2023-10-06T03:26:32.855 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking python3-paste (3.4.0+dfsg1-1) ... 2023-10-06T03:26:33.184 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2023-10-06T03:26:33.200 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../046-python-pastedeploy-tpl_2.1.0-1_all.deb ... 2023-10-06T03:26:33.275 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking python-pastedeploy-tpl (2.1.0-1) ... 2023-10-06T03:26:33.327 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-decorator. 2023-10-06T03:26:33.344 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../066-python3-decorator_4.4.2-0ubuntu1_all.deb ... 2023-10-06T03:26:33.385 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-decorator (4.4.2-0ubuntu1) ... 2023-10-06T03:26:33.605 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package python3-pastedeploy. 2023-10-06T03:26:33.624 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../047-python3-pastedeploy_2.1.0-1_all.deb ... 2023-10-06T03:26:33.669 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking python3-pastedeploy (2.1.0-1) ... 2023-10-06T03:26:33.688 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package liblbfgsb0:amd64. 2023-10-06T03:26:33.706 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../067-liblbfgsb0_3.0+dfsg.3-7build1_amd64.deb ... 2023-10-06T03:26:33.746 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking liblbfgsb0:amd64 (3.0+dfsg.3-7build1) ... 2023-10-06T03:26:33.981 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package python3-webtest. 2023-10-06T03:26:33.999 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../048-python3-webtest_2.0.32-2_all.deb ... 2023-10-06T03:26:34.039 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking python3-webtest (2.0.32-2) ... 2023-10-06T03:26:34.074 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-scipy. 2023-10-06T03:26:34.090 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../068-python3-scipy_1.3.3-3build1_amd64.deb ... 2023-10-06T03:26:34.132 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-scipy (1.3.3-3build1) ... 2023-10-06T03:26:34.383 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package python3-pecan. 2023-10-06T03:26:34.400 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../049-python3-pecan_1.3.3-3ubuntu1_all.deb ... 2023-10-06T03:26:34.441 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking python3-pecan (1.3.3-3ubuntu1) ... 2023-10-06T03:26:34.780 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package python3-werkzeug. 2023-10-06T03:26:34.799 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../050-python3-werkzeug_0.16.1+dfsg1-2ubuntu0.1_all.deb ... 2023-10-06T03:26:34.837 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking python3-werkzeug (0.16.1+dfsg1-2ubuntu0.1) ... 2023-10-06T03:26:35.199 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2023-10-06T03:26:35.216 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../051-libsqlite3-mod-ceph_17.2.6-1296-g736ac5ba-1focal_amd64.deb ... 2023-10-06T03:26:35.257 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking libsqlite3-mod-ceph (17.2.6-1296-g736ac5ba-1focal) ... 2023-10-06T03:26:35.416 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-sklearn-lib. 2023-10-06T03:26:35.436 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../069-python3-sklearn-lib_0.22.2.post1+dfsg-5_amd64.deb ... 2023-10-06T03:26:35.474 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-sklearn-lib (0.22.2.post1+dfsg-5) ... 2023-10-06T03:26:35.627 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package ceph-mgr. 2023-10-06T03:26:35.642 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../052-ceph-mgr_17.2.6-1296-g736ac5ba-1focal_amd64.deb ... 2023-10-06T03:26:35.685 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking ceph-mgr (17.2.6-1296-g736ac5ba-1focal) ... 2023-10-06T03:26:36.020 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-joblib. 2023-10-06T03:26:36.036 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../070-python3-joblib_0.14.0-3_all.deb ... 2023-10-06T03:26:36.078 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-joblib (0.14.0-3) ... 2023-10-06T03:26:36.222 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package ceph-mon. 2023-10-06T03:26:36.237 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../053-ceph-mon_17.2.6-1296-g736ac5ba-1focal_amd64.deb ... 2023-10-06T03:26:36.279 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking ceph-mon (17.2.6-1296-g736ac5ba-1focal) ... 2023-10-06T03:26:36.432 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-sklearn. 2023-10-06T03:26:36.449 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../071-python3-sklearn_0.22.2.post1+dfsg-5_all.deb ... 2023-10-06T03:26:36.489 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-sklearn (0.22.2.post1+dfsg-5) ... 2023-10-06T03:26:37.182 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2023-10-06T03:26:37.198 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../072-ceph-mgr-diskprediction-local_17.2.6-1296-g736ac5ba-1focal_all.deb ... 2023-10-06T03:26:37.240 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking ceph-mgr-diskprediction-local (17.2.6-1296-g736ac5ba-1focal) ... 2023-10-06T03:26:37.466 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package ceph-osd. 2023-10-06T03:26:37.482 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../054-ceph-osd_17.2.6-1296-g736ac5ba-1focal_amd64.deb ... 2023-10-06T03:26:37.549 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking ceph-osd (17.2.6-1296-g736ac5ba-1focal) ... 2023-10-06T03:26:38.343 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-cachetools. 2023-10-06T03:26:38.359 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../073-python3-cachetools_4.0.0-1_all.deb ... 2023-10-06T03:26:38.401 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-cachetools (4.0.0-1) ... 2023-10-06T03:26:38.729 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-pyasn1. 2023-10-06T03:26:38.746 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../074-python3-pyasn1_0.4.2-3build1_all.deb ... 2023-10-06T03:26:38.787 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-pyasn1 (0.4.2-3build1) ... 2023-10-06T03:26:39.116 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-pyasn1-modules. 2023-10-06T03:26:39.133 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../075-python3-pyasn1-modules_0.2.1-0.2build1_all.deb ... 2023-10-06T03:26:39.174 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-pyasn1-modules (0.2.1-0.2build1) ... 2023-10-06T03:26:39.885 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-rsa. 2023-10-06T03:26:39.905 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../076-python3-rsa_4.0-3_all.deb ... 2023-10-06T03:26:39.943 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-rsa (4.0-3) ... 2023-10-06T03:26:40.255 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-google-auth. 2023-10-06T03:26:40.273 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../077-python3-google-auth_1.5.1-2_all.deb ... 2023-10-06T03:26:40.313 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-google-auth (1.5.1-2) ... 2023-10-06T03:26:40.667 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-blinker. 2023-10-06T03:26:40.683 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../078-python3-blinker_1.4+dfsg1-0.3ubuntu1_all.deb ... 2023-10-06T03:26:40.725 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-blinker (1.4+dfsg1-0.3ubuntu1) ... 2023-10-06T03:26:40.738 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package ceph. 2023-10-06T03:26:40.756 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../055-ceph_17.2.6-1296-g736ac5ba-1focal_amd64.deb ... 2023-10-06T03:26:40.796 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking ceph (17.2.6-1296-g736ac5ba-1focal) ... 2023-10-06T03:26:41.028 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-oauthlib. 2023-10-06T03:26:41.045 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../079-python3-oauthlib_3.1.0-1ubuntu2_all.deb ... 2023-10-06T03:26:41.086 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-oauthlib (3.1.0-1ubuntu2) ... 2023-10-06T03:26:41.149 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package ceph-fuse. 2023-10-06T03:26:41.165 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../056-ceph-fuse_17.2.6-1296-g736ac5ba-1focal_amd64.deb ... 2023-10-06T03:26:41.207 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking ceph-fuse (17.2.6-1296-g736ac5ba-1focal) ... 2023-10-06T03:26:41.398 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-requests-oauthlib. 2023-10-06T03:26:41.414 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../080-python3-requests-oauthlib_1.0.0-1.1build1_all.deb ... 2023-10-06T03:26:41.472 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-requests-oauthlib (1.0.0-1.1build1) ... 2023-10-06T03:26:41.605 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package ceph-mds. 2023-10-06T03:26:41.621 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../057-ceph-mds_17.2.6-1296-g736ac5ba-1focal_amd64.deb ... 2023-10-06T03:26:41.662 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking ceph-mds (17.2.6-1296-g736ac5ba-1focal) ... 2023-10-06T03:26:41.870 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-websocket. 2023-10-06T03:26:41.886 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../081-python3-websocket_0.53.0-2ubuntu1_all.deb ... 2023-10-06T03:26:41.928 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-websocket (0.53.0-2ubuntu1) ... 2023-10-06T03:26:42.213 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package cephadm. 2023-10-06T03:26:42.229 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../058-cephadm_17.2.6-1296-g736ac5ba-1focal_amd64.deb ... 2023-10-06T03:26:42.240 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-kubernetes. 2023-10-06T03:26:42.260 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../082-python3-kubernetes_7.0.0~a1-2_all.deb ... 2023-10-06T03:26:42.271 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking cephadm (17.2.6-1296-g736ac5ba-1focal) ... 2023-10-06T03:26:42.298 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-kubernetes (7.0.0~a1-2) ... 2023-10-06T03:26:42.607 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package python3-jinja2. 2023-10-06T03:26:42.626 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../059-python3-jinja2_2.10.1-2_all.deb ... 2023-10-06T03:26:42.698 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking python3-jinja2 (2.10.1-2) ... 2023-10-06T03:26:42.864 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2023-10-06T03:26:42.881 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../083-ceph-mgr-k8sevents_17.2.6-1296-g736ac5ba-1focal_all.deb ... 2023-10-06T03:26:42.922 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking ceph-mgr-k8sevents (17.2.6-1296-g736ac5ba-1focal) ... 2023-10-06T03:26:43.060 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2023-10-06T03:26:43.077 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../060-ceph-mgr-cephadm_17.2.6-1296-g736ac5ba-1focal_all.deb ... 2023-10-06T03:26:43.118 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking ceph-mgr-cephadm (17.2.6-1296-g736ac5ba-1focal) ... 2023-10-06T03:26:43.267 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package libonig5:amd64. 2023-10-06T03:26:43.285 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../084-libonig5_6.9.4-1_amd64.deb ... 2023-10-06T03:26:43.334 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking libonig5:amd64 (6.9.4-1) ... 2023-10-06T03:26:43.504 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package python3-repoze.lru. 2023-10-06T03:26:43.522 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../061-python3-repoze.lru_0.7-2_all.deb ... 2023-10-06T03:26:43.562 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2023-10-06T03:26:43.687 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package libjq1:amd64. 2023-10-06T03:26:43.703 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../085-libjq1_1.6-1ubuntu0.20.04.1_amd64.deb ... 2023-10-06T03:26:43.745 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking libjq1:amd64 (1.6-1ubuntu0.20.04.1) ... 2023-10-06T03:26:43.841 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package libjs-underscore. 2023-10-06T03:26:43.856 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../062-libjs-underscore_1.9.1~dfsg-1ubuntu0.20.04.1_all.deb ... 2023-10-06T03:26:43.898 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking libjs-underscore (1.9.1~dfsg-1ubuntu0.20.04.1) ... 2023-10-06T03:26:43.998 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package jq. 2023-10-06T03:26:44.014 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../086-jq_1.6-1ubuntu0.20.04.1_amd64.deb ... 2023-10-06T03:26:44.056 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking jq (1.6-1ubuntu0.20.04.1) ... 2023-10-06T03:26:44.192 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package libjs-sphinxdoc. 2023-10-06T03:26:44.211 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../063-libjs-sphinxdoc_1.8.5-7ubuntu3_all.deb ... 2023-10-06T03:26:44.250 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking libjs-sphinxdoc (1.8.5-7ubuntu3) ... 2023-10-06T03:26:44.326 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package socat. 2023-10-06T03:26:44.342 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../087-socat_1.7.3.3-2_amd64.deb ... 2023-10-06T03:26:44.384 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking socat (1.7.3.3-2) ... 2023-10-06T03:26:44.578 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package python3-routes. 2023-10-06T03:26:44.596 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../064-python3-routes_2.4.1-2_all.deb ... 2023-10-06T03:26:44.636 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking python3-routes (2.4.1-2) ... 2023-10-06T03:26:44.696 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package xmlstarlet. 2023-10-06T03:26:44.713 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../088-xmlstarlet_1.6.1-2build1_amd64.deb ... 2023-10-06T03:26:44.754 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking xmlstarlet (1.6.1-2build1) ... 2023-10-06T03:26:44.989 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2023-10-06T03:26:45.008 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../065-ceph-mgr-dashboard_17.2.6-1296-g736ac5ba-1focal_all.deb ... 2023-10-06T03:26:45.048 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking ceph-mgr-dashboard (17.2.6-1296-g736ac5ba-1focal) ... 2023-10-06T03:26:45.117 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package ceph-test. 2023-10-06T03:26:45.134 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../089-ceph-test_17.2.6-1296-g736ac5ba-1focal_amd64.deb ... 2023-10-06T03:26:45.175 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking ceph-test (17.2.6-1296-g736ac5ba-1focal) ... 2023-10-06T03:26:45.551 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package python3-decorator. 2023-10-06T03:26:45.567 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../066-python3-decorator_4.4.2-0ubuntu1_all.deb ... 2023-10-06T03:26:45.609 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking python3-decorator (4.4.2-0ubuntu1) ... 2023-10-06T03:26:45.937 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package liblbfgsb0:amd64. 2023-10-06T03:26:45.953 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../067-liblbfgsb0_3.0+dfsg.3-7build1_amd64.deb ... 2023-10-06T03:26:45.994 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking liblbfgsb0:amd64 (3.0+dfsg.3-7build1) ... 2023-10-06T03:26:46.347 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package python3-scipy. 2023-10-06T03:26:46.651 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../068-python3-scipy_1.3.3-3build1_amd64.deb ... 2023-10-06T03:26:46.652 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking python3-scipy (1.3.3-3build1) ... 2023-10-06T03:26:47.739 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package python3-sklearn-lib. 2023-10-06T03:26:47.755 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../069-python3-sklearn-lib_0.22.2.post1+dfsg-5_amd64.deb ... 2023-10-06T03:26:47.789 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking python3-sklearn-lib (0.22.2.post1+dfsg-5) ... 2023-10-06T03:26:48.350 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package python3-joblib. 2023-10-06T03:26:48.368 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../070-python3-joblib_0.14.0-3_all.deb ... 2023-10-06T03:26:48.408 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking python3-joblib (0.14.0-3) ... 2023-10-06T03:26:48.795 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package python3-sklearn. 2023-10-06T03:26:48.813 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../071-python3-sklearn_0.22.2.post1+dfsg-5_all.deb ... 2023-10-06T03:26:48.853 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking python3-sklearn (0.22.2.post1+dfsg-5) ... 2023-10-06T03:26:49.670 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2023-10-06T03:26:49.686 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../072-ceph-mgr-diskprediction-local_17.2.6-1296-g736ac5ba-1focal_all.deb ... 2023-10-06T03:26:49.736 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking ceph-mgr-diskprediction-local (17.2.6-1296-g736ac5ba-1focal) ... 2023-10-06T03:26:50.295 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package ceph-volume. 2023-10-06T03:26:50.313 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../090-ceph-volume_17.2.6-1296-g736ac5ba-1focal_all.deb ... 2023-10-06T03:26:50.354 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking ceph-volume (17.2.6-1296-g736ac5ba-1focal) ... 2023-10-06T03:26:50.801 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package cephfs-mirror. 2023-10-06T03:26:50.818 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../091-cephfs-mirror_17.2.6-1296-g736ac5ba-1focal_amd64.deb ... 2023-10-06T03:26:50.859 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking cephfs-mirror (17.2.6-1296-g736ac5ba-1focal) ... 2023-10-06T03:26:50.898 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package python3-cachetools. 2023-10-06T03:26:50.914 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../073-python3-cachetools_4.0.0-1_all.deb ... 2023-10-06T03:26:50.956 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking python3-cachetools (4.0.0-1) ... 2023-10-06T03:26:51.212 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-colorama. 2023-10-06T03:26:51.228 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../092-python3-colorama_0.4.3-1build1_all.deb ... 2023-10-06T03:26:51.250 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package python3-pyasn1. 2023-10-06T03:26:51.268 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../074-python3-pyasn1_0.4.2-3build1_all.deb ... 2023-10-06T03:26:51.270 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-colorama (0.4.3-1build1) ... 2023-10-06T03:26:51.308 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking python3-pyasn1 (0.4.2-3build1) ... 2023-10-06T03:26:51.565 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package cephfs-shell. 2023-10-06T03:26:51.582 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../093-cephfs-shell_17.2.6-1296-g736ac5ba-1focal_all.deb ... 2023-10-06T03:26:51.623 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking cephfs-shell (17.2.6-1296-g736ac5ba-1focal) ... 2023-10-06T03:26:51.661 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package python3-pyasn1-modules. 2023-10-06T03:26:51.677 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../075-python3-pyasn1-modules_0.2.1-0.2build1_all.deb ... 2023-10-06T03:26:51.711 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking python3-pyasn1-modules (0.2.1-0.2build1) ... 2023-10-06T03:26:51.875 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package cephfs-top. 2023-10-06T03:26:51.893 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../094-cephfs-top_17.2.6-1296-g736ac5ba-1focal_all.deb ... 2023-10-06T03:26:51.925 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking cephfs-top (17.2.6-1296-g736ac5ba-1focal) ... 2023-10-06T03:26:52.022 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package python3-rsa. 2023-10-06T03:26:52.041 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../076-python3-rsa_4.0-3_all.deb ... 2023-10-06T03:26:52.072 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking python3-rsa (4.0-3) ... 2023-10-06T03:26:52.178 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package fonts-mathjax. 2023-10-06T03:26:52.197 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../095-fonts-mathjax_2.7.4+dfsg-1_all.deb ... 2023-10-06T03:26:52.236 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking fonts-mathjax (2.7.4+dfsg-1) ... 2023-10-06T03:26:52.425 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package python3-google-auth. 2023-10-06T03:26:52.441 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../077-python3-google-auth_1.5.1-2_all.deb ... 2023-10-06T03:26:52.483 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking python3-google-auth (1.5.1-2) ... 2023-10-06T03:26:52.757 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package libcephfs-dev. 2023-10-06T03:26:52.774 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../096-libcephfs-dev_17.2.6-1296-g736ac5ba-1focal_amd64.deb ... 2023-10-06T03:26:52.815 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking libcephfs-dev (17.2.6-1296-g736ac5ba-1focal) ... 2023-10-06T03:26:52.819 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package python3-blinker. 2023-10-06T03:26:52.839 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../078-python3-blinker_1.4+dfsg1-0.3ubuntu1_all.deb ... 2023-10-06T03:26:52.877 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking python3-blinker (1.4+dfsg1-0.3ubuntu1) ... 2023-10-06T03:26:53.198 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package nvme-cli. 2023-10-06T03:26:53.214 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package python3-oauthlib. 2023-10-06T03:26:53.214 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../097-nvme-cli_1.9-1ubuntu0.1_amd64.deb ... 2023-10-06T03:26:53.230 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../079-python3-oauthlib_3.1.0-1ubuntu2_all.deb ... 2023-10-06T03:26:53.264 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking nvme-cli (1.9-1ubuntu0.1) ... 2023-10-06T03:26:53.305 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking python3-oauthlib (3.1.0-1ubuntu2) ... 2023-10-06T03:26:53.611 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package libjs-mathjax. 2023-10-06T03:26:53.630 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../098-libjs-mathjax_2.7.4+dfsg-1_all.deb ... 2023-10-06T03:26:53.658 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package python3-requests-oauthlib. 2023-10-06T03:26:53.669 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking libjs-mathjax (2.7.4+dfsg-1) ... 2023-10-06T03:26:53.674 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../080-python3-requests-oauthlib_1.0.0-1.1build1_all.deb ... 2023-10-06T03:26:53.717 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking python3-requests-oauthlib (1.0.0-1.1build1) ... 2023-10-06T03:26:54.147 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package python3-websocket. 2023-10-06T03:26:54.163 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../081-python3-websocket_0.53.0-2ubuntu1_all.deb ... 2023-10-06T03:26:54.205 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking python3-websocket (0.53.0-2ubuntu1) ... 2023-10-06T03:26:54.542 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package python3-kubernetes. 2023-10-06T03:26:54.559 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../082-python3-kubernetes_7.0.0~a1-2_all.deb ... 2023-10-06T03:26:54.600 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking python3-kubernetes (7.0.0~a1-2) ... 2023-10-06T03:26:55.172 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2023-10-06T03:26:55.189 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../083-ceph-mgr-k8sevents_17.2.6-1296-g736ac5ba-1focal_all.deb ... 2023-10-06T03:26:55.230 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking ceph-mgr-k8sevents (17.2.6-1296-g736ac5ba-1focal) ... 2023-10-06T03:26:55.261 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python-natsort-doc. 2023-10-06T03:26:55.279 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../099-python-natsort-doc_7.0.1-1_all.deb ... 2023-10-06T03:26:55.319 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python-natsort-doc (7.0.1-1) ... 2023-10-06T03:26:55.575 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package libonig5:amd64. 2023-10-06T03:26:55.591 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../084-libonig5_6.9.4-1_amd64.deb ... 2023-10-06T03:26:55.633 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking libonig5:amd64 (6.9.4-1) ... 2023-10-06T03:26:55.655 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-atomicwrites. 2023-10-06T03:26:55.672 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../100-python3-atomicwrites_1.1.5-2build1_all.deb ... 2023-10-06T03:26:55.713 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-atomicwrites (1.1.5-2build1) ... 2023-10-06T03:26:55.966 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-attr. 2023-10-06T03:26:55.978 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package libjq1:amd64. 2023-10-06T03:26:55.983 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../101-python3-attr_19.3.0-2_all.deb ... 2023-10-06T03:26:55.994 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../085-libjq1_1.6-1ubuntu0.20.04.1_amd64.deb ... 2023-10-06T03:26:56.035 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-attr (19.3.0-2) ... 2023-10-06T03:26:56.036 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking libjq1:amd64 (1.6-1ubuntu0.20.04.1) ... 2023-10-06T03:26:56.328 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-webencodings. 2023-10-06T03:26:56.347 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package jq. 2023-10-06T03:26:56.348 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../102-python3-webencodings_0.5.1-1ubuntu1_all.deb ... 2023-10-06T03:26:56.365 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../086-jq_1.6-1ubuntu0.20.04.1_amd64.deb ... 2023-10-06T03:26:56.385 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-webencodings (0.5.1-1ubuntu1) ... 2023-10-06T03:26:56.405 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking jq (1.6-1ubuntu0.20.04.1) ... 2023-10-06T03:26:56.672 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-html5lib. 2023-10-06T03:26:56.688 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package socat. 2023-10-06T03:26:56.692 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../103-python3-html5lib_1.0.1-2_all.deb ... 2023-10-06T03:26:56.705 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../087-socat_1.7.3.3-2_amd64.deb ... 2023-10-06T03:26:56.730 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-html5lib (1.0.1-2) ... 2023-10-06T03:26:56.746 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking socat (1.7.3.3-2) ... 2023-10-06T03:26:57.058 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-lxml:amd64. 2023-10-06T03:26:57.078 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../104-python3-lxml_4.5.0-1ubuntu0.5_amd64.deb ... 2023-10-06T03:26:57.091 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package xmlstarlet. 2023-10-06T03:26:57.109 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../088-xmlstarlet_1.6.1-2build1_amd64.deb ... 2023-10-06T03:26:57.116 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-lxml:amd64 (4.5.0-1ubuntu0.5) ... 2023-10-06T03:26:57.149 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking xmlstarlet (1.6.1-2build1) ... 2023-10-06T03:26:57.520 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package ceph-test. 2023-10-06T03:26:57.540 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../089-ceph-test_17.2.6-1296-g736ac5ba-1focal_amd64.deb ... 2023-10-06T03:26:57.553 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-packaging. 2023-10-06T03:26:57.570 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../105-python3-packaging_20.3-1_all.deb ... 2023-10-06T03:26:57.603 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-packaging (20.3-1) ... 2023-10-06T03:26:57.611 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking ceph-test (17.2.6-1296-g736ac5ba-1focal) ... 2023-10-06T03:26:57.923 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-pastescript. 2023-10-06T03:26:57.942 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../106-python3-pastescript_2.0.2-4_all.deb ... 2023-10-06T03:26:57.981 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2023-10-06T03:26:58.302 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-pluggy. 2023-10-06T03:26:58.319 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../107-python3-pluggy_0.13.0-2_all.deb ... 2023-10-06T03:26:58.360 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-pluggy (0.13.0-2) ... 2023-10-06T03:26:58.672 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-psutil. 2023-10-06T03:26:58.690 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../108-python3-psutil_5.5.1-1ubuntu4_amd64.deb ... 2023-10-06T03:26:58.730 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-psutil (5.5.1-1ubuntu4) ... 2023-10-06T03:26:59.066 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-py. 2023-10-06T03:26:59.086 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../109-python3-py_1.8.1-1ubuntu0.1_all.deb ... 2023-10-06T03:26:59.125 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-py (1.8.1-1ubuntu0.1) ... 2023-10-06T03:26:59.454 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-pyinotify. 2023-10-06T03:26:59.470 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../110-python3-pyinotify_0.9.6-1.2ubuntu1_all.deb ... 2023-10-06T03:26:59.512 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-pyinotify (0.9.6-1.2ubuntu1) ... 2023-10-06T03:26:59.849 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-pytest. 2023-10-06T03:26:59.866 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../111-python3-pytest_4.6.9-1_all.deb ... 2023-10-06T03:26:59.923 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-pytest (4.6.9-1) ... 2023-10-06T03:27:00.261 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-simplejson. 2023-10-06T03:27:00.281 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../112-python3-simplejson_3.16.0-2ubuntu2_amd64.deb ... 2023-10-06T03:27:00.318 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-simplejson (3.16.0-2ubuntu2) ... 2023-10-06T03:27:00.575 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package qttranslations5-l10n. 2023-10-06T03:27:00.597 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../113-qttranslations5-l10n_5.12.8-0ubuntu1_all.deb ... 2023-10-06T03:27:00.650 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking qttranslations5-l10n (5.12.8-0ubuntu1) ... 2023-10-06T03:27:01.387 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package radosgw. 2023-10-06T03:27:01.404 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../114-radosgw_17.2.6-1296-g736ac5ba-1focal_amd64.deb ... 2023-10-06T03:27:01.437 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking radosgw (17.2.6-1296-g736ac5ba-1focal) ... 2023-10-06T03:27:02.682 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package ceph-volume. 2023-10-06T03:27:02.700 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../090-ceph-volume_17.2.6-1296-g736ac5ba-1focal_all.deb ... 2023-10-06T03:27:02.740 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking ceph-volume (17.2.6-1296-g736ac5ba-1focal) ... 2023-10-06T03:27:03.148 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package rbd-fuse. 2023-10-06T03:27:03.167 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../115-rbd-fuse_17.2.6-1296-g736ac5ba-1focal_amd64.deb ... 2023-10-06T03:27:03.179 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package cephfs-mirror. 2023-10-06T03:27:03.195 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../091-cephfs-mirror_17.2.6-1296-g736ac5ba-1focal_amd64.deb ... 2023-10-06T03:27:03.207 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking rbd-fuse (17.2.6-1296-g736ac5ba-1focal) ... 2023-10-06T03:27:03.236 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking cephfs-mirror (17.2.6-1296-g736ac5ba-1focal) ... 2023-10-06T03:27:03.554 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-attr (19.3.0-2) ... 2023-10-06T03:27:03.606 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package python3-colorama. 2023-10-06T03:27:03.627 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../092-python3-colorama_0.4.3-1build1_all.deb ... 2023-10-06T03:27:03.673 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking python3-colorama (0.4.3-1build1) ... 2023-10-06T03:27:03.839 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up librabbitmq4:amd64 (0.10.0-1) ... 2023-10-06T03:27:03.956 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up libdouble-conversion3:amd64 (3.1.5-4ubuntu1) ... 2023-10-06T03:27:03.962 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package cephfs-shell. 2023-10-06T03:27:03.981 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../093-cephfs-shell_17.2.6-1296-g736ac5ba-1focal_all.deb ... 2023-10-06T03:27:04.020 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking cephfs-shell (17.2.6-1296-g736ac5ba-1focal) ... 2023-10-06T03:27:04.090 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up nvme-cli (1.9-1ubuntu0.1) ... 2023-10-06T03:27:04.298 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package cephfs-top. 2023-10-06T03:27:04.317 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up cephadm (17.2.6-1296-g736ac5ba-1focal) ... 2023-10-06T03:27:04.317 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../094-cephfs-top_17.2.6-1296-g736ac5ba-1focal_all.deb ... 2023-10-06T03:27:04.356 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking cephfs-top (17.2.6-1296-g736ac5ba-1focal) ... 2023-10-06T03:27:04.625 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package fonts-mathjax. 2023-10-06T03:27:04.642 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../095-fonts-mathjax_2.7.4+dfsg-1_all.deb ... 2023-10-06T03:27:04.683 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking fonts-mathjax (2.7.4+dfsg-1) ... 2023-10-06T03:27:04.950 INFO:teuthology.orchestra.run.smithi110.stdout:Adding system user cephadm....done 2023-10-06T03:27:05.137 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up fonts-mathjax (2.7.4+dfsg-1) ... 2023-10-06T03:27:05.237 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package libcephfs-dev. 2023-10-06T03:27:05.255 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-waitress (1.4.1-1ubuntu0.1) ... 2023-10-06T03:27:05.256 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../096-libcephfs-dev_17.2.6-1296-g736ac5ba-1focal_amd64.deb ... 2023-10-06T03:27:05.294 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking libcephfs-dev (17.2.6-1296-g736ac5ba-1focal) ... 2023-10-06T03:27:05.608 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-repoze.lru (0.7-2) ... 2023-10-06T03:27:05.705 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package nvme-cli. 2023-10-06T03:27:05.722 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../097-nvme-cli_1.9-1ubuntu0.1_amd64.deb ... 2023-10-06T03:27:05.789 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking nvme-cli (1.9-1ubuntu0.1) ... 2023-10-06T03:27:05.910 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up libjs-mathjax (2.7.4+dfsg-1) ... 2023-10-06T03:27:06.069 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up liboath0 (2.6.1-1.3) ... 2023-10-06T03:27:06.135 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package libjs-mathjax. 2023-10-06T03:27:06.151 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../098-libjs-mathjax_2.7.4+dfsg-1_all.deb ... 2023-10-06T03:27:06.197 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-py (1.8.1-1ubuntu0.1) ... 2023-10-06T03:27:06.218 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking libjs-mathjax (2.7.4+dfsg-1) ... 2023-10-06T03:27:06.532 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-joblib (0.14.0-3) ... 2023-10-06T03:27:06.992 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-jwt (1.7.1-2ubuntu2.1) ... 2023-10-06T03:27:07.285 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-cachetools (4.0.0-1) ... 2023-10-06T03:27:07.586 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-colorama (0.4.3-1build1) ... 2023-10-06T03:27:07.889 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package python-natsort-doc. 2023-10-06T03:27:07.907 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../099-python-natsort-doc_7.0.1-1_all.deb ... 2023-10-06T03:27:07.912 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-bcrypt (3.1.7-2ubuntu1) ... 2023-10-06T03:27:07.972 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking python-natsort-doc (7.0.1-1) ... 2023-10-06T03:27:08.172 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-pyinotify (0.9.6-1.2ubuntu1) ... 2023-10-06T03:27:08.325 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package python3-atomicwrites. 2023-10-06T03:27:08.342 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../100-python3-atomicwrites_1.1.5-2build1_all.deb ... 2023-10-06T03:27:08.383 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking python3-atomicwrites (1.1.5-2build1) ... 2023-10-06T03:27:08.473 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-ceph-argparse (17.2.6-1296-g736ac5ba-1focal) ... 2023-10-06T03:27:08.720 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package python3-attr. 2023-10-06T03:27:08.738 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../101-python3-attr_19.3.0-2_all.deb ... 2023-10-06T03:27:08.765 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-sklearn-lib (0.22.2.post1+dfsg-5) ... 2023-10-06T03:27:08.777 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking python3-attr (19.3.0-2) ... 2023-10-06T03:27:08.925 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-pyperclip (1.7.0-1) ... 2023-10-06T03:27:09.110 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package python3-webencodings. 2023-10-06T03:27:09.122 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../102-python3-webencodings_0.5.1-1ubuntu1_all.deb ... 2023-10-06T03:27:09.163 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking python3-webencodings (0.5.1-1ubuntu1) ... 2023-10-06T03:27:09.209 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-markupsafe (1.1.0-1build2) ... 2023-10-06T03:27:09.476 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package python3-html5lib. 2023-10-06T03:27:09.496 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../103-python3-html5lib_1.0.1-2_all.deb ... 2023-10-06T03:27:09.502 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-webencodings (0.5.1-1ubuntu1) ... 2023-10-06T03:27:09.534 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking python3-html5lib (1.0.1-2) ... 2023-10-06T03:27:09.794 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up libpcre2-16-0:amd64 (10.34-7ubuntu0.1) ... 2023-10-06T03:27:09.887 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package python3-lxml:amd64. 2023-10-06T03:27:09.906 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../104-python3-lxml_4.5.0-1ubuntu0.5_amd64.deb ... 2023-10-06T03:27:09.920 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-psutil (5.5.1-1ubuntu4) ... 2023-10-06T03:27:09.945 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking python3-lxml:amd64 (4.5.0-1ubuntu0.5) ... 2023-10-06T03:27:10.296 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-natsort (7.0.1-1) ... 2023-10-06T03:27:10.440 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package python3-packaging. 2023-10-06T03:27:10.460 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../105-python3-packaging_20.3-1_all.deb ... 2023-10-06T03:27:10.498 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking python3-packaging (20.3-1) ... 2023-10-06T03:27:10.827 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package python3-pastescript. 2023-10-06T03:27:10.844 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../106-python3-pastescript_2.0.2-4_all.deb ... 2023-10-06T03:27:10.885 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2023-10-06T03:27:10.914 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-atomicwrites (1.1.5-2build1) ... 2023-10-06T03:27:11.190 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-simplejson (3.16.0-2ubuntu2) ... 2023-10-06T03:27:11.422 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package python3-pluggy. 2023-10-06T03:27:11.441 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../107-python3-pluggy_0.13.0-2_all.deb ... 2023-10-06T03:27:11.525 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-decorator (4.4.2-0ubuntu1) ... 2023-10-06T03:27:11.807 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking python3-pluggy (0.13.0-2) ... 2023-10-06T03:27:11.834 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-jinja2 (2.10.1-2) ... 2023-10-06T03:27:12.117 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package python3-psutil. 2023-10-06T03:27:12.136 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../108-python3-psutil_5.5.1-1ubuntu4_amd64.deb ... 2023-10-06T03:27:12.185 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-tempita (0.5.2-6) ... 2023-10-06T03:27:12.200 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking python3-psutil (5.5.1-1ubuntu4) ... 2023-10-06T03:27:12.486 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-packaging (20.3-1) ... 2023-10-06T03:27:12.574 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package python3-py. 2023-10-06T03:27:12.591 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../109-python3-py_1.8.1-1ubuntu0.1_all.deb ... 2023-10-06T03:27:12.632 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking python3-py (1.8.1-1ubuntu0.1) ... 2023-10-06T03:27:12.780 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python-pastedeploy-tpl (2.1.0-1) ... 2023-10-06T03:27:12.905 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up qttranslations5-l10n (5.12.8-0ubuntu1) ... 2023-10-06T03:27:12.962 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package python3-pyinotify. 2023-10-06T03:27:12.978 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../110-python3-pyinotify_0.9.6-1.2ubuntu1_all.deb ... 2023-10-06T03:27:13.018 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking python3-pyinotify (0.9.6-1.2ubuntu1) ... 2023-10-06T03:27:13.023 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-wcwidth (0.1.8+dfsg1-3) ... 2023-10-06T03:27:13.332 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-paste (3.4.0+dfsg1-1) ... 2023-10-06T03:27:13.354 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package python3-pytest. 2023-10-06T03:27:13.376 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../111-python3-pytest_4.6.9-1_all.deb ... 2023-10-06T03:27:13.412 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking python3-pytest (4.6.9-1) ... 2023-10-06T03:27:13.721 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-cmd2 (0.8.5-2.1) ... 2023-10-06T03:27:13.774 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package python3-simplejson. 2023-10-06T03:27:13.796 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../112-python3-simplejson_3.16.0-2ubuntu2_amd64.deb ... 2023-10-06T03:27:13.832 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking python3-simplejson (3.16.0-2ubuntu2) ... 2023-10-06T03:27:14.030 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-werkzeug (0.16.1+dfsg1-2ubuntu0.1) ... 2023-10-06T03:27:14.144 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package qttranslations5-l10n. 2023-10-06T03:27:14.162 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../113-qttranslations5-l10n_5.12.8-0ubuntu1_all.deb ... 2023-10-06T03:27:14.202 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking qttranslations5-l10n (5.12.8-0ubuntu1) ... 2023-10-06T03:27:14.406 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-cryptography (2.8-3ubuntu0.1) ... 2023-10-06T03:27:14.766 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up socat (1.7.3.3-2) ... 2023-10-06T03:27:14.892 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-ceph-common (17.2.6-1296-g736ac5ba-1focal) ... 2023-10-06T03:27:14.979 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package radosgw. 2023-10-06T03:27:15.000 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../114-radosgw_17.2.6-1296-g736ac5ba-1focal_amd64.deb ... 2023-10-06T03:27:15.040 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking radosgw (17.2.6-1296-g736ac5ba-1focal) ... 2023-10-06T03:27:15.218 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up libqt5core5a:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2023-10-06T03:27:15.355 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-html5lib (1.0.1-2) ... 2023-10-06T03:27:15.732 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up librdkafka1:amd64 (1.2.1-1ubuntu1) ... 2023-10-06T03:27:15.858 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up xmlstarlet (1.6.1-2build1) ... 2023-10-06T03:27:15.983 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-pluggy (0.13.0-2) ... 2023-10-06T03:27:16.293 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-lxml:amd64 (4.5.0-1ubuntu0.5) ... 2023-10-06T03:27:16.636 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up libqt5dbus5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2023-10-06T03:27:16.752 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package rbd-fuse. 2023-10-06T03:27:16.762 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up liblua5.3-0:amd64 (5.3.3-1.1ubuntu2) ... 2023-10-06T03:27:16.772 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../115-rbd-fuse_17.2.6-1296-g736ac5ba-1focal_amd64.deb ... 2023-10-06T03:27:16.802 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking rbd-fuse (17.2.6-1296-g736ac5ba-1focal) ... 2023-10-06T03:27:16.882 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-pyasn1 (0.4.2-3build1) ... 2023-10-06T03:27:17.130 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up python3-attr (19.3.0-2) ... 2023-10-06T03:27:17.208 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-singledispatch (3.4.0.3-2build1) ... 2023-10-06T03:27:17.470 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up librabbitmq4:amd64 (0.10.0-1) ... 2023-10-06T03:27:17.518 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-logutils (0.3.3-7) ... 2023-10-06T03:27:17.597 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up libdouble-conversion3:amd64 (3.1.5-4ubuntu1) ... 2023-10-06T03:27:17.724 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up nvme-cli (1.9-1ubuntu0.1) ... 2023-10-06T03:27:17.819 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up liblttng-ust0:amd64 (2.11.0-1) ... 2023-10-06T03:27:17.945 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2023-10-06T03:27:17.958 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up cephadm (17.2.6-1296-g736ac5ba-1focal) ... 2023-10-06T03:27:18.229 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-prettytable (0.7.2-5) ... 2023-10-06T03:27:18.513 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-soupsieve (1.9.5+dfsg-1) ... 2023-10-06T03:27:18.639 INFO:teuthology.orchestra.run.smithi192.stdout:Adding system user cephadm....done 2023-10-06T03:27:18.800 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up fonts-mathjax (2.7.4+dfsg-1) ... 2023-10-06T03:27:18.806 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-websocket (0.53.0-2ubuntu1) ... 2023-10-06T03:27:18.960 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up python3-waitress (1.4.1-1ubuntu0.1) ... 2023-10-06T03:27:19.249 INFO:teuthology.orchestra.run.smithi110.stdout:update-alternatives: using /usr/bin/python3-wsdump to provide /usr/bin/wsdump (wsdump) in auto mode 2023-10-06T03:27:19.312 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up python3-repoze.lru (0.7-2) ... 2023-10-06T03:27:19.500 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up libonig5:amd64 (6.9.4-1) ... 2023-10-06T03:27:19.605 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up libjs-mathjax (2.7.4+dfsg-1) ... 2023-10-06T03:27:19.626 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-mako (1.1.0+ds1-1ubuntu2.1) ... 2023-10-06T03:27:19.731 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up liboath0 (2.6.1-1.3) ... 2023-10-06T03:27:19.857 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up python3-py (1.8.1-1ubuntu0.1) ... 2023-10-06T03:27:19.961 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-cherrypy3 (8.9.1-6) ... 2023-10-06T03:27:20.200 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up python3-joblib (0.14.0-3) ... 2023-10-06T03:27:20.345 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up libjs-underscore (1.9.1~dfsg-1ubuntu0.20.04.1) ... 2023-10-06T03:27:20.471 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up liblbfgsb0:amd64 (3.0+dfsg.3-7build1) ... 2023-10-06T03:27:20.597 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-webob (1:1.8.5-2) ... 2023-10-06T03:27:20.652 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up python3-jwt (1.7.1-2ubuntu2.1) ... 2023-10-06T03:27:20.940 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-blinker (1.4+dfsg1-0.3ubuntu1) ... 2023-10-06T03:27:20.987 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up python3-cachetools (4.0.0-1) ... 2023-10-06T03:27:21.225 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up librados2 (17.2.6-1296-g736ac5ba-1focal) ... 2023-10-06T03:27:21.263 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up python3-colorama (0.4.3-1build1) ... 2023-10-06T03:27:21.351 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-scipy (1.3.3-3build1) ... 2023-10-06T03:27:21.573 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up python3-bcrypt (3.1.7-2ubuntu1) ... 2023-10-06T03:27:21.901 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up python3-pyinotify (0.9.6-1.2ubuntu1) ... 2023-10-06T03:27:22.183 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up python3-ceph-argparse (17.2.6-1296-g736ac5ba-1focal) ... 2023-10-06T03:27:22.476 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up python3-sklearn-lib (0.22.2.post1+dfsg-5) ... 2023-10-06T03:27:22.601 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up python3-pyperclip (1.7.0-1) ... 2023-10-06T03:27:22.822 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up libsqlite3-mod-ceph (17.2.6-1296-g736ac5ba-1focal) ... 2023-10-06T03:27:22.886 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up python3-markupsafe (1.1.0-1build2) ... 2023-10-06T03:27:22.948 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up libjq1:amd64 (1.6-1ubuntu0.20.04.1) ... 2023-10-06T03:27:23.074 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-pytest (4.6.9-1) ... 2023-10-06T03:27:23.163 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up python3-webencodings (0.5.1-1ubuntu1) ... 2023-10-06T03:27:23.459 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-openssl (19.0.0-1build1) ... 2023-10-06T03:27:23.464 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up libpcre2-16-0:amd64 (10.34-7ubuntu0.1) ... 2023-10-06T03:27:23.590 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up python3-psutil (5.5.1-1ubuntu4) ... 2023-10-06T03:27:23.760 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-pastedeploy (2.1.0-1) ... 2023-10-06T03:27:23.977 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up python3-natsort (7.0.1-1) ... 2023-10-06T03:27:24.045 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up libqt5network5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2023-10-06T03:27:24.171 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-bs4 (4.8.2-1) ... 2023-10-06T03:27:24.279 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up python3-atomicwrites (1.1.5-2build1) ... 2023-10-06T03:27:24.505 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up libcephfs2 (17.2.6-1296-g736ac5ba-1focal) ... 2023-10-06T03:27:24.555 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up python3-simplejson (3.16.0-2ubuntu2) ... 2023-10-06T03:27:24.664 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-oauthlib (3.1.0-1ubuntu2) ... 2023-10-06T03:27:24.923 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up python3-decorator (4.4.2-0ubuntu1) ... 2023-10-06T03:27:24.983 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up libradosstriper1 (17.2.6-1296-g736ac5ba-1focal) ... 2023-10-06T03:27:25.109 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-pyasn1-modules (0.2.1-0.2build1) ... 2023-10-06T03:27:25.224 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up python3-jinja2 (2.10.1-2) ... 2023-10-06T03:27:25.443 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-sklearn (0.22.2.post1+dfsg-5) ... 2023-10-06T03:27:25.575 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up python3-tempita (0.5.2-6) ... 2023-10-06T03:27:25.892 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up python3-packaging (20.3-1) ... 2023-10-06T03:27:26.197 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up python-pastedeploy-tpl (2.1.0-1) ... 2023-10-06T03:27:26.357 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up qttranslations5-l10n (5.12.8-0ubuntu1) ... 2023-10-06T03:27:26.483 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up python3-wcwidth (0.1.8+dfsg1-3) ... 2023-10-06T03:27:26.532 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up librbd1 (17.2.6-1296-g736ac5ba-1focal) ... 2023-10-06T03:27:26.658 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up libjs-sphinxdoc (1.8.5-7ubuntu3) ... 2023-10-06T03:27:26.776 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up python3-paste (3.4.0+dfsg1-1) ... 2023-10-06T03:27:26.784 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python-natsort-doc (7.0.1-1) ... 2023-10-06T03:27:26.910 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up jq (1.6-1ubuntu0.20.04.1) ... 2023-10-06T03:27:27.044 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-rsa (4.0-3) ... 2023-10-06T03:27:27.194 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up python3-cmd2 (0.8.5-2.1) ... 2023-10-06T03:27:27.369 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up libthrift-0.13.0 (0.13.0-2build2) ... 2023-10-06T03:27:27.496 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-webtest (2.0.32-2) ... 2023-10-06T03:27:27.529 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up python3-werkzeug (0.16.1+dfsg1-2ubuntu0.1) ... 2023-10-06T03:27:27.883 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up ceph-mgr-modules-core (17.2.6-1296-g736ac5ba-1focal) ... 2023-10-06T03:27:27.888 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up python3-cryptography (2.8-3ubuntu0.1) ... 2023-10-06T03:27:28.009 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up ceph-fuse (17.2.6-1296-g736ac5ba-1focal) ... 2023-10-06T03:27:28.233 INFO:teuthology.orchestra.run.smithi110.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2023-10-06T03:27:28.233 INFO:teuthology.orchestra.run.smithi110.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2023-10-06T03:27:28.251 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up socat (1.7.3.3-2) ... 2023-10-06T03:27:28.402 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up python3-ceph-common (17.2.6-1296-g736ac5ba-1focal) ... 2023-10-06T03:27:28.660 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up libcephfs-dev (17.2.6-1296-g736ac5ba-1focal) ... 2023-10-06T03:27:28.746 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up libqt5core5a:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2023-10-06T03:27:28.803 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-requests-oauthlib (1.0.0-1.1build1) ... 2023-10-06T03:27:28.880 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up python3-html5lib (1.0.1-2) ... 2023-10-06T03:27:29.087 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-pastescript (2.0.2-4) ... 2023-10-06T03:27:29.198 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up librdkafka1:amd64 (1.2.1-1ubuntu1) ... 2023-10-06T03:27:29.324 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up xmlstarlet (1.6.1-2build1) ... 2023-10-06T03:27:29.431 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-pecan (1.3.3-3ubuntu1) ... 2023-10-06T03:27:29.483 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up python3-pluggy (0.13.0-2) ... 2023-10-06T03:27:29.774 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-rados (17.2.6-1296-g736ac5ba-1focal) ... 2023-10-06T03:27:29.801 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up python3-lxml:amd64 (4.5.0-1ubuntu0.5) ... 2023-10-06T03:27:29.900 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up librgw2 (17.2.6-1296-g736ac5ba-1focal) ... 2023-10-06T03:27:30.026 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up cephfs-top (17.2.6-1296-g736ac5ba-1focal) ... 2023-10-06T03:27:30.152 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up libqt5dbus5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2023-10-06T03:27:30.186 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-rbd (17.2.6-1296-g736ac5ba-1focal) ... 2023-10-06T03:27:30.278 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up liblua5.3-0:amd64 (5.3.3-1.1ubuntu2) ... 2023-10-06T03:27:30.312 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up rbd-fuse (17.2.6-1296-g736ac5ba-1focal) ... 2023-10-06T03:27:30.404 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up python3-pyasn1 (0.4.2-3build1) ... 2023-10-06T03:27:30.438 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-routes (2.4.1-2) ... 2023-10-06T03:27:30.731 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-google-auth (1.5.1-2) ... 2023-10-06T03:27:30.756 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up python3-singledispatch (3.4.0.3-2build1) ... 2023-10-06T03:27:31.049 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-rgw (17.2.6-1296-g736ac5ba-1focal) ... 2023-10-06T03:27:31.065 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up python3-logutils (0.3.3-7) ... 2023-10-06T03:27:31.191 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-cephfs (17.2.6-1296-g736ac5ba-1focal) ... 2023-10-06T03:27:31.317 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up cephfs-shell (17.2.6-1296-g736ac5ba-1focal) ... 2023-10-06T03:27:31.383 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up liblttng-ust0:amd64 (2.11.0-1) ... 2023-10-06T03:27:31.443 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-kubernetes (7.0.0~a1-2) ... 2023-10-06T03:27:31.534 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2023-10-06T03:27:31.836 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up python3-prettytable (0.7.2-5) ... 2023-10-06T03:27:32.162 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up python3-soupsieve (1.9.5+dfsg-1) ... 2023-10-06T03:27:32.358 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up ceph-common (17.2.6-1296-g736ac5ba-1focal) ... 2023-10-06T03:27:32.496 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up python3-websocket (0.53.0-2ubuntu1) ... 2023-10-06T03:27:32.920 INFO:teuthology.orchestra.run.smithi192.stdout:update-alternatives: using /usr/bin/python3-wsdump to provide /usr/bin/wsdump (wsdump) in auto mode 2023-10-06T03:27:32.936 INFO:teuthology.orchestra.run.smithi110.stdout:Adding group ceph....done 2023-10-06T03:27:33.199 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up libonig5:amd64 (6.9.4-1) ... 2023-10-06T03:27:33.295 INFO:teuthology.orchestra.run.smithi110.stdout:Adding system user ceph....done 2023-10-06T03:27:33.358 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up python3-mako (1.1.0+ds1-1ubuntu2.1) ... 2023-10-06T03:27:33.471 INFO:teuthology.orchestra.run.smithi110.stdout:Setting system user ceph properties....done 2023-10-06T03:27:33.496 INFO:teuthology.orchestra.run.smithi110.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2023-10-06T03:27:33.650 INFO:teuthology.orchestra.run.smithi110.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2023-10-06T03:27:33.676 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up python3-cherrypy3 (8.9.1-6) ... 2023-10-06T03:27:33.963 INFO:teuthology.orchestra.run.smithi110.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2023-10-06T03:27:34.044 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up libjs-underscore (1.9.1~dfsg-1ubuntu0.20.04.1) ... 2023-10-06T03:27:34.170 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up liblbfgsb0:amd64 (3.0+dfsg.3-7build1) ... 2023-10-06T03:27:34.296 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up python3-webob (1:1.8.5-2) ... 2023-10-06T03:27:34.409 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up cephfs-mirror (17.2.6-1296-g736ac5ba-1focal) ... 2023-10-06T03:27:34.636 INFO:teuthology.orchestra.run.smithi110.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target -> /lib/systemd/system/cephfs-mirror.target. 2023-10-06T03:27:34.636 INFO:teuthology.orchestra.run.smithi110.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target -> /lib/systemd/system/cephfs-mirror.target. 2023-10-06T03:27:34.647 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up python3-blinker (1.4+dfsg1-0.3ubuntu1) ... 2023-10-06T03:27:34.974 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up librados2 (17.2.6-1296-g736ac5ba-1focal) ... 2023-10-06T03:27:35.069 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up ceph-test (17.2.6-1296-g736ac5ba-1focal) ... 2023-10-06T03:27:35.133 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up python3-scipy (1.3.3-3build1) ... 2023-10-06T03:27:35.170 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up radosgw (17.2.6-1296-g736ac5ba-1focal) ... 2023-10-06T03:27:35.434 INFO:teuthology.orchestra.run.smithi110.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2023-10-06T03:27:35.434 INFO:teuthology.orchestra.run.smithi110.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2023-10-06T03:27:36.057 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up ceph-base (17.2.6-1296-g736ac5ba-1focal) ... 2023-10-06T03:27:36.438 INFO:teuthology.orchestra.run.smithi110.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2023-10-06T03:27:36.614 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up libsqlite3-mod-ceph (17.2.6-1296-g736ac5ba-1focal) ... 2023-10-06T03:27:36.740 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up libjq1:amd64 (1.6-1ubuntu0.20.04.1) ... 2023-10-06T03:27:36.891 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up python3-pytest (4.6.9-1) ... 2023-10-06T03:27:36.892 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up ceph-mds (17.2.6-1296-g736ac5ba-1focal) ... 2023-10-06T03:27:37.118 INFO:teuthology.orchestra.run.smithi110.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2023-10-06T03:27:37.118 INFO:teuthology.orchestra.run.smithi110.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2023-10-06T03:27:37.276 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up python3-openssl (19.0.0-1build1) ... 2023-10-06T03:27:37.561 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up ceph-mgr (17.2.6-1296-g736ac5ba-1focal) ... 2023-10-06T03:27:37.586 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up python3-pastedeploy (2.1.0-1) ... 2023-10-06T03:27:37.788 INFO:teuthology.orchestra.run.smithi110.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2023-10-06T03:27:37.788 INFO:teuthology.orchestra.run.smithi110.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2023-10-06T03:27:37.895 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up libqt5network5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2023-10-06T03:27:38.029 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up python3-bs4 (4.8.2-1) ... 2023-10-06T03:27:38.221 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up ceph-osd (17.2.6-1296-g736ac5ba-1focal) ... 2023-10-06T03:27:38.372 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up libcephfs2 (17.2.6-1296-g736ac5ba-1focal) ... 2023-10-06T03:27:38.490 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up python3-oauthlib (3.1.0-1ubuntu2) ... 2023-10-06T03:27:38.735 INFO:teuthology.orchestra.run.smithi110.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2023-10-06T03:27:38.735 INFO:teuthology.orchestra.run.smithi110.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2023-10-06T03:27:38.842 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up libradosstriper1 (17.2.6-1296-g736ac5ba-1focal) ... 2023-10-06T03:27:39.001 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up python3-pyasn1-modules (0.2.1-0.2build1) ... 2023-10-06T03:27:39.181 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up ceph-mgr-k8sevents (17.2.6-1296-g736ac5ba-1focal) ... 2023-10-06T03:27:39.323 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up ceph-mgr-diskprediction-local (17.2.6-1296-g736ac5ba-1focal) ... 2023-10-06T03:27:39.336 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up python3-sklearn (0.22.2.post1+dfsg-5) ... 2023-10-06T03:27:39.466 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up ceph-mon (17.2.6-1296-g736ac5ba-1focal) ... 2023-10-06T03:27:39.689 INFO:teuthology.orchestra.run.smithi110.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2023-10-06T03:27:39.689 INFO:teuthology.orchestra.run.smithi110.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2023-10-06T03:27:40.126 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up ceph-mgr-cephadm (17.2.6-1296-g736ac5ba-1focal) ... 2023-10-06T03:27:40.303 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up ceph (17.2.6-1296-g736ac5ba-1focal) ... 2023-10-06T03:27:40.375 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up librbd1 (17.2.6-1296-g736ac5ba-1focal) ... 2023-10-06T03:27:40.428 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up ceph-mgr-dashboard (17.2.6-1296-g736ac5ba-1focal) ... 2023-10-06T03:27:40.505 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up libjs-sphinxdoc (1.8.5-7ubuntu3) ... 2023-10-06T03:27:40.571 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up ceph-volume (17.2.6-1296-g736ac5ba-1focal) ... 2023-10-06T03:27:40.627 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up python-natsort-doc (7.0.1-1) ... 2023-10-06T03:27:40.698 INFO:teuthology.orchestra.run.smithi110.stdout:Processing triggers for libc-bin (2.31-0ubuntu9.9) ... 2023-10-06T03:27:40.786 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up jq (1.6-1ubuntu0.20.04.1) ... 2023-10-06T03:27:40.849 INFO:teuthology.orchestra.run.smithi110.stdout:Processing triggers for systemd (245.4-4ubuntu3.17) ... 2023-10-06T03:27:40.929 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up python3-rsa (4.0-3) ... 2023-10-06T03:27:41.126 INFO:teuthology.orchestra.run.smithi110.stdout:Processing triggers for man-db (2.9.1-1) ... 2023-10-06T03:27:41.247 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up libthrift-0.13.0 (0.13.0-2build2) ... 2023-10-06T03:27:41.398 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up python3-webtest (2.0.32-2) ... 2023-10-06T03:27:41.776 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up ceph-mgr-modules-core (17.2.6-1296-g736ac5ba-1focal) ... 2023-10-06T03:27:41.903 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up ceph-fuse (17.2.6-1296-g736ac5ba-1focal) ... 2023-10-06T03:27:42.128 INFO:teuthology.orchestra.run.smithi192.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2023-10-06T03:27:42.128 INFO:teuthology.orchestra.run.smithi192.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2023-10-06T03:27:42.588 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up libcephfs-dev (17.2.6-1296-g736ac5ba-1focal) ... 2023-10-06T03:27:42.714 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up python3-requests-oauthlib (1.0.0-1.1build1) ... 2023-10-06T03:27:43.032 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up python3-pastescript (2.0.2-4) ... 2023-10-06T03:27:43.350 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up python3-pecan (1.3.3-3ubuntu1) ... 2023-10-06T03:27:43.712 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up python3-rados (17.2.6-1296-g736ac5ba-1focal) ... 2023-10-06T03:27:43.857 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up librgw2 (17.2.6-1296-g736ac5ba-1focal) ... 2023-10-06T03:27:43.983 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up cephfs-top (17.2.6-1296-g736ac5ba-1focal) ... 2023-10-06T03:27:44.062 INFO:teuthology.orchestra.run.smithi110.stdout:Processing triggers for fontconfig (2.13.1-2ubuntu3) ... 2023-10-06T03:27:44.109 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up python3-rbd (17.2.6-1296-g736ac5ba-1focal) ... 2023-10-06T03:27:44.260 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up rbd-fuse (17.2.6-1296-g736ac5ba-1focal) ... 2023-10-06T03:27:44.386 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up python3-routes (2.4.1-2) ... 2023-10-06T03:27:44.636 INFO:teuthology.orchestra.run.smithi110.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2023-10-06T03:27:44.638 DEBUG:teuthology.orchestra.run.smithi110:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install bison flex libelf-dev libssl-dev network-manager iproute2 util-linux dump indent libaio-dev libtool-bin uuid-dev xfslibs-dev 2023-10-06T03:27:44.679 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up python3-google-auth (1.5.1-2) ... 2023-10-06T03:27:44.708 INFO:teuthology.orchestra.run.smithi110.stdout:Reading package lists... 2023-10-06T03:27:44.911 INFO:teuthology.orchestra.run.smithi110.stdout:Building dependency tree... 2023-10-06T03:27:44.911 INFO:teuthology.orchestra.run.smithi110.stdout:Reading state information... 2023-10-06T03:27:44.989 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up python3-rgw (17.2.6-1296-g736ac5ba-1focal) ... 2023-10-06T03:27:45.106 INFO:teuthology.orchestra.run.smithi110.stdout:bison is already the newest version (2:3.5.1+dfsg-1). 2023-10-06T03:27:45.106 INFO:teuthology.orchestra.run.smithi110.stdout:flex is already the newest version (2.6.4-6.2). 2023-10-06T03:27:45.106 INFO:teuthology.orchestra.run.smithi110.stdout:iproute2 is already the newest version (5.5.0-1ubuntu1). 2023-10-06T03:27:45.106 INFO:teuthology.orchestra.run.smithi110.stdout:libtool-bin is already the newest version (2.4.6-14). 2023-10-06T03:27:45.106 INFO:teuthology.orchestra.run.smithi110.stdout:xfslibs-dev is already the newest version (5.3.0-1ubuntu2). 2023-10-06T03:27:45.106 INFO:teuthology.orchestra.run.smithi110.stdout:The following packages were automatically installed and are no longer required: 2023-10-06T03:27:45.107 INFO:teuthology.orchestra.run.smithi110.stdout: libboost-iostreams1.71.0 libboost-thread1.71.0 2023-10-06T03:27:45.107 INFO:teuthology.orchestra.run.smithi110.stdout:Use 'sudo apt autoremove' to remove them. 2023-10-06T03:27:45.107 INFO:teuthology.orchestra.run.smithi110.stdout:The following additional packages will be installed: 2023-10-06T03:27:45.107 INFO:teuthology.orchestra.run.smithi110.stdout: libbluetooth3 libdw1 libelf1 liblzo2-2 libmbim-glib4 libmbim-proxy 2023-10-06T03:27:45.107 INFO:teuthology.orchestra.run.smithi110.stdout: libmm-glib0 libndp0 libnm0 libqmi-glib5 libqmi-proxy libssl1.1 libteamdctl0 2023-10-06T03:27:45.108 INFO:teuthology.orchestra.run.smithi110.stdout: libuuid1 modemmanager network-manager-pptp ppp pptp-linux usb-modeswitch 2023-10-06T03:27:45.108 INFO:teuthology.orchestra.run.smithi110.stdout: usb-modeswitch-data wpasupplicant 2023-10-06T03:27:45.109 INFO:teuthology.orchestra.run.smithi110.stdout:Suggested packages: 2023-10-06T03:27:45.109 INFO:teuthology.orchestra.run.smithi110.stdout: indent-doc libssl-doc avahi-autoipd libteam-utils comgt wvdial 2023-10-06T03:27:45.109 INFO:teuthology.orchestra.run.smithi110.stdout: util-linux-locales wpagui libengine-pkcs11-openssl 2023-10-06T03:27:45.115 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up python3-cephfs (17.2.6-1296-g736ac5ba-1focal) ... 2023-10-06T03:27:45.157 INFO:teuthology.orchestra.run.smithi110.stdout:The following NEW packages will be installed: 2023-10-06T03:27:45.158 INFO:teuthology.orchestra.run.smithi110.stdout: dump indent libaio-dev libbluetooth3 libelf-dev liblzo2-2 libmbim-glib4 2023-10-06T03:27:45.158 INFO:teuthology.orchestra.run.smithi110.stdout: libmbim-proxy libmm-glib0 libndp0 libnm0 libqmi-glib5 libqmi-proxy 2023-10-06T03:27:45.158 INFO:teuthology.orchestra.run.smithi110.stdout: libssl-dev libteamdctl0 modemmanager network-manager network-manager-pptp 2023-10-06T03:27:45.159 INFO:teuthology.orchestra.run.smithi110.stdout: ppp pptp-linux usb-modeswitch usb-modeswitch-data wpasupplicant 2023-10-06T03:27:45.159 INFO:teuthology.orchestra.run.smithi110.stdout:The following packages will be upgraded: 2023-10-06T03:27:45.160 INFO:teuthology.orchestra.run.smithi110.stdout: libdw1 libelf1 libssl1.1 libuuid1 util-linux uuid-dev 2023-10-06T03:27:45.274 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up cephfs-shell (17.2.6-1296-g736ac5ba-1focal) ... 2023-10-06T03:27:45.343 INFO:teuthology.orchestra.run.smithi110.stdout:6 upgraded, 23 newly installed, 0 to remove and 247 not upgraded. 2023-10-06T03:27:45.344 INFO:teuthology.orchestra.run.smithi110.stdout:Need to get 10.4 MB of archives. 2023-10-06T03:27:45.344 INFO:teuthology.orchestra.run.smithi110.stdout:After this operation, 33.5 MB of additional disk space will be used. 2023-10-06T03:27:45.344 INFO:teuthology.orchestra.run.smithi110.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 util-linux amd64 2.34-0.1ubuntu9.4 [1,021 kB] 2023-10-06T03:27:45.400 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up python3-kubernetes (7.0.0~a1-2) ... 2023-10-06T03:27:45.966 INFO:teuthology.orchestra.run.smithi110.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 uuid-dev amd64 2.34-0.1ubuntu9.4 [33.6 kB] 2023-10-06T03:27:45.968 INFO:teuthology.orchestra.run.smithi110.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libuuid1 amd64 2.34-0.1ubuntu9.4 [20.0 kB] 2023-10-06T03:27:45.969 INFO:teuthology.orchestra.run.smithi110.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libdw1 amd64 0.176-1.1ubuntu0.1 [226 kB] 2023-10-06T03:27:45.982 INFO:teuthology.orchestra.run.smithi110.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libelf1 amd64 0.176-1.1ubuntu0.1 [44.2 kB] 2023-10-06T03:27:45.984 INFO:teuthology.orchestra.run.smithi110.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libssl1.1 amd64 1.1.1f-1ubuntu2.19 [1,321 kB] 2023-10-06T03:27:46.082 INFO:teuthology.orchestra.run.smithi110.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal/main amd64 liblzo2-2 amd64 2.10-2 [50.8 kB] 2023-10-06T03:27:46.085 INFO:teuthology.orchestra.run.smithi110.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal/universe amd64 dump amd64 0.4b46-6 [130 kB] 2023-10-06T03:27:46.092 INFO:teuthology.orchestra.run.smithi110.stdout:Get:9 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 indent amd64 2.2.12-1ubuntu0.20.04.1 [86.6 kB] 2023-10-06T03:27:46.097 INFO:teuthology.orchestra.run.smithi110.stdout:Get:10 http://archive.ubuntu.com/ubuntu focal/main amd64 libaio-dev amd64 0.3.112-5 [13.7 kB] 2023-10-06T03:27:46.098 INFO:teuthology.orchestra.run.smithi110.stdout:Get:11 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libbluetooth3 amd64 5.53-0ubuntu3.6 [60.6 kB] 2023-10-06T03:27:46.264 INFO:teuthology.orchestra.run.smithi110.stdout:Get:12 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libelf-dev amd64 0.176-1.1ubuntu0.1 [57.1 kB] 2023-10-06T03:27:46.337 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up ceph-common (17.2.6-1296-g736ac5ba-1focal) ... 2023-10-06T03:27:46.533 INFO:teuthology.orchestra.run.smithi110.stdout:Get:13 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libmbim-glib4 amd64 1.26.2-1~ubuntu20.04.1 [123 kB] 2023-10-06T03:27:46.661 INFO:teuthology.orchestra.run.smithi110.stdout:Get:14 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libmbim-proxy amd64 1.26.2-1~ubuntu20.04.1 [6,012 B] 2023-10-06T03:27:46.667 INFO:teuthology.orchestra.run.smithi110.stdout:Get:15 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libmm-glib0 amd64 1.18.6-1~ubuntu20.04.1 [203 kB] 2023-10-06T03:27:46.763 INFO:teuthology.orchestra.run.smithi110.stdout:Get:16 http://archive.ubuntu.com/ubuntu focal/main amd64 libndp0 amd64 1.7-0ubuntu1 [10.9 kB] 2023-10-06T03:27:46.766 INFO:teuthology.orchestra.run.smithi110.stdout:Get:17 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libnm0 amd64 1.22.10-1ubuntu2.3 [370 kB] 2023-10-06T03:27:46.848 INFO:teuthology.orchestra.run.smithi110.stdout:Get:18 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libqmi-glib5 amd64 1.30.4-1~ubuntu20.04.1 [642 kB] 2023-10-06T03:27:46.900 INFO:teuthology.orchestra.run.smithi192.stdout:Adding group ceph....done 2023-10-06T03:27:46.922 INFO:teuthology.orchestra.run.smithi110.stdout:Get:19 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libqmi-proxy amd64 1.30.4-1~ubuntu20.04.1 [5,908 B] 2023-10-06T03:27:46.922 INFO:teuthology.orchestra.run.smithi110.stdout:Get:20 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libssl-dev amd64 1.1.1f-1ubuntu2.19 [1,584 kB] 2023-10-06T03:27:47.012 INFO:teuthology.orchestra.run.smithi110.stdout:Get:21 http://archive.ubuntu.com/ubuntu focal/main amd64 libteamdctl0 amd64 1.30-1 [11.8 kB] 2023-10-06T03:27:47.012 INFO:teuthology.orchestra.run.smithi110.stdout:Get:22 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 modemmanager amd64 1.18.6-1~ubuntu20.04.1 [895 kB] 2023-10-06T03:27:47.206 INFO:teuthology.orchestra.run.smithi110.stdout:Get:23 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 wpasupplicant amd64 2:2.9-1ubuntu4.3 [1,183 kB] 2023-10-06T03:27:47.303 INFO:teuthology.orchestra.run.smithi192.stdout:Adding system user ceph....done 2023-10-06T03:27:47.512 INFO:teuthology.orchestra.run.smithi192.stdout:Setting system user ceph properties....done 2023-10-06T03:27:47.534 INFO:teuthology.orchestra.run.smithi192.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2023-10-06T03:27:47.675 INFO:teuthology.orchestra.run.smithi192.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2023-10-06T03:27:47.847 INFO:teuthology.orchestra.run.smithi110.stdout:Get:24 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 network-manager amd64 1.22.10-1ubuntu2.3 [1,855 kB] 2023-10-06T03:27:47.960 INFO:teuthology.orchestra.run.smithi110.stdout:Get:25 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 ppp amd64 2.4.7-2+4.1ubuntu5.1 [328 kB] 2023-10-06T03:27:47.975 INFO:teuthology.orchestra.run.smithi110.stdout:Get:26 http://archive.ubuntu.com/ubuntu focal/main amd64 pptp-linux amd64 1.10.0-1build1 [42.0 kB] 2023-10-06T03:27:47.977 INFO:teuthology.orchestra.run.smithi110.stdout:Get:27 http://archive.ubuntu.com/ubuntu focal/main amd64 network-manager-pptp amd64 1.2.8-2 [30.1 kB] 2023-10-06T03:27:47.978 INFO:teuthology.orchestra.run.smithi110.stdout:Get:28 http://archive.ubuntu.com/ubuntu focal/main amd64 usb-modeswitch-data all 20191128-3 [32.3 kB] 2023-10-06T03:27:47.979 INFO:teuthology.orchestra.run.smithi110.stdout:Get:29 http://archive.ubuntu.com/ubuntu focal/main amd64 usb-modeswitch amd64 2.5.2+repack0-2ubuntu3 [53.1 kB] 2023-10-06T03:27:48.001 INFO:teuthology.orchestra.run.smithi192.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2023-10-06T03:27:48.305 INFO:teuthology.orchestra.run.smithi110.stdout:Preconfiguring packages ... 2023-10-06T03:27:48.438 INFO:teuthology.orchestra.run.smithi110.stdout:Fetched 10.4 MB in 3s (3,708 kB/s) 2023-10-06T03:27:48.475 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up cephfs-mirror (17.2.6-1296-g736ac5ba-1focal) ... 2023-10-06T03:27:48.694 INFO:teuthology.orchestra.run.smithi192.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target -> /lib/systemd/system/cephfs-mirror.target. 2023-10-06T03:27:48.694 INFO:teuthology.orchestra.run.smithi192.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target -> /lib/systemd/system/cephfs-mirror.target. 2023-10-06T03:27:48.720 INFO:teuthology.orchestra.run.smithi110.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 152664 files and directories currently installed.) 2023-10-06T03:27:48.729 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../util-linux_2.34-0.1ubuntu9.4_amd64.deb ... 2023-10-06T03:27:48.881 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking util-linux (2.34-0.1ubuntu9.4) over (2.34-0.1ubuntu9.3) ... 2023-10-06T03:27:49.160 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up ceph-test (17.2.6-1296-g736ac5ba-1focal) ... 2023-10-06T03:27:49.286 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up radosgw (17.2.6-1296-g736ac5ba-1focal) ... 2023-10-06T03:27:49.550 INFO:teuthology.orchestra.run.smithi192.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2023-10-06T03:27:49.658 INFO:teuthology.orchestra.run.smithi192.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2023-10-06T03:27:49.978 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up util-linux (2.34-0.1ubuntu9.4) ... 2023-10-06T03:27:50.223 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up ceph-base (17.2.6-1296-g736ac5ba-1focal) ... 2023-10-06T03:27:50.600 INFO:teuthology.orchestra.run.smithi192.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2023-10-06T03:27:51.058 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up ceph-mds (17.2.6-1296-g736ac5ba-1focal) ... 2023-10-06T03:27:51.275 INFO:teuthology.orchestra.run.smithi192.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2023-10-06T03:27:51.275 INFO:teuthology.orchestra.run.smithi192.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2023-10-06T03:27:51.426 INFO:teuthology.orchestra.run.smithi110.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 152664 files and directories currently installed.) 2023-10-06T03:27:51.430 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../uuid-dev_2.34-0.1ubuntu9.4_amd64.deb ... 2023-10-06T03:27:51.561 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking uuid-dev:amd64 (2.34-0.1ubuntu9.4) over (2.34-0.1ubuntu9.3) ... 2023-10-06T03:27:51.701 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up ceph-mgr (17.2.6-1296-g736ac5ba-1focal) ... 2023-10-06T03:27:51.923 INFO:teuthology.orchestra.run.smithi192.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2023-10-06T03:27:51.923 INFO:teuthology.orchestra.run.smithi192.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2023-10-06T03:27:52.273 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../libuuid1_2.34-0.1ubuntu9.4_amd64.deb ... 2023-10-06T03:27:52.371 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up ceph-osd (17.2.6-1296-g736ac5ba-1focal) ... 2023-10-06T03:27:52.502 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking libuuid1:amd64 (2.34-0.1ubuntu9.4) over (2.34-0.1ubuntu9.3) ... 2023-10-06T03:27:52.825 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up libuuid1:amd64 (2.34-0.1ubuntu9.4) ... 2023-10-06T03:27:53.143 INFO:teuthology.orchestra.run.smithi192.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2023-10-06T03:27:53.144 INFO:teuthology.orchestra.run.smithi192.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2023-10-06T03:27:53.282 INFO:teuthology.orchestra.run.smithi110.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 152664 files and directories currently installed.) 2023-10-06T03:27:53.287 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../00-libdw1_0.176-1.1ubuntu0.1_amd64.deb ... 2023-10-06T03:27:53.426 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking libdw1:amd64 (0.176-1.1ubuntu0.1) over (0.176-1.1build1) ... 2023-10-06T03:27:53.582 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up ceph-mgr-k8sevents (17.2.6-1296-g736ac5ba-1focal) ... 2023-10-06T03:27:53.708 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up ceph-mgr-diskprediction-local (17.2.6-1296-g736ac5ba-1focal) ... 2023-10-06T03:27:53.884 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up ceph-mon (17.2.6-1296-g736ac5ba-1focal) ... 2023-10-06T03:27:53.914 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../01-libelf1_0.176-1.1ubuntu0.1_amd64.deb ... 2023-10-06T03:27:54.047 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking libelf1:amd64 (0.176-1.1ubuntu0.1) over (0.176-1.1build1) ... 2023-10-06T03:27:54.108 INFO:teuthology.orchestra.run.smithi192.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2023-10-06T03:27:54.108 INFO:teuthology.orchestra.run.smithi192.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2023-10-06T03:27:54.518 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../02-libssl1.1_1.1.1f-1ubuntu2.19_amd64.deb ... 2023-10-06T03:27:54.560 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up ceph-mgr-cephadm (17.2.6-1296-g736ac5ba-1focal) ... 2023-10-06T03:27:54.660 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking libssl1.1:amd64 (1.1.1f-1ubuntu2.19) over (1.1.1f-1ubuntu2.16) ... 2023-10-06T03:27:54.703 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up ceph (17.2.6-1296-g736ac5ba-1focal) ... 2023-10-06T03:27:54.862 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up ceph-mgr-dashboard (17.2.6-1296-g736ac5ba-1focal) ... 2023-10-06T03:27:55.005 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up ceph-volume (17.2.6-1296-g736ac5ba-1focal) ... 2023-10-06T03:27:55.132 INFO:teuthology.orchestra.run.smithi192.stdout:Processing triggers for libc-bin (2.31-0ubuntu9.9) ... 2023-10-06T03:27:55.172 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package liblzo2-2:amd64. 2023-10-06T03:27:55.191 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../03-liblzo2-2_2.10-2_amd64.deb ... 2023-10-06T03:27:55.231 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking liblzo2-2:amd64 (2.10-2) ... 2023-10-06T03:27:55.308 INFO:teuthology.orchestra.run.smithi192.stdout:Processing triggers for systemd (245.4-4ubuntu3.17) ... 2023-10-06T03:27:55.567 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package dump. 2023-10-06T03:27:55.586 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../04-dump_0.4b46-6_amd64.deb ... 2023-10-06T03:27:55.601 INFO:teuthology.orchestra.run.smithi192.stdout:Processing triggers for man-db (2.9.1-1) ... 2023-10-06T03:27:55.625 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking dump (0.4b46-6) ... 2023-10-06T03:27:55.954 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package indent. 2023-10-06T03:27:55.973 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../05-indent_2.2.12-1ubuntu0.20.04.1_amd64.deb ... 2023-10-06T03:27:56.012 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking indent (2.2.12-1ubuntu0.20.04.1) ... 2023-10-06T03:27:56.331 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package libaio-dev:amd64. 2023-10-06T03:27:56.351 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../06-libaio-dev_0.3.112-5_amd64.deb ... 2023-10-06T03:27:56.390 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking libaio-dev:amd64 (0.3.112-5) ... 2023-10-06T03:27:56.734 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package libbluetooth3:amd64. 2023-10-06T03:27:56.751 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../07-libbluetooth3_5.53-0ubuntu3.6_amd64.deb ... 2023-10-06T03:27:56.793 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking libbluetooth3:amd64 (5.53-0ubuntu3.6) ... 2023-10-06T03:27:57.062 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package libelf-dev:amd64. 2023-10-06T03:27:57.081 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../08-libelf-dev_0.176-1.1ubuntu0.1_amd64.deb ... 2023-10-06T03:27:57.120 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking libelf-dev:amd64 (0.176-1.1ubuntu0.1) ... 2023-10-06T03:27:57.478 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package libmbim-glib4:amd64. 2023-10-06T03:27:57.497 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../09-libmbim-glib4_1.26.2-1~ubuntu20.04.1_amd64.deb ... 2023-10-06T03:27:57.758 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking libmbim-glib4:amd64 (1.26.2-1~ubuntu20.04.1) ... 2023-10-06T03:27:58.103 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package libmbim-proxy. 2023-10-06T03:27:58.119 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../10-libmbim-proxy_1.26.2-1~ubuntu20.04.1_amd64.deb ... 2023-10-06T03:27:58.153 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking libmbim-proxy (1.26.2-1~ubuntu20.04.1) ... 2023-10-06T03:27:58.463 INFO:teuthology.orchestra.run.smithi192.stdout:Processing triggers for fontconfig (2.13.1-2ubuntu3) ... 2023-10-06T03:27:58.480 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package libmm-glib0:amd64. 2023-10-06T03:27:58.498 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../11-libmm-glib0_1.18.6-1~ubuntu20.04.1_amd64.deb ... 2023-10-06T03:27:58.538 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking libmm-glib0:amd64 (1.18.6-1~ubuntu20.04.1) ... 2023-10-06T03:27:58.908 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package libndp0:amd64. 2023-10-06T03:27:58.926 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../12-libndp0_1.7-0ubuntu1_amd64.deb ... 2023-10-06T03:27:58.966 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking libndp0:amd64 (1.7-0ubuntu1) ... 2023-10-06T03:27:59.059 INFO:teuthology.orchestra.run.smithi192.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2023-10-06T03:27:59.062 DEBUG:teuthology.orchestra.run.smithi192:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install bison flex libelf-dev libssl-dev network-manager iproute2 util-linux dump indent libaio-dev libtool-bin uuid-dev xfslibs-dev 2023-10-06T03:27:59.133 INFO:teuthology.orchestra.run.smithi192.stdout:Reading package lists... 2023-10-06T03:27:59.269 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package libnm0:amd64. 2023-10-06T03:27:59.287 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../13-libnm0_1.22.10-1ubuntu2.3_amd64.deb ... 2023-10-06T03:27:59.338 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking libnm0:amd64 (1.22.10-1ubuntu2.3) ... 2023-10-06T03:27:59.338 INFO:teuthology.orchestra.run.smithi192.stdout:Building dependency tree... 2023-10-06T03:27:59.339 INFO:teuthology.orchestra.run.smithi192.stdout:Reading state information... 2023-10-06T03:27:59.539 INFO:teuthology.orchestra.run.smithi192.stdout:bison is already the newest version (2:3.5.1+dfsg-1). 2023-10-06T03:27:59.540 INFO:teuthology.orchestra.run.smithi192.stdout:flex is already the newest version (2.6.4-6.2). 2023-10-06T03:27:59.540 INFO:teuthology.orchestra.run.smithi192.stdout:iproute2 is already the newest version (5.5.0-1ubuntu1). 2023-10-06T03:27:59.540 INFO:teuthology.orchestra.run.smithi192.stdout:libtool-bin is already the newest version (2.4.6-14). 2023-10-06T03:27:59.540 INFO:teuthology.orchestra.run.smithi192.stdout:xfslibs-dev is already the newest version (5.3.0-1ubuntu2). 2023-10-06T03:27:59.540 INFO:teuthology.orchestra.run.smithi192.stdout:The following packages were automatically installed and are no longer required: 2023-10-06T03:27:59.541 INFO:teuthology.orchestra.run.smithi192.stdout: libboost-iostreams1.71.0 libboost-thread1.71.0 2023-10-06T03:27:59.541 INFO:teuthology.orchestra.run.smithi192.stdout:Use 'sudo apt autoremove' to remove them. 2023-10-06T03:27:59.541 INFO:teuthology.orchestra.run.smithi192.stdout:The following additional packages will be installed: 2023-10-06T03:27:59.541 INFO:teuthology.orchestra.run.smithi192.stdout: libbluetooth3 libdw1 libelf1 liblzo2-2 libmbim-glib4 libmbim-proxy 2023-10-06T03:27:59.541 INFO:teuthology.orchestra.run.smithi192.stdout: libmm-glib0 libndp0 libnm0 libqmi-glib5 libqmi-proxy libssl1.1 libteamdctl0 2023-10-06T03:27:59.542 INFO:teuthology.orchestra.run.smithi192.stdout: libuuid1 modemmanager network-manager-pptp ppp pptp-linux usb-modeswitch 2023-10-06T03:27:59.542 INFO:teuthology.orchestra.run.smithi192.stdout: usb-modeswitch-data wpasupplicant 2023-10-06T03:27:59.543 INFO:teuthology.orchestra.run.smithi192.stdout:Suggested packages: 2023-10-06T03:27:59.543 INFO:teuthology.orchestra.run.smithi192.stdout: indent-doc libssl-doc avahi-autoipd libteam-utils comgt wvdial 2023-10-06T03:27:59.543 INFO:teuthology.orchestra.run.smithi192.stdout: util-linux-locales wpagui libengine-pkcs11-openssl 2023-10-06T03:27:59.588 INFO:teuthology.orchestra.run.smithi192.stdout:The following NEW packages will be installed: 2023-10-06T03:27:59.589 INFO:teuthology.orchestra.run.smithi192.stdout: dump indent libaio-dev libbluetooth3 libelf-dev liblzo2-2 libmbim-glib4 2023-10-06T03:27:59.589 INFO:teuthology.orchestra.run.smithi192.stdout: libmbim-proxy libmm-glib0 libndp0 libnm0 libqmi-glib5 libqmi-proxy 2023-10-06T03:27:59.590 INFO:teuthology.orchestra.run.smithi192.stdout: libssl-dev libteamdctl0 modemmanager network-manager network-manager-pptp 2023-10-06T03:27:59.590 INFO:teuthology.orchestra.run.smithi192.stdout: ppp pptp-linux usb-modeswitch usb-modeswitch-data wpasupplicant 2023-10-06T03:27:59.590 INFO:teuthology.orchestra.run.smithi192.stdout:The following packages will be upgraded: 2023-10-06T03:27:59.591 INFO:teuthology.orchestra.run.smithi192.stdout: libdw1 libelf1 libssl1.1 libuuid1 util-linux uuid-dev 2023-10-06T03:27:59.716 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package libqmi-glib5:amd64. 2023-10-06T03:27:59.733 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../14-libqmi-glib5_1.30.4-1~ubuntu20.04.1_amd64.deb ... 2023-10-06T03:27:59.773 INFO:teuthology.orchestra.run.smithi192.stdout:6 upgraded, 23 newly installed, 0 to remove and 247 not upgraded. 2023-10-06T03:27:59.773 INFO:teuthology.orchestra.run.smithi192.stdout:Need to get 10.4 MB of archives. 2023-10-06T03:27:59.774 INFO:teuthology.orchestra.run.smithi192.stdout:After this operation, 33.5 MB of additional disk space will be used. 2023-10-06T03:27:59.774 INFO:teuthology.orchestra.run.smithi192.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 util-linux amd64 2.34-0.1ubuntu9.4 [1,021 kB] 2023-10-06T03:27:59.774 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking libqmi-glib5:amd64 (1.30.4-1~ubuntu20.04.1) ... 2023-10-06T03:28:00.110 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package libqmi-proxy. 2023-10-06T03:28:00.129 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../15-libqmi-proxy_1.30.4-1~ubuntu20.04.1_amd64.deb ... 2023-10-06T03:28:00.168 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking libqmi-proxy (1.30.4-1~ubuntu20.04.1) ... 2023-10-06T03:28:00.395 INFO:teuthology.orchestra.run.smithi192.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 uuid-dev amd64 2.34-0.1ubuntu9.4 [33.6 kB] 2023-10-06T03:28:00.397 INFO:teuthology.orchestra.run.smithi192.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libuuid1 amd64 2.34-0.1ubuntu9.4 [20.0 kB] 2023-10-06T03:28:00.398 INFO:teuthology.orchestra.run.smithi192.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libdw1 amd64 0.176-1.1ubuntu0.1 [226 kB] 2023-10-06T03:28:00.411 INFO:teuthology.orchestra.run.smithi192.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libelf1 amd64 0.176-1.1ubuntu0.1 [44.2 kB] 2023-10-06T03:28:00.413 INFO:teuthology.orchestra.run.smithi192.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libssl1.1 amd64 1.1.1f-1ubuntu2.19 [1,321 kB] 2023-10-06T03:28:00.437 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package libssl-dev:amd64. 2023-10-06T03:28:00.454 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../16-libssl-dev_1.1.1f-1ubuntu2.19_amd64.deb ... 2023-10-06T03:28:00.487 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking libssl-dev:amd64 (1.1.1f-1ubuntu2.19) ... 2023-10-06T03:28:00.512 INFO:teuthology.orchestra.run.smithi192.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal/main amd64 liblzo2-2 amd64 2.10-2 [50.8 kB] 2023-10-06T03:28:00.514 INFO:teuthology.orchestra.run.smithi192.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal/universe amd64 dump amd64 0.4b46-6 [130 kB] 2023-10-06T03:28:00.521 INFO:teuthology.orchestra.run.smithi192.stdout:Get:9 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 indent amd64 2.2.12-1ubuntu0.20.04.1 [86.6 kB] 2023-10-06T03:28:00.526 INFO:teuthology.orchestra.run.smithi192.stdout:Get:10 http://archive.ubuntu.com/ubuntu focal/main amd64 libaio-dev amd64 0.3.112-5 [13.7 kB] 2023-10-06T03:28:00.527 INFO:teuthology.orchestra.run.smithi192.stdout:Get:11 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libbluetooth3 amd64 5.53-0ubuntu3.6 [60.6 kB] 2023-10-06T03:28:00.694 INFO:teuthology.orchestra.run.smithi192.stdout:Get:12 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libelf-dev amd64 0.176-1.1ubuntu0.1 [57.1 kB] 2023-10-06T03:28:00.964 INFO:teuthology.orchestra.run.smithi192.stdout:Get:13 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libmbim-glib4 amd64 1.26.2-1~ubuntu20.04.1 [123 kB] 2023-10-06T03:28:00.982 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package libteamdctl0:amd64. 2023-10-06T03:28:00.999 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../17-libteamdctl0_1.30-1_amd64.deb ... 2023-10-06T03:28:01.041 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking libteamdctl0:amd64 (1.30-1) ... 2023-10-06T03:28:01.092 INFO:teuthology.orchestra.run.smithi192.stdout:Get:14 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libmbim-proxy amd64 1.26.2-1~ubuntu20.04.1 [6,012 B] 2023-10-06T03:28:01.098 INFO:teuthology.orchestra.run.smithi192.stdout:Get:15 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libmm-glib0 amd64 1.18.6-1~ubuntu20.04.1 [203 kB] 2023-10-06T03:28:01.192 INFO:teuthology.orchestra.run.smithi192.stdout:Get:16 http://archive.ubuntu.com/ubuntu focal/main amd64 libndp0 amd64 1.7-0ubuntu1 [10.9 kB] 2023-10-06T03:28:01.197 INFO:teuthology.orchestra.run.smithi192.stdout:Get:17 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libnm0 amd64 1.22.10-1ubuntu2.3 [370 kB] 2023-10-06T03:28:01.279 INFO:teuthology.orchestra.run.smithi192.stdout:Get:18 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libqmi-glib5 amd64 1.30.4-1~ubuntu20.04.1 [642 kB] 2023-10-06T03:28:01.353 INFO:teuthology.orchestra.run.smithi192.stdout:Get:19 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libqmi-proxy amd64 1.30.4-1~ubuntu20.04.1 [5,908 B] 2023-10-06T03:28:01.354 INFO:teuthology.orchestra.run.smithi192.stdout:Get:20 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libssl-dev amd64 1.1.1f-1ubuntu2.19 [1,584 kB] 2023-10-06T03:28:01.377 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package modemmanager. 2023-10-06T03:28:01.397 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../18-modemmanager_1.18.6-1~ubuntu20.04.1_amd64.deb ... 2023-10-06T03:28:01.437 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking modemmanager (1.18.6-1~ubuntu20.04.1) ... 2023-10-06T03:28:01.443 INFO:teuthology.orchestra.run.smithi192.stdout:Get:21 http://archive.ubuntu.com/ubuntu focal/main amd64 libteamdctl0 amd64 1.30-1 [11.8 kB] 2023-10-06T03:28:01.443 INFO:teuthology.orchestra.run.smithi192.stdout:Get:22 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 modemmanager amd64 1.18.6-1~ubuntu20.04.1 [895 kB] 2023-10-06T03:28:01.638 INFO:teuthology.orchestra.run.smithi192.stdout:Get:23 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 wpasupplicant amd64 2:2.9-1ubuntu4.3 [1,183 kB] 2023-10-06T03:28:02.058 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package wpasupplicant. 2023-10-06T03:28:02.075 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../19-wpasupplicant_2%3a2.9-1ubuntu4.3_amd64.deb ... 2023-10-06T03:28:02.116 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking wpasupplicant (2:2.9-1ubuntu4.3) ... 2023-10-06T03:28:02.279 INFO:teuthology.orchestra.run.smithi192.stdout:Get:24 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 network-manager amd64 1.22.10-1ubuntu2.3 [1,855 kB] 2023-10-06T03:28:02.393 INFO:teuthology.orchestra.run.smithi192.stdout:Get:25 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 ppp amd64 2.4.7-2+4.1ubuntu5.1 [328 kB] 2023-10-06T03:28:02.408 INFO:teuthology.orchestra.run.smithi192.stdout:Get:26 http://archive.ubuntu.com/ubuntu focal/main amd64 pptp-linux amd64 1.10.0-1build1 [42.0 kB] 2023-10-06T03:28:02.410 INFO:teuthology.orchestra.run.smithi192.stdout:Get:27 http://archive.ubuntu.com/ubuntu focal/main amd64 network-manager-pptp amd64 1.2.8-2 [30.1 kB] 2023-10-06T03:28:02.411 INFO:teuthology.orchestra.run.smithi192.stdout:Get:28 http://archive.ubuntu.com/ubuntu focal/main amd64 usb-modeswitch-data all 20191128-3 [32.3 kB] 2023-10-06T03:28:02.413 INFO:teuthology.orchestra.run.smithi192.stdout:Get:29 http://archive.ubuntu.com/ubuntu focal/main amd64 usb-modeswitch amd64 2.5.2+repack0-2ubuntu3 [53.1 kB] 2023-10-06T03:28:02.663 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package network-manager. 2023-10-06T03:28:02.680 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../20-network-manager_1.22.10-1ubuntu2.3_amd64.deb ... 2023-10-06T03:28:02.741 INFO:teuthology.orchestra.run.smithi192.stdout:Preconfiguring packages ... 2023-10-06T03:28:02.746 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking network-manager (1.22.10-1ubuntu2.3) ... 2023-10-06T03:28:02.880 INFO:teuthology.orchestra.run.smithi192.stdout:Fetched 10.4 MB in 3s (3,706 kB/s) 2023-10-06T03:28:03.172 INFO:teuthology.orchestra.run.smithi192.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 152664 files and directories currently installed.) 2023-10-06T03:28:03.177 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../util-linux_2.34-0.1ubuntu9.4_amd64.deb ... 2023-10-06T03:28:03.432 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking util-linux (2.34-0.1ubuntu9.4) over (2.34-0.1ubuntu9.3) ... 2023-10-06T03:28:03.806 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package ppp. 2023-10-06T03:28:03.823 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../21-ppp_2.4.7-2+4.1ubuntu5.1_amd64.deb ... 2023-10-06T03:28:04.110 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking ppp (2.4.7-2+4.1ubuntu5.1) ... 2023-10-06T03:28:04.453 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package pptp-linux. 2023-10-06T03:28:04.471 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../22-pptp-linux_1.10.0-1build1_amd64.deb ... 2023-10-06T03:28:04.511 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking pptp-linux (1.10.0-1build1) ... 2023-10-06T03:28:04.562 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up util-linux (2.34-0.1ubuntu9.4) ... 2023-10-06T03:28:04.914 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package network-manager-pptp. 2023-10-06T03:28:04.931 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../23-network-manager-pptp_1.2.8-2_amd64.deb ... 2023-10-06T03:28:04.990 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking network-manager-pptp (1.2.8-2) ... 2023-10-06T03:28:05.226 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package usb-modeswitch-data. 2023-10-06T03:28:05.243 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../24-usb-modeswitch-data_20191128-3_all.deb ... 2023-10-06T03:28:05.293 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking usb-modeswitch-data (20191128-3) ... 2023-10-06T03:28:05.621 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package usb-modeswitch. 2023-10-06T03:28:05.638 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../25-usb-modeswitch_2.5.2+repack0-2ubuntu3_amd64.deb ... 2023-10-06T03:28:05.706 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking usb-modeswitch (2.5.2+repack0-2ubuntu3) ... 2023-10-06T03:28:05.985 INFO:teuthology.orchestra.run.smithi192.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 152664 files and directories currently installed.) 2023-10-06T03:28:05.990 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../uuid-dev_2.34-0.1ubuntu9.4_amd64.deb ... 2023-10-06T03:28:06.006 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up libssl1.1:amd64 (1.1.1f-1ubuntu2.19) ... 2023-10-06T03:28:06.154 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking uuid-dev:amd64 (2.34-0.1ubuntu9.4) over (2.34-0.1ubuntu9.3) ... 2023-10-06T03:28:06.301 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up liblzo2-2:amd64 (2.10-2) ... 2023-10-06T03:28:06.427 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up libaio-dev:amd64 (0.3.112-5) ... 2023-10-06T03:28:06.569 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up wpasupplicant (2:2.9-1ubuntu4.3) ... 2023-10-06T03:28:06.586 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../libuuid1_2.34-0.1ubuntu9.4_amd64.deb ... 2023-10-06T03:28:06.743 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking libuuid1:amd64 (2.34-0.1ubuntu9.4) over (2.34-0.1ubuntu9.3) ... 2023-10-06T03:28:06.974 INFO:teuthology.orchestra.run.smithi110.stdout:Created symlink /etc/systemd/system/dbus-fi.w1.wpa_supplicant1.service -> /lib/systemd/system/wpa_supplicant.service. 2023-10-06T03:28:06.974 INFO:teuthology.orchestra.run.smithi110.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/wpa_supplicant.service -> /lib/systemd/system/wpa_supplicant.service. 2023-10-06T03:28:07.077 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up libuuid1:amd64 (2.34-0.1ubuntu9.4) ... 2023-10-06T03:28:07.481 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up uuid-dev:amd64 (2.34-0.1ubuntu9.4) ... 2023-10-06T03:28:07.518 INFO:teuthology.orchestra.run.smithi192.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 152664 files and directories currently installed.) 2023-10-06T03:28:07.522 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../00-libdw1_0.176-1.1ubuntu0.1_amd64.deb ... 2023-10-06T03:28:07.608 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up libteamdctl0:amd64 (1.30-1) ... 2023-10-06T03:28:07.659 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking libdw1:amd64 (0.176-1.1ubuntu0.1) over (0.176-1.1build1) ... 2023-10-06T03:28:07.733 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up libssl-dev:amd64 (1.1.1f-1ubuntu2.19) ... 2023-10-06T03:28:07.851 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up usb-modeswitch-data (20191128-3) ... 2023-10-06T03:28:07.952 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up libnm0:amd64 (1.22.10-1ubuntu2.3) ... 2023-10-06T03:28:08.053 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up ppp (2.4.7-2+4.1ubuntu5.1) ... 2023-10-06T03:28:08.176 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../01-libelf1_0.176-1.1ubuntu0.1_amd64.deb ... 2023-10-06T03:28:08.494 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking libelf1:amd64 (0.176-1.1ubuntu0.1) over (0.176-1.1build1) ... 2023-10-06T03:28:09.043 INFO:teuthology.orchestra.run.smithi110.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/pppd-dns.service -> /lib/systemd/system/pppd-dns.service. 2023-10-06T03:28:09.301 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up dump (0.4b46-6) ... 2023-10-06T03:28:09.387 INFO:teuthology.orchestra.run.smithi110.stdout:update-alternatives: using /usr/sbin/rmt-dump to provide /usr/sbin/rmt (rmt) in auto mode 2023-10-06T03:28:09.433 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../02-libssl1.1_1.1.1f-1ubuntu2.19_amd64.deb ... 2023-10-06T03:28:09.444 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up libmm-glib0:amd64 (1.18.6-1~ubuntu20.04.1) ... 2023-10-06T03:28:09.545 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up libbluetooth3:amd64 (5.53-0ubuntu3.6) ... 2023-10-06T03:28:09.563 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking libssl1.1:amd64 (1.1.1f-1ubuntu2.19) over (1.1.1f-1ubuntu2.16) ... 2023-10-06T03:28:09.662 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up indent (2.2.12-1ubuntu0.20.04.1) ... 2023-10-06T03:28:09.814 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up usb-modeswitch (2.5.2+repack0-2ubuntu3) ... 2023-10-06T03:28:09.990 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up libndp0:amd64 (1.7-0ubuntu1) ... 2023-10-06T03:28:10.100 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package liblzo2-2:amd64. 2023-10-06T03:28:10.116 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up libelf1:amd64 (0.176-1.1ubuntu0.1) ... 2023-10-06T03:28:10.117 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../03-liblzo2-2_2.10-2_amd64.deb ... 2023-10-06T03:28:10.158 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking liblzo2-2:amd64 (2.10-2) ... 2023-10-06T03:28:10.275 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up libmbim-glib4:amd64 (1.26.2-1~ubuntu20.04.1) ... 2023-10-06T03:28:10.401 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up pptp-linux (1.10.0-1build1) ... 2023-10-06T03:28:10.478 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package dump. 2023-10-06T03:28:10.494 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../04-dump_0.4b46-6_amd64.deb ... 2023-10-06T03:28:10.528 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking dump (0.4b46-6) ... 2023-10-06T03:28:10.536 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up libdw1:amd64 (0.176-1.1ubuntu0.1) ... 2023-10-06T03:28:10.639 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up network-manager (1.22.10-1ubuntu2.3) ... 2023-10-06T03:28:10.865 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package indent. 2023-10-06T03:28:10.885 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../05-indent_2.2.12-1ubuntu0.20.04.1_amd64.deb ... 2023-10-06T03:28:10.923 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking indent (2.2.12-1ubuntu0.20.04.1) ... 2023-10-06T03:28:11.251 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package libaio-dev:amd64. 2023-10-06T03:28:11.268 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../06-libaio-dev_0.3.112-5_amd64.deb ... 2023-10-06T03:28:11.309 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking libaio-dev:amd64 (0.3.112-5) ... 2023-10-06T03:28:11.352 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-10-06T03:28:11.352 INFO:teuthology.orchestra.run.smithi110.stdout:The following network interfaces were found in /etc/network/interfaces 2023-10-06T03:28:11.352 INFO:teuthology.orchestra.run.smithi110.stdout:which means they are currently configured by ifupdown: 2023-10-06T03:28:11.352 INFO:teuthology.orchestra.run.smithi110.stdout:- enp3s0f1 2023-10-06T03:28:11.352 INFO:teuthology.orchestra.run.smithi110.stdout:If you want to manage those interfaces with NetworkManager instead 2023-10-06T03:28:11.352 INFO:teuthology.orchestra.run.smithi110.stdout:remove their configuration from /etc/network/interfaces. 2023-10-06T03:28:11.352 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-10-06T03:28:11.514 INFO:teuthology.orchestra.run.smithi110.stdout:Created symlink /etc/systemd/system/dbus-org.freedesktop.nm-dispatcher.service -> /lib/systemd/system/NetworkManager-dispatcher.service. 2023-10-06T03:28:11.671 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package libbluetooth3:amd64. 2023-10-06T03:28:11.689 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../07-libbluetooth3_5.53-0ubuntu3.6_amd64.deb ... 2023-10-06T03:28:11.755 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking libbluetooth3:amd64 (5.53-0ubuntu3.6) ... 2023-10-06T03:28:11.845 INFO:teuthology.orchestra.run.smithi110.stdout:Created symlink /etc/systemd/system/network-online.target.wants/NetworkManager-wait-online.service -> /lib/systemd/system/NetworkManager-wait-online.service. 2023-10-06T03:28:12.040 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package libelf-dev:amd64. 2023-10-06T03:28:12.058 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../08-libelf-dev_0.176-1.1ubuntu0.1_amd64.deb ... 2023-10-06T03:28:12.132 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking libelf-dev:amd64 (0.176-1.1ubuntu0.1) ... 2023-10-06T03:28:12.366 INFO:teuthology.orchestra.run.smithi110.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/NetworkManager.service -> /lib/systemd/system/NetworkManager.service. 2023-10-06T03:28:12.478 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package libmbim-glib4:amd64. 2023-10-06T03:28:12.497 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../09-libmbim-glib4_1.26.2-1~ubuntu20.04.1_amd64.deb ... 2023-10-06T03:28:12.536 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking libmbim-glib4:amd64 (1.26.2-1~ubuntu20.04.1) ... 2023-10-06T03:28:12.846 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package libmbim-proxy. 2023-10-06T03:28:12.857 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up libmbim-proxy (1.26.2-1~ubuntu20.04.1) ... 2023-10-06T03:28:12.871 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../10-libmbim-proxy_1.26.2-1~ubuntu20.04.1_amd64.deb ... 2023-10-06T03:28:12.905 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking libmbim-proxy (1.26.2-1~ubuntu20.04.1) ... 2023-10-06T03:28:13.034 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up network-manager-pptp (1.2.8-2) ... 2023-10-06T03:28:13.176 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up libelf-dev:amd64 (0.176-1.1ubuntu0.1) ... 2023-10-06T03:28:13.257 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package libmm-glib0:amd64. 2023-10-06T03:28:13.277 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../11-libmm-glib0_1.18.6-1~ubuntu20.04.1_amd64.deb ... 2023-10-06T03:28:13.277 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up libqmi-glib5:amd64 (1.30.4-1~ubuntu20.04.1) ... 2023-10-06T03:28:13.315 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking libmm-glib0:amd64 (1.18.6-1~ubuntu20.04.1) ... 2023-10-06T03:28:13.378 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up libqmi-proxy (1.30.4-1~ubuntu20.04.1) ... 2023-10-06T03:28:13.496 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up modemmanager (1.18.6-1~ubuntu20.04.1) ... 2023-10-06T03:28:13.710 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package libndp0:amd64. 2023-10-06T03:28:13.727 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../12-libndp0_1.7-0ubuntu1_amd64.deb ... 2023-10-06T03:28:13.768 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking libndp0:amd64 (1.7-0ubuntu1) ... 2023-10-06T03:28:13.856 INFO:teuthology.orchestra.run.smithi110.stdout:Created symlink /etc/systemd/system/dbus-org.freedesktop.ModemManager1.service -> /lib/systemd/system/ModemManager.service. 2023-10-06T03:28:13.857 INFO:teuthology.orchestra.run.smithi110.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ModemManager.service -> /lib/systemd/system/ModemManager.service. 2023-10-06T03:28:14.104 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package libnm0:amd64. 2023-10-06T03:28:14.125 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../13-libnm0_1.22.10-1ubuntu2.3_amd64.deb ... 2023-10-06T03:28:14.181 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking libnm0:amd64 (1.22.10-1ubuntu2.3) ... 2023-10-06T03:28:14.391 INFO:teuthology.orchestra.run.smithi110.stdout:Processing triggers for mime-support (3.64ubuntu1) ... 2023-10-06T03:28:14.492 INFO:teuthology.orchestra.run.smithi110.stdout:Processing triggers for hicolor-icon-theme (0.17-2) ... 2023-10-06T03:28:14.618 INFO:teuthology.orchestra.run.smithi110.stdout:Processing triggers for libc-bin (2.31-0ubuntu9.9) ... 2023-10-06T03:28:14.786 INFO:teuthology.orchestra.run.smithi110.stdout:Processing triggers for systemd (245.4-4ubuntu3.17) ... 2023-10-06T03:28:15.079 INFO:teuthology.orchestra.run.smithi110.stdout:Processing triggers for man-db (2.9.1-1) ... 2023-10-06T03:28:15.192 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package libqmi-glib5:amd64. 2023-10-06T03:28:15.210 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../14-libqmi-glib5_1.30.4-1~ubuntu20.04.1_amd64.deb ... 2023-10-06T03:28:15.251 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking libqmi-glib5:amd64 (1.30.4-1~ubuntu20.04.1) ... 2023-10-06T03:28:15.587 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package libqmi-proxy. 2023-10-06T03:28:15.607 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../15-libqmi-proxy_1.30.4-1~ubuntu20.04.1_amd64.deb ... 2023-10-06T03:28:15.645 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking libqmi-proxy (1.30.4-1~ubuntu20.04.1) ... 2023-10-06T03:28:15.931 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package libssl-dev:amd64. 2023-10-06T03:28:15.951 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../16-libssl-dev_1.1.1f-1ubuntu2.19_amd64.deb ... 2023-10-06T03:28:15.989 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking libssl-dev:amd64 (1.1.1f-1ubuntu2.19) ... 2023-10-06T03:28:16.543 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package libteamdctl0:amd64. 2023-10-06T03:28:16.560 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../17-libteamdctl0_1.30-1_amd64.deb ... 2023-10-06T03:28:16.601 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking libteamdctl0:amd64 (1.30-1) ... 2023-10-06T03:28:16.987 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package modemmanager. 2023-10-06T03:28:17.008 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../18-modemmanager_1.18.6-1~ubuntu20.04.1_amd64.deb ... 2023-10-06T03:28:17.065 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking modemmanager (1.18.6-1~ubuntu20.04.1) ... 2023-10-06T03:28:17.735 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package wpasupplicant. 2023-10-06T03:28:17.752 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../19-wpasupplicant_2%3a2.9-1ubuntu4.3_amd64.deb ... 2023-10-06T03:28:17.826 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking wpasupplicant (2:2.9-1ubuntu4.3) ... 2023-10-06T03:28:18.382 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package network-manager. 2023-10-06T03:28:18.401 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../20-network-manager_1.22.10-1ubuntu2.3_amd64.deb ... 2023-10-06T03:28:18.465 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking network-manager (1.22.10-1ubuntu2.3) ... 2023-10-06T03:28:19.256 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package ppp. 2023-10-06T03:28:19.274 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../21-ppp_2.4.7-2+4.1ubuntu5.1_amd64.deb ... 2023-10-06T03:28:19.424 INFO:teuthology.orchestra.run.smithi110.stdout:Processing triggers for dbus (1.12.16-2ubuntu2.2) ... 2023-10-06T03:28:19.516 INFO:teuthology.orchestra.run.smithi110.stdout:Processing triggers for install-info (6.7.0.dfsg.2-5) ... 2023-10-06T03:28:19.547 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking ppp (2.4.7-2+4.1ubuntu5.1) ... 2023-10-06T03:28:19.877 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package pptp-linux. 2023-10-06T03:28:19.894 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../22-pptp-linux_1.10.0-1build1_amd64.deb ... 2023-10-06T03:28:19.952 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking pptp-linux (1.10.0-1build1) ... 2023-10-06T03:28:20.319 INFO:teuthology.orchestra.run.smithi110.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2023-10-06T03:28:20.322 DEBUG:teuthology.parallel:result is None 2023-10-06T03:28:20.499 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package network-manager-pptp. 2023-10-06T03:28:20.518 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../23-network-manager-pptp_1.2.8-2_amd64.deb ... 2023-10-06T03:28:20.574 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking network-manager-pptp (1.2.8-2) ... 2023-10-06T03:28:20.852 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package usb-modeswitch-data. 2023-10-06T03:28:20.868 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../24-usb-modeswitch-data_20191128-3_all.deb ... 2023-10-06T03:28:20.910 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking usb-modeswitch-data (20191128-3) ... 2023-10-06T03:28:21.380 INFO:teuthology.orchestra.run.smithi192.stdout:Selecting previously unselected package usb-modeswitch. 2023-10-06T03:28:21.397 INFO:teuthology.orchestra.run.smithi192.stdout:Preparing to unpack .../25-usb-modeswitch_2.5.2+repack0-2ubuntu3_amd64.deb ... 2023-10-06T03:28:21.448 INFO:teuthology.orchestra.run.smithi192.stdout:Unpacking usb-modeswitch (2.5.2+repack0-2ubuntu3) ... 2023-10-06T03:28:21.774 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up libssl1.1:amd64 (1.1.1f-1ubuntu2.19) ... 2023-10-06T03:28:22.118 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up liblzo2-2:amd64 (2.10-2) ... 2023-10-06T03:28:22.244 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up libaio-dev:amd64 (0.3.112-5) ... 2023-10-06T03:28:22.395 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up wpasupplicant (2:2.9-1ubuntu4.3) ... 2023-10-06T03:28:23.265 INFO:teuthology.orchestra.run.smithi192.stdout:Created symlink /etc/systemd/system/dbus-fi.w1.wpa_supplicant1.service -> /lib/systemd/system/wpa_supplicant.service. 2023-10-06T03:28:23.265 INFO:teuthology.orchestra.run.smithi192.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/wpa_supplicant.service -> /lib/systemd/system/wpa_supplicant.service. 2023-10-06T03:28:23.266 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up uuid-dev:amd64 (2.34-0.1ubuntu9.4) ... 2023-10-06T03:28:23.375 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up libteamdctl0:amd64 (1.30-1) ... 2023-10-06T03:28:23.534 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up libssl-dev:amd64 (1.1.1f-1ubuntu2.19) ... 2023-10-06T03:28:23.746 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up usb-modeswitch-data (20191128-3) ... 2023-10-06T03:28:23.819 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up libnm0:amd64 (1.22.10-1ubuntu2.3) ... 2023-10-06T03:28:23.945 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up ppp (2.4.7-2+4.1ubuntu5.1) ... 2023-10-06T03:28:24.953 INFO:teuthology.orchestra.run.smithi192.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/pppd-dns.service -> /lib/systemd/system/pppd-dns.service. 2023-10-06T03:28:25.218 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up dump (0.4b46-6) ... 2023-10-06T03:28:25.314 INFO:teuthology.orchestra.run.smithi192.stdout:update-alternatives: using /usr/sbin/rmt-dump to provide /usr/sbin/rmt (rmt) in auto mode 2023-10-06T03:28:25.378 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up libmm-glib0:amd64 (1.18.6-1~ubuntu20.04.1) ... 2023-10-06T03:28:25.732 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up libbluetooth3:amd64 (5.53-0ubuntu3.6) ... 2023-10-06T03:28:25.864 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up indent (2.2.12-1ubuntu0.20.04.1) ... 2023-10-06T03:28:26.015 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up usb-modeswitch (2.5.2+repack0-2ubuntu3) ... 2023-10-06T03:28:26.175 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up libndp0:amd64 (1.7-0ubuntu1) ... 2023-10-06T03:28:26.326 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up libelf1:amd64 (0.176-1.1ubuntu0.1) ... 2023-10-06T03:28:26.452 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up libmbim-glib4:amd64 (1.26.2-1~ubuntu20.04.1) ... 2023-10-06T03:28:26.596 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up pptp-linux (1.10.0-1build1) ... 2023-10-06T03:28:26.771 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up libdw1:amd64 (0.176-1.1ubuntu0.1) ... 2023-10-06T03:28:26.897 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up network-manager (1.22.10-1ubuntu2.3) ... 2023-10-06T03:28:27.642 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-10-06T03:28:27.642 INFO:teuthology.orchestra.run.smithi192.stdout:The following network interfaces were found in /etc/network/interfaces 2023-10-06T03:28:27.642 INFO:teuthology.orchestra.run.smithi192.stdout:which means they are currently configured by ifupdown: 2023-10-06T03:28:27.643 INFO:teuthology.orchestra.run.smithi192.stdout:- enp3s0f1 2023-10-06T03:28:27.643 INFO:teuthology.orchestra.run.smithi192.stdout:If you want to manage those interfaces with NetworkManager instead 2023-10-06T03:28:27.643 INFO:teuthology.orchestra.run.smithi192.stdout:remove their configuration from /etc/network/interfaces. 2023-10-06T03:28:27.643 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-10-06T03:28:27.800 INFO:teuthology.orchestra.run.smithi192.stdout:Created symlink /etc/systemd/system/dbus-org.freedesktop.nm-dispatcher.service -> /lib/systemd/system/NetworkManager-dispatcher.service. 2023-10-06T03:28:28.125 INFO:teuthology.orchestra.run.smithi192.stdout:Created symlink /etc/systemd/system/network-online.target.wants/NetworkManager-wait-online.service -> /lib/systemd/system/NetworkManager-wait-online.service. 2023-10-06T03:28:28.663 INFO:teuthology.orchestra.run.smithi192.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/NetworkManager.service -> /lib/systemd/system/NetworkManager.service. 2023-10-06T03:28:29.150 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up libmbim-proxy (1.26.2-1~ubuntu20.04.1) ... 2023-10-06T03:28:29.285 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up network-manager-pptp (1.2.8-2) ... 2023-10-06T03:28:29.453 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up libelf-dev:amd64 (0.176-1.1ubuntu0.1) ... 2023-10-06T03:28:29.579 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up libqmi-glib5:amd64 (1.30.4-1~ubuntu20.04.1) ... 2023-10-06T03:28:29.705 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up libqmi-proxy (1.30.4-1~ubuntu20.04.1) ... 2023-10-06T03:28:29.865 INFO:teuthology.orchestra.run.smithi192.stdout:Setting up modemmanager (1.18.6-1~ubuntu20.04.1) ... 2023-10-06T03:28:30.133 INFO:teuthology.orchestra.run.smithi192.stdout:Created symlink /etc/systemd/system/dbus-org.freedesktop.ModemManager1.service -> /lib/systemd/system/ModemManager.service. 2023-10-06T03:28:30.133 INFO:teuthology.orchestra.run.smithi192.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ModemManager.service -> /lib/systemd/system/ModemManager.service. 2023-10-06T03:28:30.659 INFO:teuthology.orchestra.run.smithi192.stdout:Processing triggers for mime-support (3.64ubuntu1) ... 2023-10-06T03:28:30.768 INFO:teuthology.orchestra.run.smithi192.stdout:Processing triggers for hicolor-icon-theme (0.17-2) ... 2023-10-06T03:28:31.311 INFO:teuthology.orchestra.run.smithi192.stdout:Processing triggers for libc-bin (2.31-0ubuntu9.9) ... 2023-10-06T03:28:31.454 INFO:teuthology.orchestra.run.smithi192.stdout:Processing triggers for systemd (245.4-4ubuntu3.17) ... 2023-10-06T03:28:31.747 INFO:teuthology.orchestra.run.smithi192.stdout:Processing triggers for man-db (2.9.1-1) ... 2023-10-06T03:28:35.626 INFO:teuthology.orchestra.run.smithi192.stdout:Processing triggers for dbus (1.12.16-2ubuntu2.2) ... 2023-10-06T03:28:35.733 INFO:teuthology.orchestra.run.smithi192.stdout:Processing triggers for install-info (6.7.0.dfsg.2-5) ... 2023-10-06T03:28:36.610 INFO:teuthology.orchestra.run.smithi192.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2023-10-06T03:28:36.613 DEBUG:teuthology.parallel:result is None 2023-10-06T03:28:36.614 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&sha1=736ac5ba72cb47af7d6f6666b1e100f7e20a20e2 2023-10-06T03:28:36.756 DEBUG:teuthology.orchestra.run.smithi110:> dpkg-query -W -f '${Version}' ceph 2023-10-06T03:28:36.780 INFO:teuthology.orchestra.run.smithi110.stdout:17.2.6-1296-g736ac5ba-1focal 2023-10-06T03:28:36.781 INFO:teuthology.packaging:The installed version of ceph is 17.2.6-1296-g736ac5ba-1focal 2023-10-06T03:28:36.781 INFO:teuthology.task.install:The correct ceph version 17.2.6-1296-g736ac5ba-1focal is installed. 2023-10-06T03:28:36.781 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&sha1=736ac5ba72cb47af7d6f6666b1e100f7e20a20e2 2023-10-06T03:28:36.991 DEBUG:teuthology.orchestra.run.smithi192:> dpkg-query -W -f '${Version}' ceph 2023-10-06T03:28:37.015 INFO:teuthology.orchestra.run.smithi192.stdout:17.2.6-1296-g736ac5ba-1focal 2023-10-06T03:28:37.015 INFO:teuthology.packaging:The installed version of ceph is 17.2.6-1296-g736ac5ba-1focal 2023-10-06T03:28:37.015 INFO:teuthology.task.install:The correct ceph version 17.2.6-1296-g736ac5ba-1focal is installed. 2023-10-06T03:28:37.016 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2023-10-06T03:28:37.016 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2023-10-06T03:28:37.016 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-10-06T03:28:37.033 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2023-10-06T03:28:37.034 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-10-06T03:28:37.072 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2023-10-06T03:28:37.072 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2023-10-06T03:28:37.072 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd of=/usr/bin/daemon-helper 2023-10-06T03:28:37.087 DEBUG:teuthology.orchestra.run.smithi110:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-10-06T03:28:37.145 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2023-10-06T03:28:37.145 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd of=/usr/bin/daemon-helper 2023-10-06T03:28:37.161 DEBUG:teuthology.orchestra.run.smithi192:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-10-06T03:28:37.170 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2023-10-06T03:28:37.171 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2023-10-06T03:28:37.171 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd of=/usr/bin/adjust-ulimits 2023-10-06T03:28:37.185 DEBUG:teuthology.orchestra.run.smithi110:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-10-06T03:28:37.198 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2023-10-06T03:28:37.198 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd of=/usr/bin/adjust-ulimits 2023-10-06T03:28:37.229 DEBUG:teuthology.orchestra.run.smithi192:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-10-06T03:28:37.238 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2023-10-06T03:28:37.238 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2023-10-06T03:28:37.238 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd of=/usr/bin/stdin-killer 2023-10-06T03:28:37.254 DEBUG:teuthology.orchestra.run.smithi110:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-10-06T03:28:37.313 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2023-10-06T03:28:37.314 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd of=/usr/bin/stdin-killer 2023-10-06T03:28:37.329 DEBUG:teuthology.orchestra.run.smithi192:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-10-06T03:28:37.388 INFO:teuthology.run_tasks:Running task ceph... 2023-10-06T03:28:37.457 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2023-10-06T03:28:37.457 DEBUG:teuthology.orchestra.run.smithi110:> sudo chmod 777 /var/log/ceph 2023-10-06T03:28:37.459 DEBUG:teuthology.orchestra.run.smithi192:> sudo chmod 777 /var/log/ceph 2023-10-06T03:28:37.474 INFO:tasks.ceph:Disabling ceph logrotate... 2023-10-06T03:28:37.475 DEBUG:teuthology.orchestra.run.smithi110:> sudo rm -f -- /etc/logrotate.d/ceph 2023-10-06T03:28:37.516 DEBUG:teuthology.orchestra.run.smithi192:> sudo rm -f -- /etc/logrotate.d/ceph 2023-10-06T03:28:37.532 INFO:tasks.ceph:Creating extra log directories... 2023-10-06T03:28:37.532 DEBUG:teuthology.orchestra.run.smithi110:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-10-06T03:28:37.535 DEBUG:teuthology.orchestra.run.smithi192:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-10-06T03:28:37.551 INFO:tasks.ceph:Setting up log rotation with {'ceph-mds': '10G', 'ceph-osd': '10G'} 2023-10-06T03:28:37.551 INFO:tasks.ceph:writing logrotate stanza for ceph-mds 2023-10-06T03:28:37.551 INFO:tasks.ceph:writing logrotate stanza for ceph-osd 2023-10-06T03:28:37.551 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2023-10-06T03:28:37.551 DEBUG:teuthology.orchestra.run.smithi110:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2023-10-06T03:28:37.556 DEBUG:teuthology.orchestra.run.smithi110:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2023-10-06T03:28:37.636 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2023-10-06T03:28:37.636 DEBUG:teuthology.orchestra.run.smithi192:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2023-10-06T03:28:37.643 DEBUG:teuthology.orchestra.run.smithi192:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2023-10-06T03:28:37.685 INFO:tasks.ceph:Creating ceph cluster ceph... 2023-10-06T03:28:37.685 INFO:tasks.ceph:config {'conf': {'client': {'client mount timeout': 600, 'client snapdir': '.customsnapfuse', 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mds': {'debug mds': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2023-10-06T03:28:37.685 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/yuriw-2023-10-05_23:52:50-fs-wip-yuri-testing-2023-10-05-0730-quincy-distro-default-smithi/7412204', 'branch': 'wip-yuri-testing-2023-10-05-0730-quincy', 'description': 'fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/snap_schedule_snapdir}', 'email': 'yweinste@redhat.com', 'first_in_suite': False, 'job_id': '7412204', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'ktype': 'distro', 'last_in_suite': False, 'log-rotate': {'ceph-mds': '10G', 'ceph-osd': '10G'}, 'machine_type': 'smithi', 'name': 'yuriw-2023-10-05_23:52:50-fs-wip-yuri-testing-2023-10-05-0730-quincy-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 20}}, {'machine': {'disk': 200}}], 'os_type': 'ubuntu', 'os_version': '20.04', 'overrides': {'admin_socket': {'branch': 'wip-yuri-testing-2023-10-05-0730-quincy'}, 'ceph': {'cephfs': {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd']}, 'conf': {'client': {'client mount timeout': 600, 'client snapdir': '.customsnapfuse', 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mds': {'debug mds': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding'], 'sha1': '736ac5ba72cb47af7d6f6666b1e100f7e20a20e2'}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'flavor': 'default', 'sha1': '736ac5ba72cb47af7d6f6666b1e100f7e20a20e2'}}, 'kclient': {'snapdirname': '.customsnapkernel'}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-yuri-testing-2023-10-05-0730-quincy', 'sha1': 'f1e315a9732d36c05ec316678e2397dc58561c7f'}}, 'owner': 'scheduled_yuriw@teuthology', 'priority': 75, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'client.2'], ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'client.3']], 'seed': 7519, 'sha1': '736ac5ba72cb47af7d6f6666b1e100f7e20a20e2', 'sleep_before_teardown': 0, 'subset': '111/12000', 'suite': 'fs', 'suite_branch': 'wip-yuri-testing-2023-10-05-0730-quincy', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph-c_f1e315a9732d36c05ec316678e2397dc58561c7f/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph-ci.git', 'suite_sha1': 'f1e315a9732d36c05ec316678e2397dc58561c7f', 'targets': {'smithi110.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFyT/r7xuKfmg50w0OdQebr4A/ptJo1Zk6DyZNZNZM/q5rW+sznh+wvHqeT3e4YtCdZLaXyvMms8o8u8H5nXOYM=', 'smithi192.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFyT/r7xuKfmg50w0OdQebr4A/ptJo1Zk6DyZNZNZM/q5rW+sznh+wvHqeT3e4YtCdZLaXyvMms8o8u8H5nXOYM='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel']}, 'flavor': 'default', 'sha1': '736ac5ba72cb47af7d6f6666b1e100f7e20a20e2'}}, {'ceph': None}, {'ceph-fuse': None}, {'cephfs_test_runner': {'modules': ['tasks.cephfs.test_snap_schedules.TestSnapSchedulesSnapdir']}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': ['local function is_kupstream() return false end', 'local function is_kdistro() return false end', 'local function is_fuse() return true end', "local function syntax_version() return '' end"]}, 'teuthology_branch': 'main', 'teuthology_sha1': '54e62bcbac4e53d9685e08328b790d3b20d71cae', 'timestamp': '2023-10-05_23:52:50', 'tube': 'smithi', 'user': 'yuriw', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2830'} 2023-10-06T03:28:37.685 DEBUG:teuthology.orchestra.run.smithi110:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2023-10-06T03:28:37.688 DEBUG:teuthology.orchestra.run.smithi192:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2023-10-06T03:28:37.728 DEBUG:teuthology.orchestra.run.smithi110:> sudo install -d -m0777 -- /var/run/ceph 2023-10-06T03:28:37.731 DEBUG:teuthology.orchestra.run.smithi192:> sudo install -d -m0777 -- /var/run/ceph 2023-10-06T03:28:37.745 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2023-10-06T03:28:37.745 DEBUG:teuthology.orchestra.run.smithi110:> dd if=/scratch_devs of=/dev/stdout 2023-10-06T03:28:37.793 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-10-06T03:28:37.793 DEBUG:teuthology.orchestra.run.smithi110:> stat /dev/vg_nvme/lv_1 2023-10-06T03:28:37.800 INFO:teuthology.orchestra.run.smithi110.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-10-06T03:28:37.800 INFO:teuthology.orchestra.run.smithi110.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-06T03:28:37.800 INFO:teuthology.orchestra.run.smithi110.stdout:Device: 6h/6d Inode: 361 Links: 1 2023-10-06T03:28:37.800 INFO:teuthology.orchestra.run.smithi110.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-06T03:28:37.800 INFO:teuthology.orchestra.run.smithi110.stdout:Access: 2023-10-06 03:24:10.060773215 +0000 2023-10-06T03:28:37.800 INFO:teuthology.orchestra.run.smithi110.stdout:Modify: 2023-10-06 03:24:09.788779473 +0000 2023-10-06T03:28:37.800 INFO:teuthology.orchestra.run.smithi110.stdout:Change: 2023-10-06 03:24:09.788779473 +0000 2023-10-06T03:28:37.801 INFO:teuthology.orchestra.run.smithi110.stdout: Birth: - 2023-10-06T03:28:37.801 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-10-06T03:28:37.812 INFO:teuthology.orchestra.run.smithi110.stderr:1+0 records in 2023-10-06T03:28:37.813 INFO:teuthology.orchestra.run.smithi110.stderr:1+0 records out 2023-10-06T03:28:37.813 INFO:teuthology.orchestra.run.smithi110.stderr:512 bytes copied, 0.000311021 s, 1.6 MB/s 2023-10-06T03:28:37.813 DEBUG:teuthology.orchestra.run.smithi110:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-10-06T03:28:37.821 DEBUG:teuthology.orchestra.run.smithi110:> stat /dev/vg_nvme/lv_2 2023-10-06T03:28:37.827 INFO:teuthology.orchestra.run.smithi110.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-10-06T03:28:37.827 INFO:teuthology.orchestra.run.smithi110.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-06T03:28:37.827 INFO:teuthology.orchestra.run.smithi110.stdout:Device: 6h/6d Inode: 365 Links: 1 2023-10-06T03:28:37.827 INFO:teuthology.orchestra.run.smithi110.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-06T03:28:37.828 INFO:teuthology.orchestra.run.smithi110.stdout:Access: 2023-10-06 03:24:10.596760883 +0000 2023-10-06T03:28:37.828 INFO:teuthology.orchestra.run.smithi110.stdout:Modify: 2023-10-06 03:24:10.284768061 +0000 2023-10-06T03:28:37.828 INFO:teuthology.orchestra.run.smithi110.stdout:Change: 2023-10-06 03:24:10.284768061 +0000 2023-10-06T03:28:37.828 INFO:teuthology.orchestra.run.smithi110.stdout: Birth: - 2023-10-06T03:28:37.828 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-10-06T03:28:37.841 INFO:teuthology.orchestra.run.smithi110.stderr:1+0 records in 2023-10-06T03:28:37.841 INFO:teuthology.orchestra.run.smithi110.stderr:1+0 records out 2023-10-06T03:28:37.841 INFO:teuthology.orchestra.run.smithi110.stderr:512 bytes copied, 0.000252098 s, 2.0 MB/s 2023-10-06T03:28:37.842 DEBUG:teuthology.orchestra.run.smithi110:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-10-06T03:28:37.895 DEBUG:teuthology.orchestra.run.smithi110:> stat /dev/vg_nvme/lv_3 2023-10-06T03:28:37.904 INFO:teuthology.orchestra.run.smithi110.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-10-06T03:28:37.904 INFO:teuthology.orchestra.run.smithi110.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-06T03:28:37.904 INFO:teuthology.orchestra.run.smithi110.stdout:Device: 6h/6d Inode: 374 Links: 1 2023-10-06T03:28:37.904 INFO:teuthology.orchestra.run.smithi110.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-06T03:28:37.904 INFO:teuthology.orchestra.run.smithi110.stdout:Access: 2023-10-06 03:24:11.088749565 +0000 2023-10-06T03:28:37.905 INFO:teuthology.orchestra.run.smithi110.stdout:Modify: 2023-10-06 03:24:10.820755730 +0000 2023-10-06T03:28:37.905 INFO:teuthology.orchestra.run.smithi110.stdout:Change: 2023-10-06 03:24:10.820755730 +0000 2023-10-06T03:28:37.905 INFO:teuthology.orchestra.run.smithi110.stdout: Birth: - 2023-10-06T03:28:37.905 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-10-06T03:28:37.917 INFO:teuthology.orchestra.run.smithi110.stderr:1+0 records in 2023-10-06T03:28:37.918 INFO:teuthology.orchestra.run.smithi110.stderr:1+0 records out 2023-10-06T03:28:37.918 INFO:teuthology.orchestra.run.smithi110.stderr:512 bytes copied, 0.000302864 s, 1.7 MB/s 2023-10-06T03:28:37.919 DEBUG:teuthology.orchestra.run.smithi110:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-10-06T03:28:37.927 DEBUG:teuthology.orchestra.run.smithi110:> stat /dev/vg_nvme/lv_4 2023-10-06T03:28:37.932 INFO:teuthology.orchestra.run.smithi110.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-10-06T03:28:37.933 INFO:teuthology.orchestra.run.smithi110.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-06T03:28:37.933 INFO:teuthology.orchestra.run.smithi110.stdout:Device: 6h/6d Inode: 378 Links: 1 2023-10-06T03:28:37.933 INFO:teuthology.orchestra.run.smithi110.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-06T03:28:37.933 INFO:teuthology.orchestra.run.smithi110.stdout:Access: 2023-10-06 03:24:11.560738705 +0000 2023-10-06T03:28:37.933 INFO:teuthology.orchestra.run.smithi110.stdout:Modify: 2023-10-06 03:24:11.288744963 +0000 2023-10-06T03:28:37.933 INFO:teuthology.orchestra.run.smithi110.stdout:Change: 2023-10-06 03:24:11.288744963 +0000 2023-10-06T03:28:37.933 INFO:teuthology.orchestra.run.smithi110.stdout: Birth: - 2023-10-06T03:28:37.933 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-10-06T03:28:37.946 INFO:teuthology.orchestra.run.smithi110.stderr:1+0 records in 2023-10-06T03:28:37.946 INFO:teuthology.orchestra.run.smithi110.stderr:1+0 records out 2023-10-06T03:28:37.946 INFO:teuthology.orchestra.run.smithi110.stderr:512 bytes copied, 0.000288844 s, 1.8 MB/s 2023-10-06T03:28:37.947 DEBUG:teuthology.orchestra.run.smithi110:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-10-06T03:28:37.999 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-10-06T03:28:38.000 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2023-10-06T03:28:38.000 DEBUG:teuthology.orchestra.run.smithi192:> dd if=/scratch_devs of=/dev/stdout 2023-10-06T03:28:38.007 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-10-06T03:28:38.008 DEBUG:teuthology.orchestra.run.smithi192:> stat /dev/vg_nvme/lv_1 2023-10-06T03:28:38.056 INFO:teuthology.orchestra.run.smithi192.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-10-06T03:28:38.057 INFO:teuthology.orchestra.run.smithi192.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-06T03:28:38.057 INFO:teuthology.orchestra.run.smithi192.stdout:Device: 6h/6d Inode: 358 Links: 1 2023-10-06T03:28:38.057 INFO:teuthology.orchestra.run.smithi192.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-06T03:28:38.057 INFO:teuthology.orchestra.run.smithi192.stdout:Access: 2023-10-06 03:24:10.376507102 +0000 2023-10-06T03:28:38.057 INFO:teuthology.orchestra.run.smithi192.stdout:Modify: 2023-10-06 03:24:10.084513832 +0000 2023-10-06T03:28:38.057 INFO:teuthology.orchestra.run.smithi192.stdout:Change: 2023-10-06 03:24:10.084513832 +0000 2023-10-06T03:28:38.057 INFO:teuthology.orchestra.run.smithi192.stdout: Birth: - 2023-10-06T03:28:38.057 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-10-06T03:28:38.069 INFO:teuthology.orchestra.run.smithi192.stderr:1+0 records in 2023-10-06T03:28:38.069 INFO:teuthology.orchestra.run.smithi192.stderr:1+0 records out 2023-10-06T03:28:38.070 INFO:teuthology.orchestra.run.smithi192.stderr:512 bytes copied, 0.000387778 s, 1.3 MB/s 2023-10-06T03:28:38.072 DEBUG:teuthology.orchestra.run.smithi192:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-10-06T03:28:38.080 DEBUG:teuthology.orchestra.run.smithi192:> stat /dev/vg_nvme/lv_2 2023-10-06T03:28:38.087 INFO:teuthology.orchestra.run.smithi192.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-10-06T03:28:38.088 INFO:teuthology.orchestra.run.smithi192.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-06T03:28:38.088 INFO:teuthology.orchestra.run.smithi192.stdout:Device: 6h/6d Inode: 365 Links: 1 2023-10-06T03:28:38.088 INFO:teuthology.orchestra.run.smithi192.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-06T03:28:38.088 INFO:teuthology.orchestra.run.smithi192.stdout:Access: 2023-10-06 03:24:10.884495392 +0000 2023-10-06T03:28:38.089 INFO:teuthology.orchestra.run.smithi192.stdout:Modify: 2023-10-06 03:24:10.604501847 +0000 2023-10-06T03:28:38.089 INFO:teuthology.orchestra.run.smithi192.stdout:Change: 2023-10-06 03:24:10.604501847 +0000 2023-10-06T03:28:38.089 INFO:teuthology.orchestra.run.smithi192.stdout: Birth: - 2023-10-06T03:28:38.089 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-10-06T03:28:38.144 INFO:teuthology.orchestra.run.smithi192.stderr:1+0 records in 2023-10-06T03:28:38.144 INFO:teuthology.orchestra.run.smithi192.stderr:1+0 records out 2023-10-06T03:28:38.144 INFO:teuthology.orchestra.run.smithi192.stderr:512 bytes copied, 0.000392052 s, 1.3 MB/s 2023-10-06T03:28:38.145 DEBUG:teuthology.orchestra.run.smithi192:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-10-06T03:28:38.152 DEBUG:teuthology.orchestra.run.smithi192:> stat /dev/vg_nvme/lv_3 2023-10-06T03:28:38.158 INFO:teuthology.orchestra.run.smithi192.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-10-06T03:28:38.158 INFO:teuthology.orchestra.run.smithi192.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-06T03:28:38.158 INFO:teuthology.orchestra.run.smithi192.stdout:Device: 6h/6d Inode: 371 Links: 1 2023-10-06T03:28:38.158 INFO:teuthology.orchestra.run.smithi192.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-06T03:28:38.158 INFO:teuthology.orchestra.run.smithi192.stdout:Access: 2023-10-06 03:24:11.412483221 +0000 2023-10-06T03:28:38.158 INFO:teuthology.orchestra.run.smithi192.stdout:Modify: 2023-10-06 03:24:11.136489583 +0000 2023-10-06T03:28:38.158 INFO:teuthology.orchestra.run.smithi192.stdout:Change: 2023-10-06 03:24:11.136489583 +0000 2023-10-06T03:28:38.159 INFO:teuthology.orchestra.run.smithi192.stdout: Birth: - 2023-10-06T03:28:38.159 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-10-06T03:28:38.215 INFO:teuthology.orchestra.run.smithi192.stderr:1+0 records in 2023-10-06T03:28:38.215 INFO:teuthology.orchestra.run.smithi192.stderr:1+0 records out 2023-10-06T03:28:38.215 INFO:teuthology.orchestra.run.smithi192.stderr:512 bytes copied, 0.000406938 s, 1.3 MB/s 2023-10-06T03:28:38.216 DEBUG:teuthology.orchestra.run.smithi192:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-10-06T03:28:38.226 DEBUG:teuthology.orchestra.run.smithi192:> stat /dev/vg_nvme/lv_4 2023-10-06T03:28:38.231 INFO:teuthology.orchestra.run.smithi192.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-10-06T03:28:38.231 INFO:teuthology.orchestra.run.smithi192.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-06T03:28:38.232 INFO:teuthology.orchestra.run.smithi192.stdout:Device: 6h/6d Inode: 380 Links: 1 2023-10-06T03:28:38.232 INFO:teuthology.orchestra.run.smithi192.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-06T03:28:38.232 INFO:teuthology.orchestra.run.smithi192.stdout:Access: 2023-10-06 03:24:11.920471513 +0000 2023-10-06T03:28:38.232 INFO:teuthology.orchestra.run.smithi192.stdout:Modify: 2023-10-06 03:24:11.640477966 +0000 2023-10-06T03:28:38.232 INFO:teuthology.orchestra.run.smithi192.stdout:Change: 2023-10-06 03:24:11.640477966 +0000 2023-10-06T03:28:38.232 INFO:teuthology.orchestra.run.smithi192.stdout: Birth: - 2023-10-06T03:28:38.232 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-10-06T03:28:38.288 INFO:teuthology.orchestra.run.smithi192.stderr:1+0 records in 2023-10-06T03:28:38.910 INFO:teuthology.orchestra.run.smithi192.stderr:1+0 records out 2023-10-06T03:28:38.914 INFO:teuthology.orchestra.run.smithi192.stderr:512 bytes copied, 0.000406141 s, 1.3 MB/s 2023-10-06T03:28:38.915 DEBUG:teuthology.orchestra.run.smithi192:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-10-06T03:28:38.927 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-10-06T03:28:38.927 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi110.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi192.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2023-10-06T03:28:38.927 INFO:tasks.ceph:Generating config... 2023-10-06T03:28:38.929 INFO:tasks.ceph:[client] client mount timeout = 600 2023-10-06T03:28:38.929 INFO:tasks.ceph:[client] client snapdir = .customsnapfuse 2023-10-06T03:28:38.929 INFO:tasks.ceph:[client] debug client = 20 2023-10-06T03:28:38.929 INFO:tasks.ceph:[client] debug ms = 1 2023-10-06T03:28:38.929 INFO:tasks.ceph:[client] pid file = 2023-10-06T03:28:38.929 INFO:tasks.ceph:[client] rados mon op timeout = 900 2023-10-06T03:28:38.930 INFO:tasks.ceph:[client] rados osd op timeout = 900 2023-10-06T03:28:38.930 INFO:tasks.ceph:[mds] debug mds = 20 2023-10-06T03:28:38.930 INFO:tasks.ceph:[mds] debug ms = 1 2023-10-06T03:28:38.930 INFO:tasks.ceph:[mds] mds debug frag = True 2023-10-06T03:28:38.930 INFO:tasks.ceph:[mds] mds debug scatterstat = True 2023-10-06T03:28:38.930 INFO:tasks.ceph:[mds] mds op complaint time = 180 2023-10-06T03:28:38.930 INFO:tasks.ceph:[mds] mds verify scatter = True 2023-10-06T03:28:38.931 INFO:tasks.ceph:[mds] osd op complaint time = 180 2023-10-06T03:28:38.931 INFO:tasks.ceph:[mds] rados mon op timeout = 900 2023-10-06T03:28:38.931 INFO:tasks.ceph:[mds] rados osd op timeout = 900 2023-10-06T03:28:38.931 INFO:tasks.ceph:[mgr] debug mgr = 20 2023-10-06T03:28:38.931 INFO:tasks.ceph:[mgr] debug ms = 1 2023-10-06T03:28:38.931 INFO:tasks.ceph:[mon] debug mon = 20 2023-10-06T03:28:38.931 INFO:tasks.ceph:[mon] debug ms = 1 2023-10-06T03:28:38.931 INFO:tasks.ceph:[mon] debug paxos = 20 2023-10-06T03:28:38.932 INFO:tasks.ceph:[mon] mon op complaint time = 120 2023-10-06T03:28:38.932 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2023-10-06T03:28:38.932 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2023-10-06T03:28:38.932 INFO:tasks.ceph:[osd] debug bluefs = 20 2023-10-06T03:28:38.932 INFO:tasks.ceph:[osd] debug bluestore = 20 2023-10-06T03:28:38.932 INFO:tasks.ceph:[osd] debug ms = 1 2023-10-06T03:28:38.933 INFO:tasks.ceph:[osd] debug osd = 20 2023-10-06T03:28:38.933 INFO:tasks.ceph:[osd] debug rocksdb = 10 2023-10-06T03:28:38.933 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2023-10-06T03:28:38.933 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2023-10-06T03:28:38.933 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2023-10-06T03:28:38.933 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2023-10-06T03:28:38.933 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2023-10-06T03:28:38.933 INFO:tasks.ceph:[osd] osd op complaint time = 180 2023-10-06T03:28:38.934 INFO:tasks.ceph:Setting up mon.a... 2023-10-06T03:28:38.934 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2023-10-06T03:28:38.969 INFO:teuthology.orchestra.run.smithi110.stdout:creating /etc/ceph/ceph.keyring 2023-10-06T03:28:38.972 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2023-10-06T03:28:39.049 DEBUG:teuthology.orchestra.run.smithi110:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-10-06T03:28:39.064 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.110'), ('mon.b', '172.21.15.192'), ('mon.c', '[v2:172.21.15.192:3301,v1:172.21.15.192:6790]')] 2023-10-06T03:28:39.065 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.110,172.21.15.192,[v2:172.21.15.192:3301,v1:172.21.15.192:6790]'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon op complaint time': 120}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'client mount timeout': 600, 'client snapdir': '.customsnapfuse', 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mon.a': {}, 'mon.b': {}, 'mon.c': {}, 'mds.a': {}, 'mds.c': {}, 'mds.b': {}, 'mds.d': {}, 'mds': {'debug mds': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}} 2023-10-06T03:28:39.066 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2023-10-06T03:28:39.066 DEBUG:teuthology.orchestra.run.smithi110:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2023-10-06T03:28:39.072 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.110 --add b 172.21.15.192 --addv c '[v2:172.21.15.192:3301,v1:172.21.15.192:6790]' --print /home/ubuntu/cephtest/ceph.monmap 2023-10-06T03:28:39.155 INFO:teuthology.orchestra.run.smithi110.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2023-10-06T03:28:39.155 INFO:teuthology.orchestra.run.smithi110.stdout:monmaptool: generated fsid 93b8f6ac-4c92-4b45-b8e4-b59617024aa2 2023-10-06T03:28:39.155 INFO:teuthology.orchestra.run.smithi110.stdout:setting min_mon_release = octopus 2023-10-06T03:28:39.155 INFO:teuthology.orchestra.run.smithi110.stdout:epoch 0 2023-10-06T03:28:39.155 INFO:teuthology.orchestra.run.smithi110.stdout:fsid 93b8f6ac-4c92-4b45-b8e4-b59617024aa2 2023-10-06T03:28:39.155 INFO:teuthology.orchestra.run.smithi110.stdout:last_changed 2023-10-06T03:28:39.154163+0000 2023-10-06T03:28:39.155 INFO:teuthology.orchestra.run.smithi110.stdout:created 2023-10-06T03:28:39.154163+0000 2023-10-06T03:28:39.156 INFO:teuthology.orchestra.run.smithi110.stdout:min_mon_release 15 (octopus) 2023-10-06T03:28:39.156 INFO:teuthology.orchestra.run.smithi110.stdout:election_strategy: 1 2023-10-06T03:28:39.156 INFO:teuthology.orchestra.run.smithi110.stdout:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2023-10-06T03:28:39.156 INFO:teuthology.orchestra.run.smithi110.stdout:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2023-10-06T03:28:39.156 INFO:teuthology.orchestra.run.smithi110.stdout:2: [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] mon.c 2023-10-06T03:28:39.156 INFO:teuthology.orchestra.run.smithi110.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2023-10-06T03:28:39.157 DEBUG:teuthology.orchestra.run.smithi110:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2023-10-06T03:28:39.162 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 93b8f6ac-4c92-4b45-b8e4-b59617024aa2... 2023-10-06T03:28:39.163 DEBUG:teuthology.orchestra.run.smithi110:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2023-10-06T03:28:39.165 DEBUG:teuthology.orchestra.run.smithi192:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2023-10-06T03:28:39.196 INFO:teuthology.orchestra.run.smithi192.stdout:[global] 2023-10-06T03:28:39.196 INFO:teuthology.orchestra.run.smithi192.stdout: chdir = "" 2023-10-06T03:28:39.196 INFO:teuthology.orchestra.run.smithi192.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-10-06T03:28:39.197 INFO:teuthology.orchestra.run.smithi192.stdout: auth supported = cephx 2023-10-06T03:28:39.197 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-10-06T03:28:39.197 INFO:teuthology.orchestra.run.smithi192.stdout: filestore xattr use omap = true 2023-10-06T03:28:39.197 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-10-06T03:28:39.197 INFO:teuthology.orchestra.run.smithi192.stdout: mon clock drift allowed = 1.000 2023-10-06T03:28:39.197 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-10-06T03:28:39.197 INFO:teuthology.orchestra.run.smithi192.stdout: osd crush chooseleaf type = 0 2023-10-06T03:28:39.197 INFO:teuthology.orchestra.run.smithi192.stdout: auth debug = true 2023-10-06T03:28:39.198 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-10-06T03:28:39.198 INFO:teuthology.orchestra.run.smithi192.stdout: ms die on old message = true 2023-10-06T03:28:39.198 INFO:teuthology.orchestra.run.smithi192.stdout: ms die on bug = true 2023-10-06T03:28:39.198 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-10-06T03:28:39.198 INFO:teuthology.orchestra.run.smithi192.stdout: mon max pg per osd = 10000 # >= luminous 2023-10-06T03:28:39.198 INFO:teuthology.orchestra.run.smithi192.stdout: mon pg warn max object skew = 0 2023-10-06T03:28:39.198 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-10-06T03:28:39.198 INFO:teuthology.orchestra.run.smithi192.stdout: # disable pg_autoscaler by default for new pools 2023-10-06T03:28:39.198 INFO:teuthology.orchestra.run.smithi192.stdout: osd_pool_default_pg_autoscale_mode = off 2023-10-06T03:28:39.198 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-10-06T03:28:39.198 INFO:teuthology.orchestra.run.smithi192.stdout: osd pool default size = 2 2023-10-06T03:28:39.199 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-10-06T03:28:39.199 INFO:teuthology.orchestra.run.smithi192.stdout: mon osd allow primary affinity = true 2023-10-06T03:28:39.199 INFO:teuthology.orchestra.run.smithi192.stdout: mon osd allow pg remap = true 2023-10-06T03:28:39.199 INFO:teuthology.orchestra.run.smithi192.stdout: mon warn on legacy crush tunables = false 2023-10-06T03:28:39.199 INFO:teuthology.orchestra.run.smithi192.stdout: mon warn on crush straw calc version zero = false 2023-10-06T03:28:39.199 INFO:teuthology.orchestra.run.smithi192.stdout: mon warn on no sortbitwise = false 2023-10-06T03:28:39.199 INFO:teuthology.orchestra.run.smithi192.stdout: mon warn on osd down out interval zero = false 2023-10-06T03:28:39.199 INFO:teuthology.orchestra.run.smithi192.stdout: mon warn on too few osds = false 2023-10-06T03:28:39.199 INFO:teuthology.orchestra.run.smithi192.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-10-06T03:28:39.199 INFO:teuthology.orchestra.run.smithi192.stdout: mon_warn_on_pool_no_redundancy = false 2023-10-06T03:28:39.199 INFO:teuthology.orchestra.run.smithi192.stdout: mon_allow_pool_size_one = true 2023-10-06T03:28:39.200 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-10-06T03:28:39.200 INFO:teuthology.orchestra.run.smithi192.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-10-06T03:28:39.200 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-10-06T03:28:39.200 INFO:teuthology.orchestra.run.smithi192.stdout: osd default data pool replay window = 5 2023-10-06T03:28:39.200 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-10-06T03:28:39.200 INFO:teuthology.orchestra.run.smithi192.stdout: mon allow pool delete = true 2023-10-06T03:28:39.200 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-10-06T03:28:39.200 INFO:teuthology.orchestra.run.smithi192.stdout: mon cluster log file level = debug 2023-10-06T03:28:39.200 INFO:teuthology.orchestra.run.smithi192.stdout: debug asserts on shutdown = true 2023-10-06T03:28:39.200 INFO:teuthology.orchestra.run.smithi192.stdout: mon health detail to clog = false 2023-10-06T03:28:39.200 INFO:teuthology.orchestra.run.smithi192.stdout: mon host = "172.21.15.110,172.21.15.192,[v2:172.21.15.192:3301,v1:172.21.15.192:6790]" 2023-10-06T03:28:39.200 INFO:teuthology.orchestra.run.smithi192.stdout: fsid = 93b8f6ac-4c92-4b45-b8e4-b59617024aa2 2023-10-06T03:28:39.201 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-10-06T03:28:39.201 INFO:teuthology.orchestra.run.smithi192.stdout:[osd] 2023-10-06T03:28:39.201 INFO:teuthology.orchestra.run.smithi192.stdout: osd journal size = 100 2023-10-06T03:28:39.201 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-10-06T03:28:39.201 INFO:teuthology.orchestra.run.smithi192.stdout: osd scrub load threshold = 5.0 2023-10-06T03:28:39.201 INFO:teuthology.orchestra.run.smithi192.stdout: osd scrub max interval = 600 2023-10-06T03:28:39.201 INFO:teuthology.orchestra.run.smithi192.stdout: osd mclock profile = high_recovery_ops 2023-10-06T03:28:39.201 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-10-06T03:28:39.201 INFO:teuthology.orchestra.run.smithi192.stdout: osd recover clone overlap = true 2023-10-06T03:28:39.201 INFO:teuthology.orchestra.run.smithi192.stdout: osd recovery max chunk = 1048576 2023-10-06T03:28:39.201 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-10-06T03:28:39.202 INFO:teuthology.orchestra.run.smithi192.stdout: osd debug shutdown = true 2023-10-06T03:28:39.202 INFO:teuthology.orchestra.run.smithi192.stdout: osd debug op order = true 2023-10-06T03:28:39.202 INFO:teuthology.orchestra.run.smithi192.stdout: osd debug verify stray on activate = true 2023-10-06T03:28:39.202 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-10-06T03:28:39.202 INFO:teuthology.orchestra.run.smithi192.stdout: osd open classes on start = true 2023-10-06T03:28:39.202 INFO:teuthology.orchestra.run.smithi192.stdout: osd debug pg log writeout = true 2023-10-06T03:28:39.202 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-10-06T03:28:39.202 INFO:teuthology.orchestra.run.smithi192.stdout: osd deep scrub update digest min age = 30 2023-10-06T03:28:39.202 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-10-06T03:28:39.202 INFO:teuthology.orchestra.run.smithi192.stdout: osd map max advance = 10 2023-10-06T03:28:39.202 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-10-06T03:28:39.202 INFO:teuthology.orchestra.run.smithi192.stdout: journal zero on create = true 2023-10-06T03:28:39.203 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-10-06T03:28:39.203 INFO:teuthology.orchestra.run.smithi192.stdout: filestore ondisk finisher threads = 3 2023-10-06T03:28:39.203 INFO:teuthology.orchestra.run.smithi192.stdout: filestore apply finisher threads = 3 2023-10-06T03:28:39.203 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-10-06T03:28:39.203 INFO:teuthology.orchestra.run.smithi192.stdout: bdev debug aio = true 2023-10-06T03:28:39.203 INFO:teuthology.orchestra.run.smithi192.stdout: osd debug misdirected ops = true 2023-10-06T03:28:39.203 INFO:teuthology.orchestra.run.smithi192.stdout: bluestore block size = 96636764160 2023-10-06T03:28:39.203 INFO:teuthology.orchestra.run.smithi192.stdout: bluestore fsck on mount = True 2023-10-06T03:28:39.203 INFO:teuthology.orchestra.run.smithi192.stdout: debug bluefs = 20 2023-10-06T03:28:39.203 INFO:teuthology.orchestra.run.smithi192.stdout: debug bluestore = 20 2023-10-06T03:28:39.203 INFO:teuthology.orchestra.run.smithi192.stdout: debug ms = 1 2023-10-06T03:28:39.203 INFO:teuthology.orchestra.run.smithi192.stdout: debug osd = 20 2023-10-06T03:28:39.204 INFO:teuthology.orchestra.run.smithi192.stdout: debug rocksdb = 10 2023-10-06T03:28:39.204 INFO:teuthology.orchestra.run.smithi192.stdout: mon osd backfillfull_ratio = 0.85 2023-10-06T03:28:39.204 INFO:teuthology.orchestra.run.smithi192.stdout: mon osd full ratio = 0.9 2023-10-06T03:28:39.204 INFO:teuthology.orchestra.run.smithi192.stdout: mon osd nearfull ratio = 0.8 2023-10-06T03:28:39.204 INFO:teuthology.orchestra.run.smithi192.stdout: osd failsafe full ratio = 0.95 2023-10-06T03:28:39.204 INFO:teuthology.orchestra.run.smithi192.stdout: osd objectstore = bluestore 2023-10-06T03:28:39.204 INFO:teuthology.orchestra.run.smithi192.stdout: osd op complaint time = 180 2023-10-06T03:28:39.204 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-10-06T03:28:39.204 INFO:teuthology.orchestra.run.smithi192.stdout:[mgr] 2023-10-06T03:28:39.204 INFO:teuthology.orchestra.run.smithi192.stdout: debug ms = 1 2023-10-06T03:28:39.204 INFO:teuthology.orchestra.run.smithi192.stdout: debug mgr = 20 2023-10-06T03:28:39.205 INFO:teuthology.orchestra.run.smithi192.stdout: debug mon = 20 2023-10-06T03:28:39.205 INFO:teuthology.orchestra.run.smithi192.stdout: debug auth = 20 2023-10-06T03:28:39.205 INFO:teuthology.orchestra.run.smithi192.stdout: mon reweight min pgs per osd = 4 2023-10-06T03:28:39.205 INFO:teuthology.orchestra.run.smithi192.stdout: mon reweight min bytes per osd = 10 2023-10-06T03:28:39.205 INFO:teuthology.orchestra.run.smithi192.stdout: mgr/telemetry/nag = false 2023-10-06T03:28:39.205 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-10-06T03:28:39.205 INFO:teuthology.orchestra.run.smithi192.stdout:[mon] 2023-10-06T03:28:39.205 INFO:teuthology.orchestra.run.smithi192.stdout: debug ms = 1 2023-10-06T03:28:39.205 INFO:teuthology.orchestra.run.smithi192.stdout: debug mon = 20 2023-10-06T03:28:39.205 INFO:teuthology.orchestra.run.smithi192.stdout: debug paxos = 20 2023-10-06T03:28:39.205 INFO:teuthology.orchestra.run.smithi192.stdout: debug auth = 20 2023-10-06T03:28:39.205 INFO:teuthology.orchestra.run.smithi192.stdout: mon data avail warn = 5 2023-10-06T03:28:39.206 INFO:teuthology.orchestra.run.smithi192.stdout: mon mgr mkfs grace = 240 2023-10-06T03:28:39.206 INFO:teuthology.orchestra.run.smithi192.stdout: mon reweight min pgs per osd = 4 2023-10-06T03:28:39.206 INFO:teuthology.orchestra.run.smithi192.stdout: mon osd reporter subtree level = osd 2023-10-06T03:28:39.206 INFO:teuthology.orchestra.run.smithi192.stdout: mon osd prime pg temp = true 2023-10-06T03:28:39.206 INFO:teuthology.orchestra.run.smithi192.stdout: mon reweight min bytes per osd = 10 2023-10-06T03:28:39.206 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-10-06T03:28:39.206 INFO:teuthology.orchestra.run.smithi192.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-10-06T03:28:39.206 INFO:teuthology.orchestra.run.smithi192.stdout: auth mon ticket ttl = 660 # 11m 2023-10-06T03:28:39.206 INFO:teuthology.orchestra.run.smithi192.stdout: auth service ticket ttl = 240 # 4m 2023-10-06T03:28:39.206 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-10-06T03:28:39.206 INFO:teuthology.orchestra.run.smithi192.stdout: # don't complain about insecure global_id in the test suite 2023-10-06T03:28:39.207 INFO:teuthology.orchestra.run.smithi192.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-10-06T03:28:39.207 INFO:teuthology.orchestra.run.smithi192.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-10-06T03:28:39.207 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-10-06T03:28:39.207 INFO:teuthology.orchestra.run.smithi192.stdout: # 1m isn't quite enough 2023-10-06T03:28:39.207 INFO:teuthology.orchestra.run.smithi192.stdout: mon_down_mkfs_grace = 2m 2023-10-06T03:28:39.207 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-10-06T03:28:39.207 INFO:teuthology.orchestra.run.smithi192.stdout: mon_warn_on_filestore_osds = false 2023-10-06T03:28:39.207 INFO:teuthology.orchestra.run.smithi192.stdout: mon op complaint time = 120 2023-10-06T03:28:39.207 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-10-06T03:28:39.207 INFO:teuthology.orchestra.run.smithi192.stdout:[client] 2023-10-06T03:28:39.207 INFO:teuthology.orchestra.run.smithi192.stdout: rgw cache enabled = true 2023-10-06T03:28:39.207 INFO:teuthology.orchestra.run.smithi192.stdout: rgw enable ops log = true 2023-10-06T03:28:39.208 INFO:teuthology.orchestra.run.smithi192.stdout: rgw enable usage log = true 2023-10-06T03:28:39.208 INFO:teuthology.orchestra.run.smithi192.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-10-06T03:28:39.208 INFO:teuthology.orchestra.run.smithi192.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-10-06T03:28:39.208 INFO:teuthology.orchestra.run.smithi192.stdout: client mount timeout = 600 2023-10-06T03:28:39.208 INFO:teuthology.orchestra.run.smithi192.stdout: client snapdir = .customsnapfuse 2023-10-06T03:28:39.208 INFO:teuthology.orchestra.run.smithi192.stdout: debug client = 20 2023-10-06T03:28:39.208 INFO:teuthology.orchestra.run.smithi192.stdout: debug ms = 1 2023-10-06T03:28:39.208 INFO:teuthology.orchestra.run.smithi192.stdout: pid file = "" 2023-10-06T03:28:39.208 INFO:teuthology.orchestra.run.smithi192.stdout: rados mon op timeout = 900 2023-10-06T03:28:39.208 INFO:teuthology.orchestra.run.smithi192.stdout: rados osd op timeout = 900 2023-10-06T03:28:39.208 INFO:teuthology.orchestra.run.smithi192.stdout:[mon.a] 2023-10-06T03:28:39.208 INFO:teuthology.orchestra.run.smithi192.stdout:[mon.b] 2023-10-06T03:28:39.209 INFO:teuthology.orchestra.run.smithi192.stdout:[mon.c] 2023-10-06T03:28:39.209 INFO:teuthology.orchestra.run.smithi192.stdout:[mds.a] 2023-10-06T03:28:39.209 INFO:teuthology.orchestra.run.smithi192.stdout:[mds.c] 2023-10-06T03:28:39.209 INFO:teuthology.orchestra.run.smithi192.stdout:[mds.b] 2023-10-06T03:28:39.209 INFO:teuthology.orchestra.run.smithi192.stdout:[mds.d] 2023-10-06T03:28:39.209 INFO:teuthology.orchestra.run.smithi192.stdout:[mds] 2023-10-06T03:28:39.209 INFO:teuthology.orchestra.run.smithi192.stdout: debug mds = 20 2023-10-06T03:28:39.209 INFO:teuthology.orchestra.run.smithi192.stdout: debug ms = 1 2023-10-06T03:28:39.209 INFO:teuthology.orchestra.run.smithi192.stdout: mds debug frag = True 2023-10-06T03:28:39.209 INFO:teuthology.orchestra.run.smithi192.stdout: mds debug scatterstat = True 2023-10-06T03:28:39.209 INFO:teuthology.orchestra.run.smithi192.stdout: mds op complaint time = 180 2023-10-06T03:28:39.209 INFO:teuthology.orchestra.run.smithi192.stdout: mds verify scatter = True 2023-10-06T03:28:39.210 INFO:teuthology.orchestra.run.smithi192.stdout: osd op complaint time = 180 2023-10-06T03:28:39.210 INFO:teuthology.orchestra.run.smithi192.stdout: rados mon op timeout = 900 2023-10-06T03:28:39.210 INFO:teuthology.orchestra.run.smithi192.stdout: rados osd op timeout = 900 2023-10-06T03:28:39.211 INFO:teuthology.orchestra.run.smithi110.stdout:[global] 2023-10-06T03:28:39.211 INFO:teuthology.orchestra.run.smithi110.stdout: chdir = "" 2023-10-06T03:28:39.211 INFO:teuthology.orchestra.run.smithi110.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-10-06T03:28:39.212 INFO:teuthology.orchestra.run.smithi110.stdout: auth supported = cephx 2023-10-06T03:28:39.212 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-10-06T03:28:39.212 INFO:teuthology.orchestra.run.smithi110.stdout: filestore xattr use omap = true 2023-10-06T03:28:39.212 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-10-06T03:28:39.212 INFO:teuthology.orchestra.run.smithi110.stdout: mon clock drift allowed = 1.000 2023-10-06T03:28:39.212 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-10-06T03:28:39.212 INFO:teuthology.orchestra.run.smithi110.stdout: osd crush chooseleaf type = 0 2023-10-06T03:28:39.212 INFO:teuthology.orchestra.run.smithi110.stdout: auth debug = true 2023-10-06T03:28:39.212 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-10-06T03:28:39.212 INFO:teuthology.orchestra.run.smithi110.stdout: ms die on old message = true 2023-10-06T03:28:39.212 INFO:teuthology.orchestra.run.smithi110.stdout: ms die on bug = true 2023-10-06T03:28:39.212 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-10-06T03:28:39.213 INFO:teuthology.orchestra.run.smithi110.stdout: mon max pg per osd = 10000 # >= luminous 2023-10-06T03:28:39.213 INFO:teuthology.orchestra.run.smithi110.stdout: mon pg warn max object skew = 0 2023-10-06T03:28:39.213 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-10-06T03:28:39.213 INFO:teuthology.orchestra.run.smithi110.stdout: # disable pg_autoscaler by default for new pools 2023-10-06T03:28:39.213 INFO:teuthology.orchestra.run.smithi110.stdout: osd_pool_default_pg_autoscale_mode = off 2023-10-06T03:28:39.213 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-10-06T03:28:39.213 INFO:teuthology.orchestra.run.smithi110.stdout: osd pool default size = 2 2023-10-06T03:28:39.213 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-10-06T03:28:39.213 INFO:teuthology.orchestra.run.smithi110.stdout: mon osd allow primary affinity = true 2023-10-06T03:28:39.213 INFO:teuthology.orchestra.run.smithi110.stdout: mon osd allow pg remap = true 2023-10-06T03:28:39.213 INFO:teuthology.orchestra.run.smithi110.stdout: mon warn on legacy crush tunables = false 2023-10-06T03:28:39.213 INFO:teuthology.orchestra.run.smithi110.stdout: mon warn on crush straw calc version zero = false 2023-10-06T03:28:39.214 INFO:teuthology.orchestra.run.smithi110.stdout: mon warn on no sortbitwise = false 2023-10-06T03:28:39.214 INFO:teuthology.orchestra.run.smithi110.stdout: mon warn on osd down out interval zero = false 2023-10-06T03:28:39.214 INFO:teuthology.orchestra.run.smithi110.stdout: mon warn on too few osds = false 2023-10-06T03:28:39.214 INFO:teuthology.orchestra.run.smithi110.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-10-06T03:28:39.214 INFO:teuthology.orchestra.run.smithi110.stdout: mon_warn_on_pool_no_redundancy = false 2023-10-06T03:28:39.214 INFO:teuthology.orchestra.run.smithi110.stdout: mon_allow_pool_size_one = true 2023-10-06T03:28:39.214 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-10-06T03:28:39.214 INFO:teuthology.orchestra.run.smithi110.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-10-06T03:28:39.214 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-10-06T03:28:39.214 INFO:teuthology.orchestra.run.smithi110.stdout: osd default data pool replay window = 5 2023-10-06T03:28:39.214 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-10-06T03:28:39.215 INFO:teuthology.orchestra.run.smithi110.stdout: mon allow pool delete = true 2023-10-06T03:28:39.215 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-10-06T03:28:39.215 INFO:teuthology.orchestra.run.smithi110.stdout: mon cluster log file level = debug 2023-10-06T03:28:39.215 INFO:teuthology.orchestra.run.smithi110.stdout: debug asserts on shutdown = true 2023-10-06T03:28:39.215 INFO:teuthology.orchestra.run.smithi110.stdout: mon health detail to clog = false 2023-10-06T03:28:39.215 INFO:teuthology.orchestra.run.smithi110.stdout: mon host = "172.21.15.110,172.21.15.192,[v2:172.21.15.192:3301,v1:172.21.15.192:6790]" 2023-10-06T03:28:39.215 INFO:teuthology.orchestra.run.smithi110.stdout: fsid = 93b8f6ac-4c92-4b45-b8e4-b59617024aa2 2023-10-06T03:28:39.215 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-10-06T03:28:39.215 INFO:teuthology.orchestra.run.smithi110.stdout:[osd] 2023-10-06T03:28:39.215 INFO:teuthology.orchestra.run.smithi110.stdout: osd journal size = 100 2023-10-06T03:28:39.215 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-10-06T03:28:39.215 INFO:teuthology.orchestra.run.smithi110.stdout: osd scrub load threshold = 5.0 2023-10-06T03:28:39.216 INFO:teuthology.orchestra.run.smithi110.stdout: osd scrub max interval = 600 2023-10-06T03:28:39.216 INFO:teuthology.orchestra.run.smithi110.stdout: osd mclock profile = high_recovery_ops 2023-10-06T03:28:39.216 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-10-06T03:28:39.216 INFO:teuthology.orchestra.run.smithi110.stdout: osd recover clone overlap = true 2023-10-06T03:28:39.216 INFO:teuthology.orchestra.run.smithi110.stdout: osd recovery max chunk = 1048576 2023-10-06T03:28:39.216 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-10-06T03:28:39.216 INFO:teuthology.orchestra.run.smithi110.stdout: osd debug shutdown = true 2023-10-06T03:28:39.216 INFO:teuthology.orchestra.run.smithi110.stdout: osd debug op order = true 2023-10-06T03:28:39.216 INFO:teuthology.orchestra.run.smithi110.stdout: osd debug verify stray on activate = true 2023-10-06T03:28:39.216 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-10-06T03:28:39.216 INFO:teuthology.orchestra.run.smithi110.stdout: osd open classes on start = true 2023-10-06T03:28:39.216 INFO:teuthology.orchestra.run.smithi110.stdout: osd debug pg log writeout = true 2023-10-06T03:28:39.217 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-10-06T03:28:39.217 INFO:teuthology.orchestra.run.smithi110.stdout: osd deep scrub update digest min age = 30 2023-10-06T03:28:39.217 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-10-06T03:28:39.217 INFO:teuthology.orchestra.run.smithi110.stdout: osd map max advance = 10 2023-10-06T03:28:39.217 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-10-06T03:28:39.217 INFO:teuthology.orchestra.run.smithi110.stdout: journal zero on create = true 2023-10-06T03:28:39.217 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-10-06T03:28:39.217 INFO:teuthology.orchestra.run.smithi110.stdout: filestore ondisk finisher threads = 3 2023-10-06T03:28:39.217 INFO:teuthology.orchestra.run.smithi110.stdout: filestore apply finisher threads = 3 2023-10-06T03:28:39.217 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-10-06T03:28:39.217 INFO:teuthology.orchestra.run.smithi110.stdout: bdev debug aio = true 2023-10-06T03:28:39.218 INFO:teuthology.orchestra.run.smithi110.stdout: osd debug misdirected ops = true 2023-10-06T03:28:39.218 INFO:teuthology.orchestra.run.smithi110.stdout: bluestore block size = 96636764160 2023-10-06T03:28:39.218 INFO:teuthology.orchestra.run.smithi110.stdout: bluestore fsck on mount = True 2023-10-06T03:28:39.218 INFO:teuthology.orchestra.run.smithi110.stdout: debug bluefs = 20 2023-10-06T03:28:39.218 INFO:teuthology.orchestra.run.smithi110.stdout: debug bluestore = 20 2023-10-06T03:28:39.218 INFO:teuthology.orchestra.run.smithi110.stdout: debug ms = 1 2023-10-06T03:28:39.218 INFO:teuthology.orchestra.run.smithi110.stdout: debug osd = 20 2023-10-06T03:28:39.218 INFO:teuthology.orchestra.run.smithi110.stdout: debug rocksdb = 10 2023-10-06T03:28:39.218 INFO:teuthology.orchestra.run.smithi110.stdout: mon osd backfillfull_ratio = 0.85 2023-10-06T03:28:39.218 INFO:teuthology.orchestra.run.smithi110.stdout: mon osd full ratio = 0.9 2023-10-06T03:28:39.218 INFO:teuthology.orchestra.run.smithi110.stdout: mon osd nearfull ratio = 0.8 2023-10-06T03:28:39.219 INFO:teuthology.orchestra.run.smithi110.stdout: osd failsafe full ratio = 0.95 2023-10-06T03:28:39.219 INFO:teuthology.orchestra.run.smithi110.stdout: osd objectstore = bluestore 2023-10-06T03:28:39.219 INFO:teuthology.orchestra.run.smithi110.stdout: osd op complaint time = 180 2023-10-06T03:28:39.219 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-10-06T03:28:39.219 INFO:teuthology.orchestra.run.smithi110.stdout:[mgr] 2023-10-06T03:28:39.219 INFO:teuthology.orchestra.run.smithi110.stdout: debug ms = 1 2023-10-06T03:28:39.219 INFO:teuthology.orchestra.run.smithi110.stdout: debug mgr = 20 2023-10-06T03:28:39.219 INFO:teuthology.orchestra.run.smithi110.stdout: debug mon = 20 2023-10-06T03:28:39.219 INFO:teuthology.orchestra.run.smithi110.stdout: debug auth = 20 2023-10-06T03:28:39.219 INFO:teuthology.orchestra.run.smithi110.stdout: mon reweight min pgs per osd = 4 2023-10-06T03:28:39.219 INFO:teuthology.orchestra.run.smithi110.stdout: mon reweight min bytes per osd = 10 2023-10-06T03:28:39.219 INFO:teuthology.orchestra.run.smithi110.stdout: mgr/telemetry/nag = false 2023-10-06T03:28:39.220 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-10-06T03:28:39.220 INFO:teuthology.orchestra.run.smithi110.stdout:[mon] 2023-10-06T03:28:39.220 INFO:teuthology.orchestra.run.smithi110.stdout: debug ms = 1 2023-10-06T03:28:39.220 INFO:teuthology.orchestra.run.smithi110.stdout: debug mon = 20 2023-10-06T03:28:39.220 INFO:teuthology.orchestra.run.smithi110.stdout: debug paxos = 20 2023-10-06T03:28:39.220 INFO:teuthology.orchestra.run.smithi110.stdout: debug auth = 20 2023-10-06T03:28:39.220 INFO:teuthology.orchestra.run.smithi110.stdout: mon data avail warn = 5 2023-10-06T03:28:39.220 INFO:teuthology.orchestra.run.smithi110.stdout: mon mgr mkfs grace = 240 2023-10-06T03:28:39.220 INFO:teuthology.orchestra.run.smithi110.stdout: mon reweight min pgs per osd = 4 2023-10-06T03:28:39.220 INFO:teuthology.orchestra.run.smithi110.stdout: mon osd reporter subtree level = osd 2023-10-06T03:28:39.220 INFO:teuthology.orchestra.run.smithi110.stdout: mon osd prime pg temp = true 2023-10-06T03:28:39.220 INFO:teuthology.orchestra.run.smithi110.stdout: mon reweight min bytes per osd = 10 2023-10-06T03:28:39.221 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-10-06T03:28:39.221 INFO:teuthology.orchestra.run.smithi110.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-10-06T03:28:39.221 INFO:teuthology.orchestra.run.smithi110.stdout: auth mon ticket ttl = 660 # 11m 2023-10-06T03:28:39.221 INFO:teuthology.orchestra.run.smithi110.stdout: auth service ticket ttl = 240 # 4m 2023-10-06T03:28:39.221 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-10-06T03:28:39.221 INFO:teuthology.orchestra.run.smithi110.stdout: # don't complain about insecure global_id in the test suite 2023-10-06T03:28:39.221 INFO:teuthology.orchestra.run.smithi110.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-10-06T03:28:39.221 INFO:teuthology.orchestra.run.smithi110.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-10-06T03:28:39.221 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-10-06T03:28:39.221 INFO:teuthology.orchestra.run.smithi110.stdout: # 1m isn't quite enough 2023-10-06T03:28:39.221 INFO:teuthology.orchestra.run.smithi110.stdout: mon_down_mkfs_grace = 2m 2023-10-06T03:28:39.221 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-10-06T03:28:39.222 INFO:teuthology.orchestra.run.smithi110.stdout: mon_warn_on_filestore_osds = false 2023-10-06T03:28:39.222 INFO:teuthology.orchestra.run.smithi110.stdout: mon op complaint time = 120 2023-10-06T03:28:39.222 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-10-06T03:28:39.222 INFO:teuthology.orchestra.run.smithi110.stdout:[client] 2023-10-06T03:28:39.222 INFO:teuthology.orchestra.run.smithi110.stdout: rgw cache enabled = true 2023-10-06T03:28:39.222 INFO:teuthology.orchestra.run.smithi110.stdout: rgw enable ops log = true 2023-10-06T03:28:39.222 INFO:teuthology.orchestra.run.smithi110.stdout: rgw enable usage log = true 2023-10-06T03:28:39.222 INFO:teuthology.orchestra.run.smithi110.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-10-06T03:28:39.222 INFO:teuthology.orchestra.run.smithi110.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-10-06T03:28:39.222 INFO:teuthology.orchestra.run.smithi110.stdout: client mount timeout = 600 2023-10-06T03:28:39.222 INFO:teuthology.orchestra.run.smithi110.stdout: client snapdir = .customsnapfuse 2023-10-06T03:28:39.223 INFO:teuthology.orchestra.run.smithi110.stdout: debug client = 20 2023-10-06T03:28:39.223 INFO:teuthology.orchestra.run.smithi110.stdout: debug ms = 1 2023-10-06T03:28:39.223 INFO:teuthology.orchestra.run.smithi110.stdout: pid file = "" 2023-10-06T03:28:39.223 INFO:teuthology.orchestra.run.smithi110.stdout: rados mon op timeout = 900 2023-10-06T03:28:39.223 INFO:teuthology.orchestra.run.smithi110.stdout: rados osd op timeout = 900 2023-10-06T03:28:39.223 INFO:teuthology.orchestra.run.smithi110.stdout:[mon.a] 2023-10-06T03:28:39.223 INFO:teuthology.orchestra.run.smithi110.stdout:[mon.b] 2023-10-06T03:28:39.223 INFO:teuthology.orchestra.run.smithi110.stdout:[mon.c] 2023-10-06T03:28:39.223 INFO:teuthology.orchestra.run.smithi110.stdout:[mds.a] 2023-10-06T03:28:39.223 INFO:teuthology.orchestra.run.smithi110.stdout:[mds.c] 2023-10-06T03:28:39.223 INFO:teuthology.orchestra.run.smithi110.stdout:[mds.b] 2023-10-06T03:28:39.224 INFO:teuthology.orchestra.run.smithi110.stdout:[mds.d] 2023-10-06T03:28:39.224 INFO:teuthology.orchestra.run.smithi110.stdout:[mds] 2023-10-06T03:28:39.224 INFO:teuthology.orchestra.run.smithi110.stdout: debug mds = 20 2023-10-06T03:28:39.224 INFO:teuthology.orchestra.run.smithi110.stdout: debug ms = 1 2023-10-06T03:28:39.224 INFO:teuthology.orchestra.run.smithi110.stdout: mds debug frag = True 2023-10-06T03:28:39.224 INFO:teuthology.orchestra.run.smithi110.stdout: mds debug scatterstat = True 2023-10-06T03:28:39.224 INFO:teuthology.orchestra.run.smithi110.stdout: mds op complaint time = 180 2023-10-06T03:28:39.224 INFO:teuthology.orchestra.run.smithi110.stdout: mds verify scatter = True 2023-10-06T03:28:39.224 INFO:teuthology.orchestra.run.smithi110.stdout: osd op complaint time = 180 2023-10-06T03:28:39.224 INFO:teuthology.orchestra.run.smithi110.stdout: rados mon op timeout = 900 2023-10-06T03:28:39.224 INFO:teuthology.orchestra.run.smithi110.stdout: rados osd op timeout = 900 2023-10-06T03:28:39.225 INFO:tasks.ceph:Creating admin key on mon.a... 2023-10-06T03:28:39.225 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2023-10-06T03:28:39.308 INFO:tasks.ceph:Copying monmap to all nodes... 2023-10-06T03:28:39.308 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2023-10-06T03:28:39.309 DEBUG:teuthology.orchestra.run.smithi110:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2023-10-06T03:28:39.315 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2023-10-06T03:28:39.315 DEBUG:teuthology.orchestra.run.smithi110:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2023-10-06T03:28:39.322 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi110.front.sepia.ceph.com 2023-10-06T03:28:39.322 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2023-10-06T03:28:39.323 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd of=/etc/ceph/ceph.keyring 2023-10-06T03:28:39.323 DEBUG:teuthology.orchestra.run.smithi110:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-10-06T03:28:39.346 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2023-10-06T03:28:39.347 DEBUG:teuthology.orchestra.run.smithi110:> dd of=/home/ubuntu/cephtest/ceph.monmap 2023-10-06T03:28:39.351 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi192.front.sepia.ceph.com 2023-10-06T03:28:39.351 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2023-10-06T03:28:39.351 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd of=/etc/ceph/ceph.keyring 2023-10-06T03:28:39.351 DEBUG:teuthology.orchestra.run.smithi192:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-10-06T03:28:39.378 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2023-10-06T03:28:39.379 DEBUG:teuthology.orchestra.run.smithi192:> dd of=/home/ubuntu/cephtest/ceph.monmap 2023-10-06T03:28:39.428 INFO:tasks.ceph:Setting up mon nodes... 2023-10-06T03:28:39.428 INFO:tasks.ceph:Setting up mgr nodes... 2023-10-06T03:28:39.429 DEBUG:teuthology.orchestra.run.smithi110:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2023-10-06T03:28:39.474 INFO:teuthology.orchestra.run.smithi110.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2023-10-06T03:28:39.477 DEBUG:teuthology.orchestra.run.smithi192:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2023-10-06T03:28:39.522 INFO:teuthology.orchestra.run.smithi192.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2023-10-06T03:28:39.524 INFO:tasks.ceph:Setting up mds nodes... 2023-10-06T03:28:39.525 DEBUG:teuthology.orchestra.run.smithi110:> sudo mkdir -p /var/lib/ceph/mds/ceph-a && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.a /var/lib/ceph/mds/ceph-a/keyring 2023-10-06T03:28:39.569 INFO:teuthology.orchestra.run.smithi110.stdout:creating /var/lib/ceph/mds/ceph-a/keyring 2023-10-06T03:28:39.571 DEBUG:teuthology.orchestra.run.smithi110:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-a 2023-10-06T03:28:39.587 DEBUG:teuthology.orchestra.run.smithi110:> sudo mkdir -p /var/lib/ceph/mds/ceph-c && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.c /var/lib/ceph/mds/ceph-c/keyring 2023-10-06T03:28:39.626 INFO:teuthology.orchestra.run.smithi110.stdout:creating /var/lib/ceph/mds/ceph-c/keyring 2023-10-06T03:28:39.629 DEBUG:teuthology.orchestra.run.smithi110:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-c 2023-10-06T03:28:39.687 DEBUG:teuthology.orchestra.run.smithi192:> sudo mkdir -p /var/lib/ceph/mds/ceph-b && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.b /var/lib/ceph/mds/ceph-b/keyring 2023-10-06T03:28:39.736 INFO:teuthology.orchestra.run.smithi192.stdout:creating /var/lib/ceph/mds/ceph-b/keyring 2023-10-06T03:28:39.738 DEBUG:teuthology.orchestra.run.smithi192:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-b 2023-10-06T03:28:39.753 DEBUG:teuthology.orchestra.run.smithi192:> sudo mkdir -p /var/lib/ceph/mds/ceph-d && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.d /var/lib/ceph/mds/ceph-d/keyring 2023-10-06T03:28:39.791 INFO:teuthology.orchestra.run.smithi192.stdout:creating /var/lib/ceph/mds/ceph-d/keyring 2023-10-06T03:28:39.793 DEBUG:teuthology.orchestra.run.smithi192:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-d 2023-10-06T03:28:39.808 INFO:tasks.ceph_client:Setting up client nodes... 2023-10-06T03:28:39.808 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2023-10-06T03:28:39.844 INFO:teuthology.orchestra.run.smithi110.stdout:creating /etc/ceph/ceph.client.0.keyring 2023-10-06T03:28:39.857 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.2 /etc/ceph/ceph.client.2.keyring && sudo chmod 0644 /etc/ceph/ceph.client.2.keyring 2023-10-06T03:28:39.890 INFO:teuthology.orchestra.run.smithi110.stdout:creating /etc/ceph/ceph.client.2.keyring 2023-10-06T03:28:39.902 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2023-10-06T03:28:39.932 INFO:teuthology.orchestra.run.smithi192.stdout:creating /etc/ceph/ceph.client.1.keyring 2023-10-06T03:28:39.944 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.3 /etc/ceph/ceph.client.3.keyring && sudo chmod 0644 /etc/ceph/ceph.client.3.keyring 2023-10-06T03:28:39.973 INFO:teuthology.orchestra.run.smithi192.stdout:creating /etc/ceph/ceph.client.3.keyring 2023-10-06T03:28:39.984 INFO:tasks.ceph:Running mkfs on osd nodes... 2023-10-06T03:28:39.984 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi110.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi192.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2023-10-06T03:28:39.985 DEBUG:teuthology.orchestra.run.smithi110:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2023-10-06T03:28:40.001 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-10-06T03:28:40.001 INFO:tasks.ceph:role: osd.0 2023-10-06T03:28:40.001 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi110.front.sepia.ceph.com 2023-10-06T03:28:40.002 DEBUG:teuthology.orchestra.run.smithi110:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-10-06T03:28:40.041 INFO:teuthology.orchestra.run.smithi110.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-10-06T03:28:40.041 INFO:teuthology.orchestra.run.smithi110.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-06T03:28:40.041 INFO:teuthology.orchestra.run.smithi110.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-06T03:28:40.041 INFO:teuthology.orchestra.run.smithi110.stdout: = reflink=1 2023-10-06T03:28:40.041 INFO:teuthology.orchestra.run.smithi110.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-06T03:28:40.041 INFO:teuthology.orchestra.run.smithi110.stdout: = sunit=0 swidth=0 blks 2023-10-06T03:28:40.042 INFO:teuthology.orchestra.run.smithi110.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-06T03:28:40.042 INFO:teuthology.orchestra.run.smithi110.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-10-06T03:28:40.042 INFO:teuthology.orchestra.run.smithi110.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-06T03:28:40.042 INFO:teuthology.orchestra.run.smithi110.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-06T03:28:40.043 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi110.front.sepia.ceph.com -o noatime 2023-10-06T03:28:40.043 DEBUG:teuthology.orchestra.run.smithi110:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2023-10-06T03:28:40.064 DEBUG:teuthology.orchestra.run.smithi110:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2023-10-06T03:28:40.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-06T03:28:40.071 INFO:teuthology.orchestra.run.smithi110.stderr:sudo: /sbin/restorecon: command not found 2023-10-06T03:28:40.071 DEBUG:teuthology.orchestra.run.smithi110:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2023-10-06T03:28:40.081 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-10-06T03:28:40.082 INFO:tasks.ceph:role: osd.1 2023-10-06T03:28:40.082 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi110.front.sepia.ceph.com 2023-10-06T03:28:40.082 DEBUG:teuthology.orchestra.run.smithi110:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-10-06T03:28:40.166 INFO:teuthology.orchestra.run.smithi110.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-10-06T03:28:40.166 INFO:teuthology.orchestra.run.smithi110.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-06T03:28:40.166 INFO:teuthology.orchestra.run.smithi110.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-06T03:28:40.166 INFO:teuthology.orchestra.run.smithi110.stdout: = reflink=1 2023-10-06T03:28:40.167 INFO:teuthology.orchestra.run.smithi110.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-06T03:28:40.167 INFO:teuthology.orchestra.run.smithi110.stdout: = sunit=0 swidth=0 blks 2023-10-06T03:28:40.167 INFO:teuthology.orchestra.run.smithi110.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-06T03:28:40.167 INFO:teuthology.orchestra.run.smithi110.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-10-06T03:28:40.167 INFO:teuthology.orchestra.run.smithi110.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-06T03:28:40.167 INFO:teuthology.orchestra.run.smithi110.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-06T03:28:40.168 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi110.front.sepia.ceph.com -o noatime 2023-10-06T03:28:40.168 DEBUG:teuthology.orchestra.run.smithi110:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2023-10-06T03:28:40.185 DEBUG:teuthology.orchestra.run.smithi110:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2023-10-06T03:28:40.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-06T03:28:40.193 INFO:teuthology.orchestra.run.smithi110.stderr:sudo: /sbin/restorecon: command not found 2023-10-06T03:28:40.194 DEBUG:teuthology.orchestra.run.smithi110:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2023-10-06T03:28:40.204 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-10-06T03:28:40.204 INFO:tasks.ceph:role: osd.2 2023-10-06T03:28:40.204 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi110.front.sepia.ceph.com 2023-10-06T03:28:40.204 DEBUG:teuthology.orchestra.run.smithi110:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-10-06T03:28:40.290 INFO:teuthology.orchestra.run.smithi110.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-10-06T03:28:40.290 INFO:teuthology.orchestra.run.smithi110.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-06T03:28:40.290 INFO:teuthology.orchestra.run.smithi110.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-06T03:28:40.290 INFO:teuthology.orchestra.run.smithi110.stdout: = reflink=1 2023-10-06T03:28:40.290 INFO:teuthology.orchestra.run.smithi110.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-06T03:28:40.290 INFO:teuthology.orchestra.run.smithi110.stdout: = sunit=0 swidth=0 blks 2023-10-06T03:28:40.290 INFO:teuthology.orchestra.run.smithi110.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-06T03:28:40.290 INFO:teuthology.orchestra.run.smithi110.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-10-06T03:28:40.291 INFO:teuthology.orchestra.run.smithi110.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-06T03:28:40.291 INFO:teuthology.orchestra.run.smithi110.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-06T03:28:40.291 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi110.front.sepia.ceph.com -o noatime 2023-10-06T03:28:40.292 DEBUG:teuthology.orchestra.run.smithi110:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2023-10-06T03:28:40.308 DEBUG:teuthology.orchestra.run.smithi110:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2023-10-06T03:28:40.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-06T03:28:40.318 INFO:teuthology.orchestra.run.smithi110.stderr:sudo: /sbin/restorecon: command not found 2023-10-06T03:28:40.318 DEBUG:teuthology.orchestra.run.smithi110:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2023-10-06T03:28:40.328 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-10-06T03:28:40.328 INFO:tasks.ceph:role: osd.3 2023-10-06T03:28:40.328 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi110.front.sepia.ceph.com 2023-10-06T03:28:40.328 DEBUG:teuthology.orchestra.run.smithi110:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2023-10-06T03:28:40.369 INFO:teuthology.orchestra.run.smithi110.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2023-10-06T03:28:40.369 INFO:teuthology.orchestra.run.smithi110.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-06T03:28:40.369 INFO:teuthology.orchestra.run.smithi110.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-06T03:28:40.369 INFO:teuthology.orchestra.run.smithi110.stdout: = reflink=1 2023-10-06T03:28:40.369 INFO:teuthology.orchestra.run.smithi110.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-06T03:28:40.369 INFO:teuthology.orchestra.run.smithi110.stdout: = sunit=0 swidth=0 blks 2023-10-06T03:28:40.369 INFO:teuthology.orchestra.run.smithi110.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-06T03:28:40.369 INFO:teuthology.orchestra.run.smithi110.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-10-06T03:28:40.370 INFO:teuthology.orchestra.run.smithi110.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-06T03:28:40.370 INFO:teuthology.orchestra.run.smithi110.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-06T03:28:40.370 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi110.front.sepia.ceph.com -o noatime 2023-10-06T03:28:40.371 DEBUG:teuthology.orchestra.run.smithi110:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2023-10-06T03:28:40.434 DEBUG:teuthology.orchestra.run.smithi110:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2023-10-06T03:28:40.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-06T03:28:40.448 INFO:teuthology.orchestra.run.smithi110.stderr:sudo: /sbin/restorecon: command not found 2023-10-06T03:28:40.448 DEBUG:teuthology.orchestra.run.smithi110:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-06T03:28:40.512 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:28:40.507+0000 7f94731f43c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2023-10-06T03:28:40.512 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:28:40.507+0000 7f94731f43c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2023-10-06T03:28:40.512 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:28:40.507+0000 7f94731f43c0 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-10-06T03:28:40.512 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:28:40.507+0000 7f94731f43c0 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-10-06T03:28:40.512 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:28:40.507+0000 7f94731f43c0 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-10-06T03:28:40.513 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:28:40.507+0000 7f94731f43c0 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2023-10-06T03:28:41.066 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:28:41.059+0000 7f94731f43c0 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-10-06T03:28:41.847 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:28:41.843+0000 7f94731f43c0 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-10-06T03:28:42.629 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:28:42.623+0000 7f94731f43c0 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-10-06T03:28:43.483 DEBUG:teuthology.orchestra.run.smithi110:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2023-10-06T03:28:43.499 DEBUG:teuthology.orchestra.run.smithi110:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-06T03:28:43.541 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:28:43.535+0000 7f20098103c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2023-10-06T03:28:43.541 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:28:43.535+0000 7f20098103c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2023-10-06T03:28:43.541 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:28:43.535+0000 7f20098103c0 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-10-06T03:28:43.541 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:28:43.535+0000 7f20098103c0 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-10-06T03:28:43.541 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:28:43.535+0000 7f20098103c0 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-10-06T03:28:43.542 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:28:43.535+0000 7f20098103c0 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2023-10-06T03:28:44.095 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:28:44.091+0000 7f20098103c0 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-10-06T03:28:44.890 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:28:44.883+0000 7f20098103c0 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-10-06T03:28:45.678 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:28:45.671+0000 7f20098103c0 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-10-06T03:28:46.519 DEBUG:teuthology.orchestra.run.smithi110:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2023-10-06T03:28:46.533 DEBUG:teuthology.orchestra.run.smithi110:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-06T03:28:46.619 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:28:46.615+0000 7ff600b4a3c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2023-10-06T03:28:46.619 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:28:46.615+0000 7ff600b4a3c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2023-10-06T03:28:46.620 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:28:46.615+0000 7ff600b4a3c0 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-10-06T03:28:46.620 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:28:46.615+0000 7ff600b4a3c0 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-10-06T03:28:46.620 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:28:46.615+0000 7ff600b4a3c0 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-10-06T03:28:46.620 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:28:46.615+0000 7ff600b4a3c0 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2023-10-06T03:28:47.174 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:28:47.167+0000 7ff600b4a3c0 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-10-06T03:28:47.964 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:28:47.959+0000 7ff600b4a3c0 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-10-06T03:28:48.741 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:28:48.735+0000 7ff600b4a3c0 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-10-06T03:28:49.594 DEBUG:teuthology.orchestra.run.smithi110:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2023-10-06T03:28:49.609 DEBUG:teuthology.orchestra.run.smithi110:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-06T03:28:49.648 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:28:49.643+0000 7f0d095ea3c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2023-10-06T03:28:49.648 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:28:49.643+0000 7f0d095ea3c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2023-10-06T03:28:49.648 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:28:49.643+0000 7f0d095ea3c0 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2023-10-06T03:28:49.648 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:28:49.643+0000 7f0d095ea3c0 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2023-10-06T03:28:49.649 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:28:49.643+0000 7f0d095ea3c0 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2023-10-06T03:28:49.649 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:28:49.643+0000 7f0d095ea3c0 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2023-10-06T03:28:50.213 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:28:50.207+0000 7f0d095ea3c0 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-10-06T03:28:50.994 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:28:50.987+0000 7f0d095ea3c0 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-10-06T03:28:51.769 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:28:51.763+0000 7f0d095ea3c0 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-10-06T03:28:52.634 DEBUG:teuthology.orchestra.run.smithi110:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2023-10-06T03:28:52.651 DEBUG:teuthology.orchestra.run.smithi192:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2023-10-06T03:28:52.670 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-10-06T03:28:52.670 INFO:tasks.ceph:role: osd.4 2023-10-06T03:28:52.671 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi192.front.sepia.ceph.com 2023-10-06T03:28:52.671 DEBUG:teuthology.orchestra.run.smithi192:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-10-06T03:28:53.247 INFO:teuthology.orchestra.run.smithi192.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-10-06T03:28:53.247 INFO:teuthology.orchestra.run.smithi192.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-06T03:28:53.247 INFO:teuthology.orchestra.run.smithi192.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-06T03:28:53.247 INFO:teuthology.orchestra.run.smithi192.stdout: = reflink=1 2023-10-06T03:28:53.247 INFO:teuthology.orchestra.run.smithi192.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-06T03:28:53.247 INFO:teuthology.orchestra.run.smithi192.stdout: = sunit=0 swidth=0 blks 2023-10-06T03:28:53.247 INFO:teuthology.orchestra.run.smithi192.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-06T03:28:53.247 INFO:teuthology.orchestra.run.smithi192.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-10-06T03:28:53.247 INFO:teuthology.orchestra.run.smithi192.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-06T03:28:53.248 INFO:teuthology.orchestra.run.smithi192.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-06T03:28:53.249 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi192.front.sepia.ceph.com -o noatime 2023-10-06T03:28:53.249 DEBUG:teuthology.orchestra.run.smithi192:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2023-10-06T03:28:53.340 DEBUG:teuthology.orchestra.run.smithi192:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2023-10-06T03:28:53.353 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: /sbin/restorecon: command not found 2023-10-06T03:28:53.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-06T03:28:53.354 DEBUG:teuthology.orchestra.run.smithi192:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2023-10-06T03:28:53.364 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-10-06T03:28:53.364 INFO:tasks.ceph:role: osd.5 2023-10-06T03:28:53.364 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi192.front.sepia.ceph.com 2023-10-06T03:28:53.365 DEBUG:teuthology.orchestra.run.smithi192:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-10-06T03:28:53.913 INFO:teuthology.orchestra.run.smithi192.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-10-06T03:28:53.914 INFO:teuthology.orchestra.run.smithi192.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-06T03:28:53.915 INFO:teuthology.orchestra.run.smithi192.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-06T03:28:53.915 INFO:teuthology.orchestra.run.smithi192.stdout: = reflink=1 2023-10-06T03:28:53.915 INFO:teuthology.orchestra.run.smithi192.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-06T03:28:53.916 INFO:teuthology.orchestra.run.smithi192.stdout: = sunit=0 swidth=0 blks 2023-10-06T03:28:53.916 INFO:teuthology.orchestra.run.smithi192.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-06T03:28:53.918 INFO:teuthology.orchestra.run.smithi192.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-10-06T03:28:53.918 INFO:teuthology.orchestra.run.smithi192.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-06T03:28:53.918 INFO:teuthology.orchestra.run.smithi192.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-06T03:28:53.919 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi192.front.sepia.ceph.com -o noatime 2023-10-06T03:28:53.920 DEBUG:teuthology.orchestra.run.smithi192:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2023-10-06T03:28:53.961 DEBUG:teuthology.orchestra.run.smithi192:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2023-10-06T03:28:53.973 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: /sbin/restorecon: command not found 2023-10-06T03:28:53.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-06T03:28:53.974 DEBUG:teuthology.orchestra.run.smithi192:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2023-10-06T03:28:54.034 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-10-06T03:28:54.034 INFO:tasks.ceph:role: osd.6 2023-10-06T03:28:54.034 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi192.front.sepia.ceph.com 2023-10-06T03:28:54.035 DEBUG:teuthology.orchestra.run.smithi192:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-10-06T03:28:54.506 INFO:teuthology.orchestra.run.smithi192.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-10-06T03:28:54.506 INFO:teuthology.orchestra.run.smithi192.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-06T03:28:54.506 INFO:teuthology.orchestra.run.smithi192.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-06T03:28:54.506 INFO:teuthology.orchestra.run.smithi192.stdout: = reflink=1 2023-10-06T03:28:54.506 INFO:teuthology.orchestra.run.smithi192.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-06T03:28:54.506 INFO:teuthology.orchestra.run.smithi192.stdout: = sunit=0 swidth=0 blks 2023-10-06T03:28:54.506 INFO:teuthology.orchestra.run.smithi192.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-06T03:28:54.506 INFO:teuthology.orchestra.run.smithi192.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-10-06T03:28:54.506 INFO:teuthology.orchestra.run.smithi192.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-06T03:28:54.507 INFO:teuthology.orchestra.run.smithi192.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-06T03:28:54.507 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi192.front.sepia.ceph.com -o noatime 2023-10-06T03:28:54.508 DEBUG:teuthology.orchestra.run.smithi192:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2023-10-06T03:28:54.569 DEBUG:teuthology.orchestra.run.smithi192:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2023-10-06T03:28:54.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-06T03:28:54.585 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: /sbin/restorecon: command not found 2023-10-06T03:28:54.585 DEBUG:teuthology.orchestra.run.smithi192:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2023-10-06T03:28:54.595 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-10-06T03:28:54.595 INFO:tasks.ceph:role: osd.7 2023-10-06T03:28:54.595 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi192.front.sepia.ceph.com 2023-10-06T03:28:54.595 DEBUG:teuthology.orchestra.run.smithi192:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2023-10-06T03:28:55.144 INFO:teuthology.orchestra.run.smithi192.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2023-10-06T03:28:55.144 INFO:teuthology.orchestra.run.smithi192.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-06T03:28:55.144 INFO:teuthology.orchestra.run.smithi192.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-06T03:28:55.144 INFO:teuthology.orchestra.run.smithi192.stdout: = reflink=1 2023-10-06T03:28:55.144 INFO:teuthology.orchestra.run.smithi192.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-06T03:28:55.144 INFO:teuthology.orchestra.run.smithi192.stdout: = sunit=0 swidth=0 blks 2023-10-06T03:28:55.144 INFO:teuthology.orchestra.run.smithi192.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-06T03:28:55.144 INFO:teuthology.orchestra.run.smithi192.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-10-06T03:28:55.144 INFO:teuthology.orchestra.run.smithi192.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-06T03:28:55.145 INFO:teuthology.orchestra.run.smithi192.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-06T03:28:55.146 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi192.front.sepia.ceph.com -o noatime 2023-10-06T03:28:55.146 DEBUG:teuthology.orchestra.run.smithi192:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2023-10-06T03:28:55.216 DEBUG:teuthology.orchestra.run.smithi192:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2023-10-06T03:28:55.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-06T03:28:55.228 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: /sbin/restorecon: command not found 2023-10-06T03:28:55.229 DEBUG:teuthology.orchestra.run.smithi192:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-06T03:28:55.350 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:28:55.342+0000 7fe8843a23c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2023-10-06T03:28:55.354 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:28:55.346+0000 7fe8843a23c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2023-10-06T03:28:55.354 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:28:55.350+0000 7fe8843a23c0 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2023-10-06T03:28:55.354 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:28:55.350+0000 7fe8843a23c0 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2023-10-06T03:28:55.355 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:28:55.350+0000 7fe8843a23c0 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2023-10-06T03:28:55.359 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:28:55.354+0000 7fe8843a23c0 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2023-10-06T03:28:56.171 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:28:56.166+0000 7fe8843a23c0 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-10-06T03:28:56.958 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:28:56.950+0000 7fe8843a23c0 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-10-06T03:28:57.732 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:28:57.726+0000 7fe8843a23c0 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-10-06T03:28:58.599 DEBUG:teuthology.orchestra.run.smithi192:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2023-10-06T03:28:58.612 DEBUG:teuthology.orchestra.run.smithi192:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-06T03:28:58.649 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:28:58.642+0000 7f58e1f9a3c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2023-10-06T03:28:58.650 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:28:58.642+0000 7f58e1f9a3c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2023-10-06T03:28:58.650 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:28:58.642+0000 7f58e1f9a3c0 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2023-10-06T03:28:58.650 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:28:58.642+0000 7f58e1f9a3c0 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2023-10-06T03:28:58.650 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:28:58.642+0000 7f58e1f9a3c0 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2023-10-06T03:28:58.650 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:28:58.646+0000 7f58e1f9a3c0 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2023-10-06T03:28:59.213 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:28:59.206+0000 7f58e1f9a3c0 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-10-06T03:29:00.001 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:28:59.994+0000 7f58e1f9a3c0 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-10-06T03:29:00.783 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:00.778+0000 7f58e1f9a3c0 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-10-06T03:29:01.634 DEBUG:teuthology.orchestra.run.smithi192:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2023-10-06T03:29:01.649 DEBUG:teuthology.orchestra.run.smithi192:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-06T03:29:01.735 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:01.730+0000 7fa6cc7043c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2023-10-06T03:29:01.735 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:01.730+0000 7fa6cc7043c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2023-10-06T03:29:01.736 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:01.730+0000 7fa6cc7043c0 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2023-10-06T03:29:01.736 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:01.730+0000 7fa6cc7043c0 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2023-10-06T03:29:01.736 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:01.730+0000 7fa6cc7043c0 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2023-10-06T03:29:01.736 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:01.730+0000 7fa6cc7043c0 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2023-10-06T03:29:02.290 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:02.286+0000 7fa6cc7043c0 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-10-06T03:29:03.079 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:03.074+0000 7fa6cc7043c0 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-10-06T03:29:03.867 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:03.862+0000 7fa6cc7043c0 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-10-06T03:29:04.722 DEBUG:teuthology.orchestra.run.smithi192:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2023-10-06T03:29:04.737 DEBUG:teuthology.orchestra.run.smithi192:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-06T03:29:04.777 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:04.770+0000 7f740bd263c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2023-10-06T03:29:04.777 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:04.770+0000 7f740bd263c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2023-10-06T03:29:04.777 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:04.770+0000 7f740bd263c0 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2023-10-06T03:29:04.778 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:04.770+0000 7f740bd263c0 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2023-10-06T03:29:04.778 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:04.770+0000 7f740bd263c0 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2023-10-06T03:29:04.778 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:04.774+0000 7f740bd263c0 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2023-10-06T03:29:05.334 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:05.326+0000 7f740bd263c0 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-10-06T03:29:06.136 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:06.130+0000 7f740bd263c0 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-10-06T03:29:06.936 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:06.930+0000 7f740bd263c0 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-10-06T03:29:07.687 DEBUG:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-06T03:29:07.689 DEBUG:teuthology.orchestra.run.smithi192:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-06T03:29:07.797 DEBUG:teuthology.orchestra.run.smithi192:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2023-10-06T03:29:07.810 INFO:tasks.ceph:Reading keys from all nodes... 2023-10-06T03:29:07.811 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2023-10-06T03:29:07.811 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2023-10-06T03:29:07.827 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2023-10-06T03:29:07.828 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd if=/var/lib/ceph/mds/ceph-a/keyring of=/dev/stdout 2023-10-06T03:29:07.843 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2023-10-06T03:29:07.844 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd if=/var/lib/ceph/mds/ceph-c/keyring of=/dev/stdout 2023-10-06T03:29:07.902 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2023-10-06T03:29:07.903 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2023-10-06T03:29:07.912 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2023-10-06T03:29:07.912 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2023-10-06T03:29:07.927 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2023-10-06T03:29:07.927 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2023-10-06T03:29:07.987 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2023-10-06T03:29:07.987 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2023-10-06T03:29:07.998 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2023-10-06T03:29:07.998 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2023-10-06T03:29:08.014 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2023-10-06T03:29:08.014 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd if=/var/lib/ceph/mds/ceph-b/keyring of=/dev/stdout 2023-10-06T03:29:08.072 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2023-10-06T03:29:08.072 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd if=/var/lib/ceph/mds/ceph-d/keyring of=/dev/stdout 2023-10-06T03:29:08.087 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2023-10-06T03:29:08.087 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2023-10-06T03:29:08.102 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2023-10-06T03:29:08.102 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2023-10-06T03:29:08.114 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2023-10-06T03:29:08.114 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2023-10-06T03:29:08.124 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2023-10-06T03:29:08.124 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2023-10-06T03:29:08.182 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2023-10-06T03:29:08.182 DEBUG:teuthology.orchestra.run.smithi110:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2023-10-06T03:29:08.189 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2023-10-06T03:29:08.189 DEBUG:teuthology.orchestra.run.smithi110:> dd if=/etc/ceph/ceph.client.2.keyring of=/dev/stdout 2023-10-06T03:29:08.237 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2023-10-06T03:29:08.237 DEBUG:teuthology.orchestra.run.smithi192:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2023-10-06T03:29:08.244 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2023-10-06T03:29:08.244 DEBUG:teuthology.orchestra.run.smithi192:> dd if=/etc/ceph/ceph.client.3.keyring of=/dev/stdout 2023-10-06T03:29:08.251 INFO:tasks.ceph:Adding keys to all mons... 2023-10-06T03:29:08.252 DEBUG:teuthology.orchestra.run.smithi110:> sudo tee -a /etc/ceph/ceph.keyring 2023-10-06T03:29:08.254 DEBUG:teuthology.orchestra.run.smithi192:> sudo tee -a /etc/ceph/ceph.keyring 2023-10-06T03:29:08.296 INFO:teuthology.orchestra.run.smithi110.stdout:[mgr.y] 2023-10-06T03:29:08.296 INFO:teuthology.orchestra.run.smithi110.stdout: key = AQDnfh9lH5IzHBAAM72W1T6GBAluxB2R2Gcqtg== 2023-10-06T03:29:08.296 INFO:teuthology.orchestra.run.smithi110.stdout:[mds.a] 2023-10-06T03:29:08.297 INFO:teuthology.orchestra.run.smithi110.stdout: key = AQDnfh9l2k3cIRAAP843sFwH4LEZoCwTml4blQ== 2023-10-06T03:29:08.297 INFO:teuthology.orchestra.run.smithi110.stdout:[mds.c] 2023-10-06T03:29:08.297 INFO:teuthology.orchestra.run.smithi110.stdout: key = AQDnfh9lzTNRJRAAeLHxIFxyWzK0ECu9UEhjSA== 2023-10-06T03:29:08.297 INFO:teuthology.orchestra.run.smithi110.stdout:[osd.0] 2023-10-06T03:29:08.297 INFO:teuthology.orchestra.run.smithi110.stdout: key = AQDofh9ldKF3HhAAYvogispXU05WmG+yC/SnaA== 2023-10-06T03:29:08.297 INFO:teuthology.orchestra.run.smithi110.stdout:[osd.1] 2023-10-06T03:29:08.297 INFO:teuthology.orchestra.run.smithi110.stdout: key = AQDrfh9lrZcsIBAA6tz8bfWAcz7Oti3fQgKVbQ== 2023-10-06T03:29:08.297 INFO:teuthology.orchestra.run.smithi110.stdout:[osd.2] 2023-10-06T03:29:08.297 INFO:teuthology.orchestra.run.smithi110.stdout: key = AQDufh9lC27aJBAAjATAn2aGrDjlOuVJx/iEpw== 2023-10-06T03:29:08.297 INFO:teuthology.orchestra.run.smithi110.stdout:[osd.3] 2023-10-06T03:29:08.297 INFO:teuthology.orchestra.run.smithi110.stdout: key = AQDxfh9lZb2JJhAAfBL3LEcvAP+ZzWh3sZzw+w== 2023-10-06T03:29:08.297 INFO:teuthology.orchestra.run.smithi110.stdout:[mgr.x] 2023-10-06T03:29:08.298 INFO:teuthology.orchestra.run.smithi110.stdout: key = AQDnfh9lleAUHxAAPxyJMFMFjicyyb9MOxspEw== 2023-10-06T03:29:08.298 INFO:teuthology.orchestra.run.smithi110.stdout:[mds.b] 2023-10-06T03:29:08.298 INFO:teuthology.orchestra.run.smithi110.stdout: key = AQDnfh9lq/rVKxAAKyRhV1HafDy0SM+kEboXOQ== 2023-10-06T03:29:08.298 INFO:teuthology.orchestra.run.smithi110.stdout:[mds.d] 2023-10-06T03:29:08.298 INFO:teuthology.orchestra.run.smithi110.stdout: key = AQDnfh9lAHsbLxAA3HjlMJWG9MusdTNd0t/OBw== 2023-10-06T03:29:08.298 INFO:teuthology.orchestra.run.smithi110.stdout:[osd.4] 2023-10-06T03:29:08.298 INFO:teuthology.orchestra.run.smithi110.stdout: key = AQD3fh9lkOzRFBAAy8fDHNpPO1M6ebXU8QDvyw== 2023-10-06T03:29:08.298 INFO:teuthology.orchestra.run.smithi110.stdout:[osd.5] 2023-10-06T03:29:08.298 INFO:teuthology.orchestra.run.smithi110.stdout: key = AQD6fh9lj46oJhAAxiJqS3dqdeqaBS+m8omEKQ== 2023-10-06T03:29:08.298 INFO:teuthology.orchestra.run.smithi110.stdout:[osd.6] 2023-10-06T03:29:08.298 INFO:teuthology.orchestra.run.smithi110.stdout: key = AQD9fh9lxYvJKxAAwGvw/+afqFHVCYdJ+Dr0kw== 2023-10-06T03:29:08.299 INFO:teuthology.orchestra.run.smithi110.stdout:[osd.7] 2023-10-06T03:29:08.299 INFO:teuthology.orchestra.run.smithi110.stdout: key = AQAAfx9lY3tFLhAAvVSBIrNSWYBJZflMyT8eLw== 2023-10-06T03:29:08.299 INFO:teuthology.orchestra.run.smithi110.stdout:[client.0] 2023-10-06T03:29:08.299 INFO:teuthology.orchestra.run.smithi110.stdout: key = AQDnfh9lfTlDMhAAJ5dLa302KWyGhkB1sSMs8A== 2023-10-06T03:29:08.299 INFO:teuthology.orchestra.run.smithi110.stdout:[client.2] 2023-10-06T03:29:08.299 INFO:teuthology.orchestra.run.smithi110.stdout: key = AQDnfh9l42YJNRAAERQlQnUirB+3D3cl8QkkJg== 2023-10-06T03:29:08.299 INFO:teuthology.orchestra.run.smithi110.stdout:[client.1] 2023-10-06T03:29:08.299 INFO:teuthology.orchestra.run.smithi110.stdout: key = AQDnfh9lurKGNxAALcTu8iRFmiO7SiFKaMHAOw== 2023-10-06T03:29:08.299 INFO:teuthology.orchestra.run.smithi110.stdout:[client.3] 2023-10-06T03:29:08.299 INFO:teuthology.orchestra.run.smithi110.stdout: key = AQDnfh9lNbT3ORAAr/6KGmfvwFrZzYzRjnuLuw== 2023-10-06T03:29:08.309 INFO:teuthology.orchestra.run.smithi192.stdout:[mgr.y] 2023-10-06T03:29:08.309 INFO:teuthology.orchestra.run.smithi192.stdout: key = AQDnfh9lH5IzHBAAM72W1T6GBAluxB2R2Gcqtg== 2023-10-06T03:29:08.309 INFO:teuthology.orchestra.run.smithi192.stdout:[mds.a] 2023-10-06T03:29:08.310 INFO:teuthology.orchestra.run.smithi192.stdout: key = AQDnfh9l2k3cIRAAP843sFwH4LEZoCwTml4blQ== 2023-10-06T03:29:08.310 INFO:teuthology.orchestra.run.smithi192.stdout:[mds.c] 2023-10-06T03:29:08.310 INFO:teuthology.orchestra.run.smithi192.stdout: key = AQDnfh9lzTNRJRAAeLHxIFxyWzK0ECu9UEhjSA== 2023-10-06T03:29:08.310 INFO:teuthology.orchestra.run.smithi192.stdout:[osd.0] 2023-10-06T03:29:08.310 INFO:teuthology.orchestra.run.smithi192.stdout: key = AQDofh9ldKF3HhAAYvogispXU05WmG+yC/SnaA== 2023-10-06T03:29:08.310 INFO:teuthology.orchestra.run.smithi192.stdout:[osd.1] 2023-10-06T03:29:08.310 INFO:teuthology.orchestra.run.smithi192.stdout: key = AQDrfh9lrZcsIBAA6tz8bfWAcz7Oti3fQgKVbQ== 2023-10-06T03:29:08.310 INFO:teuthology.orchestra.run.smithi192.stdout:[osd.2] 2023-10-06T03:29:08.310 INFO:teuthology.orchestra.run.smithi192.stdout: key = AQDufh9lC27aJBAAjATAn2aGrDjlOuVJx/iEpw== 2023-10-06T03:29:08.310 INFO:teuthology.orchestra.run.smithi192.stdout:[osd.3] 2023-10-06T03:29:08.310 INFO:teuthology.orchestra.run.smithi192.stdout: key = AQDxfh9lZb2JJhAAfBL3LEcvAP+ZzWh3sZzw+w== 2023-10-06T03:29:08.311 INFO:teuthology.orchestra.run.smithi192.stdout:[mgr.x] 2023-10-06T03:29:08.311 INFO:teuthology.orchestra.run.smithi192.stdout: key = AQDnfh9lleAUHxAAPxyJMFMFjicyyb9MOxspEw== 2023-10-06T03:29:08.311 INFO:teuthology.orchestra.run.smithi192.stdout:[mds.b] 2023-10-06T03:29:08.311 INFO:teuthology.orchestra.run.smithi192.stdout: key = AQDnfh9lq/rVKxAAKyRhV1HafDy0SM+kEboXOQ== 2023-10-06T03:29:08.311 INFO:teuthology.orchestra.run.smithi192.stdout:[mds.d] 2023-10-06T03:29:08.311 INFO:teuthology.orchestra.run.smithi192.stdout: key = AQDnfh9lAHsbLxAA3HjlMJWG9MusdTNd0t/OBw== 2023-10-06T03:29:08.311 INFO:teuthology.orchestra.run.smithi192.stdout:[osd.4] 2023-10-06T03:29:08.311 INFO:teuthology.orchestra.run.smithi192.stdout: key = AQD3fh9lkOzRFBAAy8fDHNpPO1M6ebXU8QDvyw== 2023-10-06T03:29:08.311 INFO:teuthology.orchestra.run.smithi192.stdout:[osd.5] 2023-10-06T03:29:08.311 INFO:teuthology.orchestra.run.smithi192.stdout: key = AQD6fh9lj46oJhAAxiJqS3dqdeqaBS+m8omEKQ== 2023-10-06T03:29:08.311 INFO:teuthology.orchestra.run.smithi192.stdout:[osd.6] 2023-10-06T03:29:08.311 INFO:teuthology.orchestra.run.smithi192.stdout: key = AQD9fh9lxYvJKxAAwGvw/+afqFHVCYdJ+Dr0kw== 2023-10-06T03:29:08.312 INFO:teuthology.orchestra.run.smithi192.stdout:[osd.7] 2023-10-06T03:29:08.312 INFO:teuthology.orchestra.run.smithi192.stdout: key = AQAAfx9lY3tFLhAAvVSBIrNSWYBJZflMyT8eLw== 2023-10-06T03:29:08.312 INFO:teuthology.orchestra.run.smithi192.stdout:[client.0] 2023-10-06T03:29:08.312 INFO:teuthology.orchestra.run.smithi192.stdout: key = AQDnfh9lfTlDMhAAJ5dLa302KWyGhkB1sSMs8A== 2023-10-06T03:29:08.312 INFO:teuthology.orchestra.run.smithi192.stdout:[client.2] 2023-10-06T03:29:08.312 INFO:teuthology.orchestra.run.smithi192.stdout: key = AQDnfh9l42YJNRAAERQlQnUirB+3D3cl8QkkJg== 2023-10-06T03:29:08.312 INFO:teuthology.orchestra.run.smithi192.stdout:[client.1] 2023-10-06T03:29:08.312 INFO:teuthology.orchestra.run.smithi192.stdout: key = AQDnfh9lurKGNxAALcTu8iRFmiO7SiFKaMHAOw== 2023-10-06T03:29:08.312 INFO:teuthology.orchestra.run.smithi192.stdout:[client.3] 2023-10-06T03:29:08.312 INFO:teuthology.orchestra.run.smithi192.stdout: key = AQDnfh9lNbT3ORAAr/6KGmfvwFrZzYzRjnuLuw== 2023-10-06T03:29:08.313 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-10-06T03:29:08.344 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-10-06T03:29:08.389 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-10-06T03:29:08.392 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-10-06T03:29:08.437 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-10-06T03:29:08.440 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-10-06T03:29:08.528 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-06T03:29:08.531 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-06T03:29:08.575 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-06T03:29:08.578 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-06T03:29:08.665 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-06T03:29:08.670 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-06T03:29:08.713 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-06T03:29:08.716 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-06T03:29:08.798 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-10-06T03:29:08.801 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-10-06T03:29:08.845 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-10-06T03:29:08.848 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-10-06T03:29:08.893 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-10-06T03:29:08.937 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-10-06T03:29:08.984 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-06T03:29:08.986 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-06T03:29:09.027 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-06T03:29:09.029 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-06T03:29:09.109 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-06T03:29:09.112 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-06T03:29:09.156 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-06T03:29:09.159 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-06T03:29:09.202 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-10-06T03:29:09.204 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-10-06T03:29:09.296 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-10-06T03:29:09.299 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-10-06T03:29:09.347 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-10-06T03:29:09.350 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-10-06T03:29:09.393 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-10-06T03:29:09.396 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-10-06T03:29:09.490 INFO:tasks.ceph:Running mkfs on mon nodes... 2023-10-06T03:29:09.491 DEBUG:teuthology.orchestra.run.smithi110:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2023-10-06T03:29:09.507 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-10-06T03:29:09.614 DEBUG:teuthology.orchestra.run.smithi110:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2023-10-06T03:29:09.863 DEBUG:teuthology.orchestra.run.smithi192:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2023-10-06T03:29:09.882 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-10-06T03:29:09.990 DEBUG:teuthology.orchestra.run.smithi192:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2023-10-06T03:29:10.010 DEBUG:teuthology.orchestra.run.smithi192:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2023-10-06T03:29:10.072 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-10-06T03:29:10.128 DEBUG:teuthology.orchestra.run.smithi192:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2023-10-06T03:29:10.144 DEBUG:teuthology.orchestra.run.smithi110:> rm -- /home/ubuntu/cephtest/ceph.monmap 2023-10-06T03:29:10.147 DEBUG:teuthology.orchestra.run.smithi192:> rm -- /home/ubuntu/cephtest/ceph.monmap 2023-10-06T03:29:10.193 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2023-10-06T03:29:10.193 INFO:tasks.ceph.mon.a:Restarting daemon 2023-10-06T03:29:10.193 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2023-10-06T03:29:10.198 INFO:tasks.ceph.mon.a:Started 2023-10-06T03:29:10.199 INFO:tasks.ceph.mon.b:Restarting daemon 2023-10-06T03:29:10.199 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2023-10-06T03:29:10.203 INFO:tasks.ceph.mon.b:Started 2023-10-06T03:29:10.203 INFO:tasks.ceph.mon.c:Restarting daemon 2023-10-06T03:29:10.203 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2023-10-06T03:29:10.205 INFO:tasks.ceph.mon.c:Started 2023-10-06T03:29:10.206 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2023-10-06T03:29:10.206 INFO:tasks.ceph.mgr.y:Restarting daemon 2023-10-06T03:29:10.206 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2023-10-06T03:29:10.210 INFO:tasks.ceph.mgr.y:Started 2023-10-06T03:29:10.210 INFO:tasks.ceph.mgr.x:Restarting daemon 2023-10-06T03:29:10.210 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2023-10-06T03:29:10.216 INFO:tasks.ceph.mgr.x:Started 2023-10-06T03:29:10.216 INFO:tasks.ceph:Setting crush tunables to default 2023-10-06T03:29:10.216 DEBUG:teuthology.orchestra.run.smithi110:> sudo ceph --cluster ceph osd crush tunables default 2023-10-06T03:29:10.318 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:10.310+0000 7f263a5a9700 1 Processor -- start 2023-10-06T03:29:10.319 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:10.310+0000 7f263a5a9700 1 -- start start 2023-10-06T03:29:10.321 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:10.314+0000 7f263a5a9700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f2634069d70 0x7f26340af160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:10.321 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:10.314+0000 7f263a5a9700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f26341554b0 0x7f26340694e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:10.321 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:10.314+0000 7f2638da6700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f2634069d70 0x7f26340af160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:10.321 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:10.314+0000 7f26395a7700 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f26341550e0 0x7f263406caa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:43760/0 (socket says 172.21.15.110:43760) 2023-10-06T03:29:10.321 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:10.314+0000 7f26395a7700 1 -- 172.21.15.110:0/3322207125 learned_addr learned my addr 172.21.15.110:0/3322207125 (peer_addr_for_me v1:172.21.15.110:0/0) 2023-10-06T03:29:10.321 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:10.314+0000 7f263a5a9700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26340aecd0 con 0x7f26341550e0 2023-10-06T03:29:10.321 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:10.314+0000 7f263a5a9700 1 -- 172.21.15.110:0/3322207125 --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f263406d1b0 con 0x7f2634069d70 2023-10-06T03:29:10.321 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:10.314+0000 7f263a5a9700 1 -- 172.21.15.110:0/3322207125 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26340ae980 con 0x7f26341554b0 2023-10-06T03:29:10.321 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:10.314+0000 7f2639da8700 1 --2- 172.21.15.110:0/3322207125 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f26341554b0 0x7f26340694e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:10.322 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:10.314+0000 7f2638da6700 1 -- 172.21.15.110:0/3322207125 >> v2:172.21.15.110:3300/0 conn(0x7f2634069d70 msgr2=0x7f26340af160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-10-06T03:29:10.322 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:10.314+0000 7f2638da6700 1 -- 172.21.15.110:0/3322207125 >> v2:172.21.15.110:3300/0 conn(0x7f2634069d70 msgr2=0x7f26340af160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-06T03:29:10.322 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:10.314+0000 7f2638da6700 1 --2- 172.21.15.110:0/3322207125 >> v2:172.21.15.110:3300/0 conn(0x7f2634069d70 0x7f26340af160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-10-06T03:29:10.322 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:10.314+0000 7f2638da6700 1 --2- 172.21.15.110:0/3322207125 >> v2:172.21.15.110:3300/0 conn(0x7f2634069d70 0x7f26340af160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-06T03:29:10.322 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:10.314+0000 7f2639da8700 1 -- 172.21.15.110:0/3322207125 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f26341554b0 msgr2=0x7f26340694e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2023-10-06T03:29:10.322 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:10.314+0000 7f2639da8700 1 -- 172.21.15.110:0/3322207125 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f26341554b0 msgr2=0x7f26340694e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-06T03:29:10.322 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:10.314+0000 7f2639da8700 1 --2- 172.21.15.110:0/3322207125 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f26341554b0 0x7f26340694e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-10-06T03:29:10.322 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:10.314+0000 7f2639da8700 1 --2- 172.21.15.110:0/3322207125 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f26341554b0 0x7f26340694e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-06T03:29:10.524 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:10.518+0000 7f2638da6700 1 --2- 172.21.15.110:0/3322207125 >> v2:172.21.15.110:3300/0 conn(0x7f2634069d70 0x7f26340af160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:10.524 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:10.518+0000 7f2639da8700 1 --2- 172.21.15.110:0/3322207125 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f26341554b0 0x7f26340694e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:10.524 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:10.518+0000 7f2638da6700 1 -- 172.21.15.110:0/3322207125 >> v2:172.21.15.110:3300/0 conn(0x7f2634069d70 msgr2=0x7f26340af160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-10-06T03:29:10.524 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:10.518+0000 7f2638da6700 1 -- 172.21.15.110:0/3322207125 >> v2:172.21.15.110:3300/0 conn(0x7f2634069d70 msgr2=0x7f26340af160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-06T03:29:10.524 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:10.518+0000 7f2638da6700 1 --2- 172.21.15.110:0/3322207125 >> v2:172.21.15.110:3300/0 conn(0x7f2634069d70 0x7f26340af160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-10-06T03:29:10.524 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:10.518+0000 7f2638da6700 1 --2- 172.21.15.110:0/3322207125 >> v2:172.21.15.110:3300/0 conn(0x7f2634069d70 0x7f26340af160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-06T03:29:10.524 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:10.518+0000 7f2639da8700 1 -- 172.21.15.110:0/3322207125 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f26341554b0 msgr2=0x7f26340694e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2023-10-06T03:29:10.524 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:10.518+0000 7f2639da8700 1 -- 172.21.15.110:0/3322207125 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f26341554b0 msgr2=0x7f26340694e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-06T03:29:10.525 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:10.518+0000 7f2639da8700 1 --2- 172.21.15.110:0/3322207125 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f26341554b0 0x7f26340694e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-10-06T03:29:10.525 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:10.518+0000 7f2639da8700 1 --2- 172.21.15.110:0/3322207125 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f26341554b0 0x7f26340694e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-06T03:29:10.548 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:10.542+0000 7f2633fff700 1 -- 172.21.15.110:0/3322207125 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 143470651 0 0) 0x7f26340aecd0 con 0x7f26341550e0 2023-10-06T03:29:10.548 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:10.542+0000 7f2633fff700 1 -- 172.21.15.110:0/3322207125 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f261c003570 con 0x7f26341550e0 2023-10-06T03:29:10.549 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:10.542+0000 7f2633fff700 1 -- 172.21.15.110:0/3322207125 <== mon.0 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3569150112 0 0) 0x7f261c003570 con 0x7f26341550e0 2023-10-06T03:29:10.549 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:10.542+0000 7f2633fff700 1 -- 172.21.15.110:0/3322207125 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f26341554b0 msgr2=0x7f26340694e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-06T03:29:10.549 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:10.542+0000 7f2633fff700 1 --2- 172.21.15.110:0/3322207125 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f26341554b0 0x7f26340694e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:10.549 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:10.542+0000 7f2633fff700 1 -- 172.21.15.110:0/3322207125 >> v2:172.21.15.110:3300/0 conn(0x7f2634069d70 msgr2=0x7f26340af160 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-06T03:29:10.549 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:10.542+0000 7f2633fff700 1 --2- 172.21.15.110:0/3322207125 >> v2:172.21.15.110:3300/0 conn(0x7f2634069d70 0x7f26340af160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:10.549 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:10.542+0000 7f2633fff700 1 -- 172.21.15.110:0/3322207125 --> v1:172.21.15.110:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f263406a260 con 0x7f26341550e0 2023-10-06T03:29:10.549 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:10.542+0000 7f2633fff700 1 -- 172.21.15.110:0/3322207125 <== mon.0 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 118603703 0 0) 0x7f2628001470 con 0x7f26341550e0 2023-10-06T03:29:10.549 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:10.542+0000 7f2633fff700 1 -- 172.21.15.110:0/3322207125 >> v1:172.21.15.110:6789/0 conn(0x7f26341550e0 legacy=0x7f263406caa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:10.549 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:10.542+0000 7f2633fff700 1 --2- 172.21.15.110:0/3322207125 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f261c003a60 0x7f261c003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:10.549 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:10.542+0000 7f2633fff700 1 --2- 172.21.15.110:0/3322207125 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f261c004810 0x7f261c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:10.549 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:10.542+0000 7f2633fff700 1 --2- 172.21.15.110:0/3322207125 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f26341554b0 0x7f261c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:10.550 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:10.542+0000 7f2633fff700 1 -- 172.21.15.110:0/3322207125 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f263406d1b0 con 0x7f261c003a60 2023-10-06T03:29:10.550 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:10.542+0000 7f2633fff700 1 -- 172.21.15.110:0/3322207125 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26340ae980 con 0x7f261c004810 2023-10-06T03:29:10.550 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:10.542+0000 7f2633fff700 1 -- 172.21.15.110:0/3322207125 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f261c00bb90 con 0x7f26341554b0 2023-10-06T03:29:10.551 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:10.542+0000 7f26395a7700 1 --2- 172.21.15.110:0/3322207125 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f261c003a60 0x7f261c003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:10.551 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:10.542+0000 7f2639da8700 1 --2- 172.21.15.110:0/3322207125 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f26341554b0 0x7f261c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:10.551 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:10.542+0000 7f2638da6700 1 --2- 172.21.15.110:0/3322207125 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f261c004810 0x7f261c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:10.551 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:10.542+0000 7f26395a7700 1 -- 172.21.15.110:0/3322207125 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f26341554b0 msgr2=0x7f261c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:29:10.551 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:10.542+0000 7f26395a7700 1 --2- 172.21.15.110:0/3322207125 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f26341554b0 0x7f261c0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:10.551 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:10.542+0000 7f26395a7700 1 -- 172.21.15.110:0/3322207125 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f261c004810 msgr2=0x7f261c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:29:10.551 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:10.542+0000 7f26395a7700 1 --2- 172.21.15.110:0/3322207125 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f261c004810 0x7f261c004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:10.551 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:10.542+0000 7f26395a7700 1 -- 172.21.15.110:0/3322207125 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f261c003570 con 0x7f261c003a60 2023-10-06T03:29:10.551 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:10.542+0000 7f26395a7700 1 --2- 172.21.15.110:0/3322207125 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f261c003a60 0x7f261c003e70 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f262800e6d0 tx=0x7f26340aec40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b2024f1ea343a15 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:29:10.552 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:10.542+0000 7f2633fff700 1 -- 172.21.15.110:0/3322207125 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2628013070 con 0x7f261c003a60 2023-10-06T03:29:10.552 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:10.542+0000 7f2633fff700 1 -- 172.21.15.110:0/3322207125 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f262800e110 con 0x7f261c003a60 2023-10-06T03:29:10.552 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:10.546+0000 7f263a5a9700 1 -- 172.21.15.110:0/3322207125 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f261c003a60 msgr2=0x7f261c003e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:10.553 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:10.546+0000 7f263a5a9700 1 --2- 172.21.15.110:0/3322207125 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f261c003a60 0x7f261c003e70 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f262800e6d0 tx=0x7f26340aec40 comp rx=0 tx=0).stop 2023-10-06T03:29:10.553 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:10.546+0000 7f2639da8700 1 -- 172.21.15.110:0/3322207125 reap_dead start 2023-10-06T03:29:10.553 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:10.546+0000 7f263a5a9700 1 -- 172.21.15.110:0/3322207125 shutdown_connections 2023-10-06T03:29:10.553 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:10.546+0000 7f263a5a9700 1 -- 172.21.15.110:0/3322207125 >> 172.21.15.110:0/3322207125 conn(0x7f26340cbac0 msgr2=0x7f26340cbec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:29:10.553 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:10.546+0000 7f263a5a9700 1 -- 172.21.15.110:0/3322207125 shutdown_connections 2023-10-06T03:29:10.554 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:10.546+0000 7f263a5a9700 1 -- 172.21.15.110:0/3322207125 wait complete. 2023-10-06T03:29:10.555 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:10.550+0000 7f263a5a9700 1 Processor -- start 2023-10-06T03:29:10.555 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:10.550+0000 7f263a5a9700 1 -- start start 2023-10-06T03:29:10.558 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:10.550+0000 7f263a5a9700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f261c003a60 0x7f2634156910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:10.558 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:10.550+0000 7f263a5a9700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f26341688a0 0x7f2634140820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:10.558 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:10.550+0000 7f263a5a9700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f26340c1be0 0x7f2634167cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:10.558 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:10.550+0000 7f263a5a9700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26340c61e0 con 0x7f26340c1be0 2023-10-06T03:29:10.558 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:10.550+0000 7f263a5a9700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26340c2810 con 0x7f26341688a0 2023-10-06T03:29:10.558 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:10.550+0000 7f263a5a9700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26340c2530 con 0x7f261c003a60 2023-10-06T03:29:10.559 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:10.550+0000 7f2638da6700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f26341688a0 0x7f2634140820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:10.559 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:10.550+0000 7f26395a7700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f261c003a60 0x7f2634156910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:10.559 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:10.550+0000 7f2638da6700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f26341688a0 0x7f2634140820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.110:49370/0 (socket says 172.21.15.110:49370) 2023-10-06T03:29:10.559 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:10.550+0000 7f2638da6700 1 -- 172.21.15.110:0/3281799130 learned_addr learned my addr 172.21.15.110:0/3281799130 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:29:10.559 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:10.550+0000 7f2639da8700 1 --2- 172.21.15.110:0/3281799130 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f26340c1be0 0x7f2634167cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:10.559 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:10.550+0000 7f2639da8700 1 -- 172.21.15.110:0/3281799130 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f261c003a60 msgr2=0x7f2634156910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:10.559 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:10.550+0000 7f2639da8700 1 --2- 172.21.15.110:0/3281799130 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f261c003a60 0x7f2634156910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:10.559 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:10.550+0000 7f2639da8700 1 -- 172.21.15.110:0/3281799130 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f26341688a0 msgr2=0x7f2634140820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:10.559 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:10.550+0000 7f2639da8700 1 --2- 172.21.15.110:0/3281799130 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f26341688a0 0x7f2634140820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:10.559 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:10.550+0000 7f2639da8700 1 -- 172.21.15.110:0/3281799130 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26340c5b90 con 0x7f26340c1be0 2023-10-06T03:29:10.560 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:10.550+0000 7f2639da8700 1 --2- 172.21.15.110:0/3281799130 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f26340c1be0 0x7f2634167cf0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f2634158dd0 tx=0x7f262c008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:29:10.561 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:10.550+0000 7f26317fa700 1 -- 172.21.15.110:0/3281799130 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f262c010d90 con 0x7f26340c1be0 2023-10-06T03:29:10.561 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:10.550+0000 7f26317fa700 1 -- 172.21.15.110:0/3281799130 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f262c01b440 con 0x7f26340c1be0 2023-10-06T03:29:10.561 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:10.550+0000 7f263a5a9700 1 -- 172.21.15.110:0/3281799130 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26340c5e70 con 0x7f26340c1be0 2023-10-06T03:29:10.561 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:10.550+0000 7f26317fa700 1 -- 172.21.15.110:0/3281799130 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f262c01b800 con 0x7f26340c1be0 2023-10-06T03:29:10.561 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:10.554+0000 7f263a5a9700 1 -- 172.21.15.110:0/3281799130 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2634140d60 con 0x7f26340c1be0 2023-10-06T03:29:10.561 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:10.554+0000 7f26317fa700 1 -- 172.21.15.110:0/3281799130 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7f262c01bc30 con 0x7f26340c1be0 2023-10-06T03:29:10.561 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:10.554+0000 7f26317fa700 1 -- 172.21.15.110:0/3281799130 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f262c006370 con 0x7f26340c1be0 2023-10-06T03:29:10.561 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:10.554+0000 7f263a5a9700 1 -- 172.21.15.110:0/3281799130 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f26340c59f0 con 0x7f26340c1be0 2023-10-06T03:29:10.563 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:10.558+0000 7f26317fa700 1 -- 172.21.15.110:0/3281799130 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7f26340c59f0 con 0x7f26340c1be0 2023-10-06T03:29:10.632 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:10.626+0000 7f263a5a9700 1 -- 172.21.15.110:0/3281799130 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7f26340cb920 con 0x7f26340c1be0 2023-10-06T03:29:10.636 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:10.630+0000 7f26317fa700 1 -- 172.21.15.110:0/3281799130 <== mon.0 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) v1 ==== 124+0+0 (secure 0 0 0) 0x7f26340cb920 con 0x7f26340c1be0 2023-10-06T03:29:10.638 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:10.630+0000 7f263a5a9700 1 -- 172.21.15.110:0/3281799130 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f26340c1be0 msgr2=0x7f2634167cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:10.638 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:10.630+0000 7f263a5a9700 1 --2- 172.21.15.110:0/3281799130 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f26340c1be0 0x7f2634167cf0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f2634158dd0 tx=0x7f262c008040 comp rx=0 tx=0).stop 2023-10-06T03:29:10.638 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:10.630+0000 7f263a5a9700 1 -- 172.21.15.110:0/3281799130 shutdown_connections 2023-10-06T03:29:10.638 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:10.630+0000 7f263a5a9700 1 --2- 172.21.15.110:0/3281799130 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f26340c1be0 0x7f2634167cf0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:10.638 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:10.630+0000 7f263a5a9700 1 --2- 172.21.15.110:0/3281799130 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f26341688a0 0x7f2634140820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:10.638 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:10.630+0000 7f263a5a9700 1 --2- 172.21.15.110:0/3281799130 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f261c003a60 0x7f2634156910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:10.638 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:10.630+0000 7f263a5a9700 1 -- 172.21.15.110:0/3281799130 >> 172.21.15.110:0/3281799130 conn(0x7f26340cbac0 msgr2=0x7f26340c87d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:29:10.638 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:10.630+0000 7f263a5a9700 1 -- 172.21.15.110:0/3281799130 shutdown_connections 2023-10-06T03:29:10.638 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:10.630+0000 7f263a5a9700 1 -- 172.21.15.110:0/3281799130 wait complete. 2023-10-06T03:29:10.639 INFO:teuthology.orchestra.run.smithi110.stderr:adjusted tunables profile to default 2023-10-06T03:29:10.650 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2023-10-06T03:29:10.651 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2023-10-06T03:29:10.651 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2023-10-06T03:29:10.666 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2023-10-06T03:29:10.666 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2023-10-06T03:29:10.676 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2023-10-06T03:29:10.676 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2023-10-06T03:29:10.691 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2023-10-06T03:29:10.691 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2023-10-06T03:29:10.743 INFO:tasks.ceph.mgr.x.smithi192.stderr:2023-10-06T03:29:10.738+0000 7f0930332000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-06T03:29:10.750 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2023-10-06T03:29:10.751 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2023-10-06T03:29:10.761 INFO:tasks.ceph.mgr.y.smithi110.stderr:2023-10-06T03:29:10.754+0000 7f5523871000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-06T03:29:10.766 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2023-10-06T03:29:10.766 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2023-10-06T03:29:10.824 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2023-10-06T03:29:10.824 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2023-10-06T03:29:10.837 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2023-10-06T03:29:10.837 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2023-10-06T03:29:10.848 DEBUG:teuthology.orchestra.run.smithi192:> sudo ceph --cluster ceph osd new 5866eb6c-2adc-46e4-be7a-e5bcda0f4065 0 2023-10-06T03:29:10.948 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:10.942+0000 7f5c33fff700 1 Processor -- start 2023-10-06T03:29:10.948 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:10.942+0000 7f5c33fff700 1 -- start start 2023-10-06T03:29:10.950 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:10.942+0000 7f5c33fff700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f5c340b1d40 0x7f5c340b2110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:10.950 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:10.942+0000 7f5c33fff700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5c340afc70 0x7f5c340b00c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:10.950 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:10.942+0000 7f5c33fff700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c340caa70 con 0x7f5c341558d0 2023-10-06T03:29:10.950 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:10.942+0000 7f5c33fff700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5c340b0600 con 0x7f5c340b1d40 2023-10-06T03:29:10.950 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:10.942+0000 7f5c33fff700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c340ca740 con 0x7f5c340afc70 2023-10-06T03:29:10.951 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:10.942+0000 7f5c32ffd700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f5c340b1d40 0x7f5c340b2110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:10.951 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:10.942+0000 7f5c327fc700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5c340afc70 0x7f5c340b00c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:10.951 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:10.942+0000 7f5c32ffd700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f5c340b1d40 0x7f5c340b2110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.192:52830/0 (socket says 172.21.15.192:52830) 2023-10-06T03:29:10.951 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:10.942+0000 7f5c32ffd700 1 -- 172.21.15.192:0/871420421 learned_addr learned my addr 172.21.15.192:0/871420421 (peer_addr_for_me v2:172.21.15.192:0/0) 2023-10-06T03:29:10.951 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:10.942+0000 7f5c32ffd700 1 -- 172.21.15.192:0/871420421 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5c340afc70 msgr2=0x7f5c340b00c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:29:10.951 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:10.942+0000 7f5c32ffd700 1 --2- 172.21.15.192:0/871420421 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5c340afc70 0x7f5c340b00c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:10.951 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:10.942+0000 7f5c32ffd700 1 -- 172.21.15.192:0/871420421 >> v1:172.21.15.110:6789/0 conn(0x7f5c341558d0 legacy=0x7f5c34155d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:29:10.951 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:10.942+0000 7f5c32ffd700 1 -- 172.21.15.192:0/871420421 --> v2:172.21.15.192:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c34159cc0 con 0x7f5c340b1d40 2023-10-06T03:29:10.951 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:10.942+0000 7f5c327fc700 1 --2- 172.21.15.192:0/871420421 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5c340afc70 0x7f5c340b00c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-06T03:29:10.951 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:10.942+0000 7f5c32ffd700 1 --2- 172.21.15.192:0/871420421 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5c340b1d40 0x7f5c340b2110 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f5c240099b0 tx=0x7f5c24009250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8be349d03e7b2128 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:29:10.952 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:10.946+0000 7f5c31ffb700 1 -- 172.21.15.192:0/871420421 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c24009ea0 con 0x7f5c340b1d40 2023-10-06T03:29:10.952 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:10.946+0000 7f5c31ffb700 1 -- 172.21.15.192:0/871420421 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c2400b820 con 0x7f5c340b1d40 2023-10-06T03:29:10.952 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:10.946+0000 7f5c31ffb700 1 -- 172.21.15.192:0/871420421 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c2400bbe0 con 0x7f5c340b1d40 2023-10-06T03:29:10.953 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:10.946+0000 7f5c33fff700 1 -- 172.21.15.192:0/871420421 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5c340b1d40 msgr2=0x7f5c340b2110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:10.953 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:10.946+0000 7f5c33fff700 1 --2- 172.21.15.192:0/871420421 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5c340b1d40 0x7f5c340b2110 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f5c240099b0 tx=0x7f5c24009250 comp rx=0 tx=0).stop 2023-10-06T03:29:10.953 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:10.946+0000 7f5c33fff700 1 -- 172.21.15.192:0/871420421 shutdown_connections 2023-10-06T03:29:10.953 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:10.946+0000 7f5c33fff700 1 --2- 172.21.15.192:0/871420421 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5c340afc70 0x7f5c340b00c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:10.953 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:10.946+0000 7f5c33fff700 1 --2- 172.21.15.192:0/871420421 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5c340b1d40 0x7f5c340b2110 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:10.953 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:10.946+0000 7f5c33fff700 1 -- 172.21.15.192:0/871420421 >> 172.21.15.192:0/871420421 conn(0x7f5c34068470 msgr2=0x7f5c34068870 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:29:10.953 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:10.946+0000 7f5c33fff700 1 -- 172.21.15.192:0/871420421 shutdown_connections 2023-10-06T03:29:10.953 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:10.946+0000 7f5c33fff700 1 -- 172.21.15.192:0/871420421 wait complete. 2023-10-06T03:29:10.953 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:10.946+0000 7f5c33fff700 1 Processor -- start 2023-10-06T03:29:10.954 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:10.946+0000 7f5c33fff700 1 -- start start 2023-10-06T03:29:10.956 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:10.950+0000 7f5c33fff700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5c340afc70 0x7f5c3414ed40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:10.957 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:10.950+0000 7f5c33fff700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5c340b1d40 0x7f5c3414ba90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:10.957 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:10.950+0000 7f5c33fff700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5c341558d0 0x7f5c341499d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:10.957 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:10.950+0000 7f5c33fff700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c3414bfd0 con 0x7f5c340afc70 2023-10-06T03:29:10.957 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:10.950+0000 7f5c33fff700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c3414ce10 con 0x7f5c341558d0 2023-10-06T03:29:10.957 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:10.950+0000 7f5c33fff700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c34149f10 con 0x7f5c340b1d40 2023-10-06T03:29:10.957 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:10.950+0000 7f5c337fe700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5c341558d0 0x7f5c341499d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:10.957 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:10.950+0000 7f5c327fc700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5c340b1d40 0x7f5c3414ba90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:10.957 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:10.950+0000 7f5c337fe700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5c341558d0 0x7f5c341499d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.192:52846/0 (socket says 172.21.15.192:52846) 2023-10-06T03:29:10.957 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:10.950+0000 7f5c337fe700 1 -- 172.21.15.192:0/1428888284 learned_addr learned my addr 172.21.15.192:0/1428888284 (peer_addr_for_me v2:172.21.15.192:0/0) 2023-10-06T03:29:10.957 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:10.950+0000 7f5c32ffd700 1 --2- 172.21.15.192:0/1428888284 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5c340afc70 0x7f5c3414ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:10.957 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:10.950+0000 7f5c337fe700 1 -- 172.21.15.192:0/1428888284 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5c340b1d40 msgr2=0x7f5c3414ba90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:10.958 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:10.950+0000 7f5c337fe700 1 --2- 172.21.15.192:0/1428888284 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5c340b1d40 0x7f5c3414ba90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:10.958 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:10.950+0000 7f5c337fe700 1 -- 172.21.15.192:0/1428888284 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5c340afc70 msgr2=0x7f5c3414ed40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:10.958 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:10.950+0000 7f5c337fe700 1 --2- 172.21.15.192:0/1428888284 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5c340afc70 0x7f5c3414ed40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:10.958 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:10.950+0000 7f5c337fe700 1 -- 172.21.15.192:0/1428888284 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c28009420 con 0x7f5c341558d0 2023-10-06T03:29:10.958 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:10.950+0000 7f5c32ffd700 1 --2- 172.21.15.192:0/1428888284 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5c340afc70 0x7f5c3414ed40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-06T03:29:10.958 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:10.950+0000 7f5c337fe700 1 --2- 172.21.15.192:0/1428888284 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5c341558d0 0x7f5c341499d0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f5c2c00bb60 tx=0x7f5c2c0097f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:29:10.958 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:10.950+0000 7f5c1affd700 1 -- 172.21.15.192:0/1428888284 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c2c00c8c0 con 0x7f5c341558d0 2023-10-06T03:29:10.958 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:10.950+0000 7f5c1affd700 1 -- 172.21.15.192:0/1428888284 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c2c00ca60 con 0x7f5c341558d0 2023-10-06T03:29:10.958 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:10.950+0000 7f5c33fff700 1 -- 172.21.15.192:0/1428888284 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c240094f0 con 0x7f5c341558d0 2023-10-06T03:29:10.958 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:10.950+0000 7f5c1affd700 1 -- 172.21.15.192:0/1428888284 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c2c00ce20 con 0x7f5c341558d0 2023-10-06T03:29:10.958 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:10.950+0000 7f5c33fff700 1 -- 172.21.15.192:0/1428888284 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c3414c4e0 con 0x7f5c341558d0 2023-10-06T03:29:10.959 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:10.950+0000 7f5c1affd700 1 -- 172.21.15.192:0/1428888284 <== mon.1 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7f5c2c014620 con 0x7f5c341558d0 2023-10-06T03:29:10.959 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:10.950+0000 7f5c1affd700 1 -- 172.21.15.192:0/1428888284 <== mon.1 v2:172.21.15.192:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 721+0+0 (secure 0 0 0) 0x7f5c2c01d4c0 con 0x7f5c341558d0 2023-10-06T03:29:10.959 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:10.950+0000 7f5c33fff700 1 -- 172.21.15.192:0/1428888284 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c34070ff0 con 0x7f5c341558d0 2023-10-06T03:29:10.960 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:10.954+0000 7f5c1affd700 1 -- 172.21.15.192:0/1428888284 <== mon.1 v2:172.21.15.192:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7f5c2c022060 con 0x7f5c341558d0 2023-10-06T03:29:11.036 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:11.030+0000 7f5c33fff700 1 -- 172.21.15.192:0/1428888284 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "osd new", "uuid": "5866eb6c-2adc-46e4-be7a-e5bcda0f4065", "id": 0} v 0) v1 -- 0x7f5c3414cf50 con 0x7f5c341558d0 2023-10-06T03:29:11.046 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:11.038+0000 7f5c1affd700 1 -- 172.21.15.192:0/1428888284 <== mon.1 v2:172.21.15.192:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "5866eb6c-2adc-46e4-be7a-e5bcda0f4065", "id": 0}]=0 v3) v1 ==== 112+0+2 (secure 0 0 0) 0x7f5c2c014a60 con 0x7f5c341558d0 2023-10-06T03:29:11.046 INFO:teuthology.orchestra.run.smithi192.stdout:0 2023-10-06T03:29:11.047 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:11.042+0000 7f5c33fff700 1 -- 172.21.15.192:0/1428888284 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5c341558d0 msgr2=0x7f5c341499d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:11.048 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:11.042+0000 7f5c33fff700 1 --2- 172.21.15.192:0/1428888284 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5c341558d0 0x7f5c341499d0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f5c2c00bb60 tx=0x7f5c2c0097f0 comp rx=0 tx=0).stop 2023-10-06T03:29:11.048 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:11.042+0000 7f5c33fff700 1 -- 172.21.15.192:0/1428888284 shutdown_connections 2023-10-06T03:29:11.048 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:11.042+0000 7f5c33fff700 1 --2- 172.21.15.192:0/1428888284 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5c341558d0 0x7f5c341499d0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:11.048 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:11.042+0000 7f5c33fff700 1 --2- 172.21.15.192:0/1428888284 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5c340b1d40 0x7f5c3414ba90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:11.048 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:11.042+0000 7f5c33fff700 1 --2- 172.21.15.192:0/1428888284 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5c340afc70 0x7f5c3414ed40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:11.048 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:11.042+0000 7f5c33fff700 1 -- 172.21.15.192:0/1428888284 >> 172.21.15.192:0/1428888284 conn(0x7f5c34068470 msgr2=0x7f5c341515e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:29:11.048 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:11.042+0000 7f5c33fff700 1 -- 172.21.15.192:0/1428888284 shutdown_connections 2023-10-06T03:29:11.048 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:11.042+0000 7f5c33fff700 1 -- 172.21.15.192:0/1428888284 wait complete. 2023-10-06T03:29:11.060 DEBUG:teuthology.orchestra.run.smithi192:> sudo ceph --cluster ceph osd new a38de118-062b-4a05-9782-2ceddf6ff4df 1 2023-10-06T03:29:11.166 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:11.158+0000 7f1d25aa9700 1 Processor -- start 2023-10-06T03:29:11.166 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:11.158+0000 7f1d25aa9700 1 -- start start 2023-10-06T03:29:11.167 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:11.162+0000 7f1d25aa9700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f1d20157010 0x7f1d2006c9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:11.168 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:11.162+0000 7f1d25aa9700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f1d201573e0 0x7f1d2015b750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:11.168 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:11.162+0000 7f1d25aa9700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d20157860 con 0x7f1d200670a0 2023-10-06T03:29:11.168 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:11.162+0000 7f1d25aa9700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1d2015bc90 con 0x7f1d20157010 2023-10-06T03:29:11.168 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:11.162+0000 7f1d25aa9700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1d20157ae0 con 0x7f1d201573e0 2023-10-06T03:29:11.168 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:11.162+0000 7f1d1effd700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f1d201573e0 0x7f1d2015b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:11.168 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:11.162+0000 7f1d1effd700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f1d201573e0 0x7f1d2015b750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.192:52854/0 (socket says 172.21.15.192:52854) 2023-10-06T03:29:11.168 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:11.162+0000 7f1d1effd700 1 -- 172.21.15.192:0/1821007655 learned_addr learned my addr 172.21.15.192:0/1821007655 (peer_addr_for_me v2:172.21.15.192:0/0) 2023-10-06T03:29:11.168 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:11.162+0000 7f1d1f7fe700 1 --2- 172.21.15.192:0/1821007655 >> v2:172.21.15.110:3300/0 conn(0x7f1d20157010 0x7f1d2006c9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:11.170 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:11.162+0000 7f1d1effd700 1 -- 172.21.15.192:0/1821007655 >> v2:172.21.15.110:3300/0 conn(0x7f1d20157010 msgr2=0x7f1d2006c9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:29:11.170 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:11.162+0000 7f1d1effd700 1 --2- 172.21.15.192:0/1821007655 >> v2:172.21.15.110:3300/0 conn(0x7f1d20157010 0x7f1d2006c9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:11.170 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:11.162+0000 7f1d1effd700 1 -- 172.21.15.192:0/1821007655 >> v1:172.21.15.110:6789/0 conn(0x7f1d200670a0 legacy=0x7f1d200acc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:11.170 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:11.162+0000 7f1d1effd700 1 -- 172.21.15.192:0/1821007655 --> v2:172.21.15.192:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d20157dc0 con 0x7f1d201573e0 2023-10-06T03:29:11.171 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:11.162+0000 7f1d1effd700 1 --2- 172.21.15.192:0/1821007655 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1d201573e0 0x7f1d2015b750 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f1d20069290 tx=0x7f1d14009250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c7a662485a669e17 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:29:11.171 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:11.162+0000 7f1d1e7fc700 1 -- 172.21.15.192:0/1821007655 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d1400b860 con 0x7f1d201573e0 2023-10-06T03:29:11.171 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:11.162+0000 7f1d1e7fc700 1 -- 172.21.15.192:0/1821007655 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d1400b9c0 con 0x7f1d201573e0 2023-10-06T03:29:11.171 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:11.162+0000 7f1d1e7fc700 1 -- 172.21.15.192:0/1821007655 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d1400bd80 con 0x7f1d201573e0 2023-10-06T03:29:11.171 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:11.162+0000 7f1d25aa9700 1 -- 172.21.15.192:0/1821007655 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1d201573e0 msgr2=0x7f1d2015b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:11.171 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:11.162+0000 7f1d25aa9700 1 --2- 172.21.15.192:0/1821007655 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1d201573e0 0x7f1d2015b750 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f1d20069290 tx=0x7f1d14009250 comp rx=0 tx=0).stop 2023-10-06T03:29:11.171 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:11.162+0000 7f1d25aa9700 1 -- 172.21.15.192:0/1821007655 shutdown_connections 2023-10-06T03:29:11.171 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:11.162+0000 7f1d25aa9700 1 --2- 172.21.15.192:0/1821007655 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1d201573e0 0x7f1d2015b750 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:11.171 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:11.162+0000 7f1d25aa9700 1 --2- 172.21.15.192:0/1821007655 >> v2:172.21.15.110:3300/0 conn(0x7f1d20157010 0x7f1d2006c9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:11.171 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:11.162+0000 7f1d25aa9700 1 -- 172.21.15.192:0/1821007655 >> 172.21.15.192:0/1821007655 conn(0x7f1d200ccd30 msgr2=0x7f1d200cd130 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:29:11.171 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:11.162+0000 7f1d25aa9700 1 -- 172.21.15.192:0/1821007655 shutdown_connections 2023-10-06T03:29:11.172 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:11.162+0000 7f1d25aa9700 1 -- 172.21.15.192:0/1821007655 wait complete. 2023-10-06T03:29:11.174 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:11.166+0000 7f1d25aa9700 1 Processor -- start 2023-10-06T03:29:11.174 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:11.166+0000 7f1d25aa9700 1 -- start start 2023-10-06T03:29:11.174 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:11.166+0000 7f1d25aa9700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1d200670a0 0x7f1d200c4750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:11.174 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:11.166+0000 7f1d25aa9700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1d20157010 0x7f1d20167da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:11.174 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:11.166+0000 7f1d25aa9700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1d201573e0 0x7f1d20153b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:11.174 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:11.166+0000 7f1d25aa9700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d200c1f90 con 0x7f1d200670a0 2023-10-06T03:29:11.174 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:11.166+0000 7f1d25aa9700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d200c4e50 con 0x7f1d20157010 2023-10-06T03:29:11.174 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:11.166+0000 7f1d25aa9700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d200c1660 con 0x7f1d201573e0 2023-10-06T03:29:11.174 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:11.166+0000 7f1d1effd700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1d20157010 0x7f1d20167da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:11.174 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:11.166+0000 7f1d1effd700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1d20157010 0x7f1d20167da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.192:52856/0 (socket says 172.21.15.192:52856) 2023-10-06T03:29:11.175 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:11.166+0000 7f1d1effd700 1 -- 172.21.15.192:0/638709744 learned_addr learned my addr 172.21.15.192:0/638709744 (peer_addr_for_me v2:172.21.15.192:0/0) 2023-10-06T03:29:11.175 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:11.166+0000 7f1d1ffff700 1 --2- 172.21.15.192:0/638709744 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1d201573e0 0x7f1d20153b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:11.175 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:11.166+0000 7f1d1f7fe700 1 --2- 172.21.15.192:0/638709744 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1d200670a0 0x7f1d200c4750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:11.175 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:11.166+0000 7f1d1effd700 1 -- 172.21.15.192:0/638709744 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1d201573e0 msgr2=0x7f1d20153b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:11.175 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:11.166+0000 7f1d1effd700 1 --2- 172.21.15.192:0/638709744 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1d201573e0 0x7f1d20153b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:11.175 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:11.166+0000 7f1d1effd700 1 -- 172.21.15.192:0/638709744 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1d200670a0 msgr2=0x7f1d200c4750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:11.175 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:11.166+0000 7f1d1effd700 1 --2- 172.21.15.192:0/638709744 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1d200670a0 0x7f1d200c4750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:11.175 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:11.166+0000 7f1d1effd700 1 -- 172.21.15.192:0/638709744 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d140094f0 con 0x7f1d20157010 2023-10-06T03:29:11.175 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:11.166+0000 7f1d1effd700 1 --2- 172.21.15.192:0/638709744 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1d20157010 0x7f1d20167da0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f1d2006f8c0 tx=0x7f1d14017000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:29:11.175 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:11.166+0000 7f1d077fe700 1 -- 172.21.15.192:0/638709744 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d1401f070 con 0x7f1d20157010 2023-10-06T03:29:11.175 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:11.166+0000 7f1d077fe700 1 -- 172.21.15.192:0/638709744 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d14002f30 con 0x7f1d20157010 2023-10-06T03:29:11.176 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:11.166+0000 7f1d077fe700 1 -- 172.21.15.192:0/638709744 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d14011b20 con 0x7f1d20157010 2023-10-06T03:29:11.176 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:11.166+0000 7f1d25aa9700 1 -- 172.21.15.192:0/638709744 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d200c1940 con 0x7f1d20157010 2023-10-06T03:29:11.176 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:11.166+0000 7f1d25aa9700 1 -- 172.21.15.192:0/638709744 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d201682e0 con 0x7f1d20157010 2023-10-06T03:29:11.176 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:11.166+0000 7f1d077fe700 1 -- 172.21.15.192:0/638709744 <== mon.1 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7f1d14023070 con 0x7f1d20157010 2023-10-06T03:29:11.176 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:11.166+0000 7f1d077fe700 1 -- 172.21.15.192:0/638709744 <== mon.1 v2:172.21.15.192:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 836+0+0 (secure 0 0 0) 0x7f1d140257e0 con 0x7f1d20157010 2023-10-06T03:29:11.176 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:11.170+0000 7f1d25aa9700 1 -- 172.21.15.192:0/638709744 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d200c2100 con 0x7f1d20157010 2023-10-06T03:29:11.178 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:11.174+0000 7f1d077fe700 1 -- 172.21.15.192:0/638709744 <== mon.1 v2:172.21.15.192:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7f1d1402a070 con 0x7f1d20157010 2023-10-06T03:29:11.252 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:11.246+0000 7f1d25aa9700 1 -- 172.21.15.192:0/638709744 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "osd new", "uuid": "a38de118-062b-4a05-9782-2ceddf6ff4df", "id": 1} v 0) v1 -- 0x7f1d2006ff60 con 0x7f1d20157010 2023-10-06T03:29:11.263 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:11.258+0000 7f1d077fe700 1 -- 172.21.15.192:0/638709744 <== mon.1 v2:172.21.15.192:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "a38de118-062b-4a05-9782-2ceddf6ff4df", "id": 1}]=0 v4) v1 ==== 112+0+2 (secure 0 0 0) 0x7f1d14015020 con 0x7f1d20157010 2023-10-06T03:29:11.263 INFO:teuthology.orchestra.run.smithi192.stdout:1 2023-10-06T03:29:11.264 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:11.258+0000 7f1d25aa9700 1 -- 172.21.15.192:0/638709744 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1d20157010 msgr2=0x7f1d20167da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:11.264 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:11.258+0000 7f1d25aa9700 1 --2- 172.21.15.192:0/638709744 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1d20157010 0x7f1d20167da0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f1d2006f8c0 tx=0x7f1d14017000 comp rx=0 tx=0).stop 2023-10-06T03:29:11.264 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:11.258+0000 7f1d25aa9700 1 -- 172.21.15.192:0/638709744 shutdown_connections 2023-10-06T03:29:11.264 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:11.258+0000 7f1d25aa9700 1 --2- 172.21.15.192:0/638709744 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1d201573e0 0x7f1d20153b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:11.264 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:11.258+0000 7f1d25aa9700 1 --2- 172.21.15.192:0/638709744 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1d20157010 0x7f1d20167da0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:11.264 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:11.258+0000 7f1d25aa9700 1 --2- 172.21.15.192:0/638709744 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1d200670a0 0x7f1d200c4750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:11.265 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:11.258+0000 7f1d25aa9700 1 -- 172.21.15.192:0/638709744 >> 172.21.15.192:0/638709744 conn(0x7f1d200ccd30 msgr2=0x7f1d20159a60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:29:11.265 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:11.258+0000 7f1d25aa9700 1 -- 172.21.15.192:0/638709744 shutdown_connections 2023-10-06T03:29:11.265 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:11.258+0000 7f1d25aa9700 1 -- 172.21.15.192:0/638709744 wait complete. 2023-10-06T03:29:11.276 DEBUG:teuthology.orchestra.run.smithi192:> sudo ceph --cluster ceph osd new 56e39be2-3407-4b69-877e-cf16617b06d6 2 2023-10-06T03:29:11.349 INFO:tasks.ceph.mgr.x.smithi192.stderr:2023-10-06T03:29:11.342+0000 7f0930332000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-06T03:29:11.373 INFO:tasks.ceph.mgr.y.smithi110.stderr:2023-10-06T03:29:11.366+0000 7f5523871000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-06T03:29:11.386 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:11.378+0000 7f76db589700 1 Processor -- start 2023-10-06T03:29:11.386 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:11.378+0000 7f76db589700 1 -- start start 2023-10-06T03:29:11.387 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:11.382+0000 7f76db589700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f76d4155fe0 0x7f76d4160dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:11.387 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:11.382+0000 7f76db589700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76d40caa70 con 0x7f76d40b1d40 2023-10-06T03:29:11.388 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:11.382+0000 7f76db589700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76d40ca740 con 0x7f76d40b2110 2023-10-06T03:29:11.388 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:11.382+0000 7f76db589700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f76d40b2560 con 0x7f76d4155fe0 2023-10-06T03:29:11.388 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:11.382+0000 7f76dad88700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f76d4155fe0 0x7f76d4160dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:11.388 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:11.382+0000 7f76d9d86700 1 --1- >> v1:172.21.15.192:6789/0 conn(0x7f76d40b2110 0x7f76d41558d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.192:49214/0 (socket says 172.21.15.192:49214) 2023-10-06T03:29:11.388 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:11.382+0000 7f76d9d86700 1 -- 172.21.15.192:0/848742135 learned_addr learned my addr 172.21.15.192:0/848742135 (peer_addr_for_me v1:172.21.15.192:0/0) 2023-10-06T03:29:11.388 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:11.382+0000 7f76dad88700 1 -- 172.21.15.192:0/848742135 >> v1:172.21.15.192:6789/0 conn(0x7f76d40b2110 legacy=0x7f76d41558d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:11.388 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:11.382+0000 7f76dad88700 1 -- 172.21.15.192:0/848742135 >> v1:172.21.15.110:6789/0 conn(0x7f76d40b1d40 legacy=0x7f76d40afc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:11.388 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:11.382+0000 7f76dad88700 1 -- 172.21.15.192:0/848742135 --> v2:172.21.15.192:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76d4159cc0 con 0x7f76d4155fe0 2023-10-06T03:29:11.388 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:11.382+0000 7f76d9585700 1 -- 172.21.15.192:0/848742135 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2403343996 0 0) 0x7f76d40ca740 con 0x7f76d40b2110 2023-10-06T03:29:11.388 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:11.382+0000 7f76dad88700 1 --2- 172.21.15.192:0/848742135 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f76d4155fe0 0x7f76d4160dc0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f76cc009e60 tx=0x7f76cc009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=98d57eaeed8c93e6 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:29:11.388 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:11.382+0000 7f76d9585700 1 -- 172.21.15.192:0/848742135 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76cc00e070 con 0x7f76d4155fe0 2023-10-06T03:29:11.389 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:11.382+0000 7f76d9585700 1 -- 172.21.15.192:0/848742135 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f76cc00bc30 con 0x7f76d4155fe0 2023-10-06T03:29:11.389 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:11.382+0000 7f76d9585700 1 -- 172.21.15.192:0/848742135 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76cc012430 con 0x7f76d4155fe0 2023-10-06T03:29:11.389 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:11.382+0000 7f76db589700 1 -- 172.21.15.192:0/848742135 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f76d4155fe0 msgr2=0x7f76d4160dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:11.390 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:11.382+0000 7f76db589700 1 --2- 172.21.15.192:0/848742135 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f76d4155fe0 0x7f76d4160dc0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f76cc009e60 tx=0x7f76cc009780 comp rx=0 tx=0).stop 2023-10-06T03:29:11.390 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:11.382+0000 7f76db589700 1 -- 172.21.15.192:0/848742135 shutdown_connections 2023-10-06T03:29:11.390 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:11.382+0000 7f76db589700 1 --2- 172.21.15.192:0/848742135 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f76d4155fe0 0x7f76d4160dc0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:11.390 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:11.382+0000 7f76db589700 1 -- 172.21.15.192:0/848742135 >> 172.21.15.192:0/848742135 conn(0x7f76d4068470 msgr2=0x7f76d4068870 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:29:11.390 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:11.382+0000 7f76db589700 1 -- 172.21.15.192:0/848742135 shutdown_connections 2023-10-06T03:29:11.390 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:11.382+0000 7f76db589700 1 -- 172.21.15.192:0/848742135 wait complete. 2023-10-06T03:29:11.391 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:11.386+0000 7f76db589700 1 Processor -- start 2023-10-06T03:29:11.391 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:11.386+0000 7f76db589700 1 -- start start 2023-10-06T03:29:11.393 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:11.386+0000 7f76db589700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f76d40b1d40 0x7f76d40c4a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:11.393 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:11.386+0000 7f76db589700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f76d40b2110 0x7f76d4164b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:11.393 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:11.386+0000 7f76db589700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f76d4155fe0 0x7f76d41650d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:11.393 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:11.386+0000 7f76db589700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76d40c6140 con 0x7f76d4155fe0 2023-10-06T03:29:11.393 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:11.386+0000 7f76db589700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76d40c5530 con 0x7f76d40b2110 2023-10-06T03:29:11.393 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:11.386+0000 7f76db589700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76d40c5810 con 0x7f76d40b1d40 2023-10-06T03:29:11.393 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:11.386+0000 7f76da587700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f76d40b1d40 0x7f76d40c4a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:11.393 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:11.386+0000 7f76da587700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f76d40b1d40 0x7f76d40c4a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3301/0 says I am v2:172.21.15.192:60540/0 (socket says 172.21.15.192:60540) 2023-10-06T03:29:11.393 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:11.386+0000 7f76da587700 1 -- 172.21.15.192:0/140696363 learned_addr learned my addr 172.21.15.192:0/140696363 (peer_addr_for_me v2:172.21.15.192:0/0) 2023-10-06T03:29:11.393 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:11.386+0000 7f76d9d86700 1 --2- 172.21.15.192:0/140696363 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f76d40b2110 0x7f76d4164b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:11.393 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:11.386+0000 7f76dad88700 1 --2- 172.21.15.192:0/140696363 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f76d4155fe0 0x7f76d41650d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:11.394 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:11.386+0000 7f76da587700 1 -- 172.21.15.192:0/140696363 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f76d40b2110 msgr2=0x7f76d4164b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:11.394 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:11.386+0000 7f76da587700 1 --2- 172.21.15.192:0/140696363 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f76d40b2110 0x7f76d4164b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:11.394 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:11.386+0000 7f76da587700 1 -- 172.21.15.192:0/140696363 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f76d4155fe0 msgr2=0x7f76d41650d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:11.394 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:11.386+0000 7f76da587700 1 --2- 172.21.15.192:0/140696363 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f76d4155fe0 0x7f76d41650d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:11.394 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:11.386+0000 7f76da587700 1 -- 172.21.15.192:0/140696363 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76cc009a20 con 0x7f76d40b1d40 2023-10-06T03:29:11.394 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:11.386+0000 7f76da587700 1 --2- 172.21.15.192:0/140696363 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f76d40b1d40 0x7f76d40c4a70 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f76d000c790 tx=0x7f76d0007f40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:29:11.394 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:11.386+0000 7f76ca7fc700 1 -- 172.21.15.192:0/140696363 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76d0011590 con 0x7f76d40b1d40 2023-10-06T03:29:11.394 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:11.386+0000 7f76ca7fc700 1 -- 172.21.15.192:0/140696363 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f76d00116f0 con 0x7f76d40b1d40 2023-10-06T03:29:11.394 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:11.386+0000 7f76db589700 1 -- 172.21.15.192:0/140696363 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76d40c5af0 con 0x7f76d40b1d40 2023-10-06T03:29:11.394 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:11.386+0000 7f76ca7fc700 1 -- 172.21.15.192:0/140696363 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76d0011ab0 con 0x7f76d40b1d40 2023-10-06T03:29:11.394 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:11.386+0000 7f76db589700 1 -- 172.21.15.192:0/140696363 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76d40c6ea0 con 0x7f76d40b1d40 2023-10-06T03:29:11.395 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:11.386+0000 7f76ca7fc700 1 -- 172.21.15.192:0/140696363 <== mon.2 v2:172.21.15.192:3301/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7f76d001c6c0 con 0x7f76d40b1d40 2023-10-06T03:29:11.395 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:11.386+0000 7f76ca7fc700 1 -- 172.21.15.192:0/140696363 <== mon.2 v2:172.21.15.192:3301/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 951+0+0 (secure 0 0 0) 0x7f76d001cda0 con 0x7f76d40b1d40 2023-10-06T03:29:11.395 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:11.390+0000 7f76db589700 1 -- 172.21.15.192:0/140696363 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f76d4070ff0 con 0x7f76d40b1d40 2023-10-06T03:29:11.398 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:11.390+0000 7f76ca7fc700 1 -- 172.21.15.192:0/140696363 <== mon.2 v2:172.21.15.192:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7f76d0017070 con 0x7f76d40b1d40 2023-10-06T03:29:11.405 INFO:tasks.ceph.mgr.x.smithi192.stderr:2023-10-06T03:29:11.398+0000 7f0930332000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-06T03:29:11.430 INFO:tasks.ceph.mgr.y.smithi110.stderr:2023-10-06T03:29:11.422+0000 7f5523871000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-06T03:29:11.473 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:11.466+0000 7f76db589700 1 -- 172.21.15.192:0/140696363 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_command({"prefix": "osd new", "uuid": "56e39be2-3407-4b69-877e-cf16617b06d6", "id": 2} v 0) v1 -- 0x7f76d40b07e0 con 0x7f76d40b1d40 2023-10-06T03:29:11.483 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:11.478+0000 7f76ca7fc700 1 -- 172.21.15.192:0/140696363 <== mon.2 v2:172.21.15.192:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "56e39be2-3407-4b69-877e-cf16617b06d6", "id": 2}]=0 v5) v1 ==== 112+0+2 (secure 0 0 0) 0x7f76d001c970 con 0x7f76d40b1d40 2023-10-06T03:29:11.483 INFO:teuthology.orchestra.run.smithi192.stdout:2 2023-10-06T03:29:11.485 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:11.478+0000 7f76db589700 1 -- 172.21.15.192:0/140696363 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f76d40b1d40 msgr2=0x7f76d40c4a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:11.485 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:11.478+0000 7f76db589700 1 --2- 172.21.15.192:0/140696363 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f76d40b1d40 0x7f76d40c4a70 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f76d000c790 tx=0x7f76d0007f40 comp rx=0 tx=0).stop 2023-10-06T03:29:11.486 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:11.478+0000 7f76db589700 1 -- 172.21.15.192:0/140696363 shutdown_connections 2023-10-06T03:29:11.486 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:11.478+0000 7f76db589700 1 --2- 172.21.15.192:0/140696363 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f76d4155fe0 0x7f76d41650d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:11.486 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:11.478+0000 7f76db589700 1 --2- 172.21.15.192:0/140696363 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f76d40b2110 0x7f76d4164b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:11.486 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:11.478+0000 7f76db589700 1 --2- 172.21.15.192:0/140696363 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f76d40b1d40 0x7f76d40c4a70 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:11.486 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:11.478+0000 7f76db589700 1 -- 172.21.15.192:0/140696363 >> 172.21.15.192:0/140696363 conn(0x7f76d4068470 msgr2=0x7f76d40afc70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:29:11.486 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:11.478+0000 7f76db589700 1 -- 172.21.15.192:0/140696363 shutdown_connections 2023-10-06T03:29:11.486 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:11.478+0000 7f76db589700 1 -- 172.21.15.192:0/140696363 wait complete. 2023-10-06T03:29:11.501 DEBUG:teuthology.orchestra.run.smithi192:> sudo ceph --cluster ceph osd new 01c82d13-c1d4-4bde-b55e-3e0089b40115 3 2023-10-06T03:29:11.514 INFO:tasks.ceph.mgr.x.smithi192.stderr:2023-10-06T03:29:11.510+0000 7f0930332000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-06T03:29:11.546 INFO:tasks.ceph.mgr.y.smithi110.stderr:2023-10-06T03:29:11.538+0000 7f5523871000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-06T03:29:11.598 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:11.590+0000 7f4e15031700 1 Processor -- start 2023-10-06T03:29:11.598 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:11.594+0000 7f4e15031700 1 -- start start 2023-10-06T03:29:11.600 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:11.594+0000 7f4e15031700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f4e1006b7c0 0x7f4e100ce170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:11.600 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:11.594+0000 7f4e15031700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4e1006c2a0 0x7f4e100b0a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:11.600 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:11.594+0000 7f4e15031700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e100cfa00 con 0x7f4e100ce6b0 2023-10-06T03:29:11.600 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:11.594+0000 7f4e15031700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4e100cb1c0 con 0x7f4e1006b7c0 2023-10-06T03:29:11.600 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:11.594+0000 7f4e15031700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e100cedf0 con 0x7f4e1006c2a0 2023-10-06T03:29:11.600 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:11.594+0000 7f4e0ffff700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f4e1006b7c0 0x7f4e100ce170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:11.600 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:11.594+0000 7f4e14830700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4e1006c2a0 0x7f4e100b0a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:11.600 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:11.594+0000 7f4e0ffff700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f4e1006b7c0 0x7f4e100ce170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.192:52902/0 (socket says 172.21.15.192:52902) 2023-10-06T03:29:11.600 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:11.594+0000 7f4e0ffff700 1 -- 172.21.15.192:0/936023805 learned_addr learned my addr 172.21.15.192:0/936023805 (peer_addr_for_me v2:172.21.15.192:0/0) 2023-10-06T03:29:11.602 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:11.594+0000 7f4e0ffff700 1 -- 172.21.15.192:0/936023805 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4e1006c2a0 msgr2=0x7f4e100b0a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:29:11.603 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:11.594+0000 7f4e0ffff700 1 --2- 172.21.15.192:0/936023805 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4e1006c2a0 0x7f4e100b0a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:11.603 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:11.594+0000 7f4e0ffff700 1 -- 172.21.15.192:0/936023805 >> v1:172.21.15.110:6789/0 conn(0x7f4e100ce6b0 legacy=0x7f4e1006bb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:11.603 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:11.594+0000 7f4e0effd700 1 -- 172.21.15.192:0/936023805 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1071493832 0 0) 0x7f4e100cfa00 con 0x7f4e100ce6b0 2023-10-06T03:29:11.603 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:11.594+0000 7f4e0ffff700 1 -- 172.21.15.192:0/936023805 --> v2:172.21.15.192:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e100cf0d0 con 0x7f4e1006b7c0 2023-10-06T03:29:11.603 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:11.594+0000 7f4e0ffff700 1 --2- 172.21.15.192:0/936023805 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4e1006b7c0 0x7f4e100ce170 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f4e000099b0 tx=0x7f4e00009250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=59fc333f9121b1c1 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:29:11.603 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:11.594+0000 7f4e0effd700 1 -- 172.21.15.192:0/936023805 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e00009ea0 con 0x7f4e1006b7c0 2023-10-06T03:29:11.603 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:11.594+0000 7f4e0effd700 1 -- 172.21.15.192:0/936023805 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e0000b860 con 0x7f4e1006b7c0 2023-10-06T03:29:11.603 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:11.594+0000 7f4e0effd700 1 -- 172.21.15.192:0/936023805 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e0000bc00 con 0x7f4e1006b7c0 2023-10-06T03:29:11.604 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:11.594+0000 7f4e15031700 1 -- 172.21.15.192:0/936023805 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4e1006b7c0 msgr2=0x7f4e100ce170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:11.604 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:11.594+0000 7f4e15031700 1 --2- 172.21.15.192:0/936023805 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4e1006b7c0 0x7f4e100ce170 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f4e000099b0 tx=0x7f4e00009250 comp rx=0 tx=0).stop 2023-10-06T03:29:11.604 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:11.594+0000 7f4e15031700 1 -- 172.21.15.192:0/936023805 shutdown_connections 2023-10-06T03:29:11.604 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:11.594+0000 7f4e15031700 1 --2- 172.21.15.192:0/936023805 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4e1006c2a0 0x7f4e100b0a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:11.604 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:11.594+0000 7f4e15031700 1 --2- 172.21.15.192:0/936023805 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4e1006b7c0 0x7f4e100ce170 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:11.604 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:11.594+0000 7f4e15031700 1 -- 172.21.15.192:0/936023805 >> 172.21.15.192:0/936023805 conn(0x7f4e100cfea0 msgr2=0x7f4e100d02a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:29:11.604 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:11.598+0000 7f4e15031700 1 -- 172.21.15.192:0/936023805 shutdown_connections 2023-10-06T03:29:11.604 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:11.598+0000 7f4e15031700 1 -- 172.21.15.192:0/936023805 wait complete. 2023-10-06T03:29:11.605 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:11.598+0000 7f4e15031700 1 Processor -- start 2023-10-06T03:29:11.605 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:11.598+0000 7f4e15031700 1 -- start start 2023-10-06T03:29:11.608 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:11.598+0000 7f4e15031700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4e1006b7c0 0x7f4e1016c8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:11.608 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:11.598+0000 7f4e15031700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4e1006c2a0 0x7f4e101478b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:11.608 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:11.602+0000 7f4e15031700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4e100ce6b0 0x7f4e1015b310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:11.608 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:11.602+0000 7f4e15031700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e1016d860 con 0x7f4e1006b7c0 2023-10-06T03:29:11.609 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:11.602+0000 7f4e15031700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e100ca240 con 0x7f4e1006c2a0 2023-10-06T03:29:11.609 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:11.602+0000 7f4e15031700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e10155610 con 0x7f4e100ce6b0 2023-10-06T03:29:11.609 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:11.602+0000 7f4e0f7fe700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4e1006c2a0 0x7f4e101478b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:11.609 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:11.602+0000 7f4e14830700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4e100ce6b0 0x7f4e1015b310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:11.609 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:11.602+0000 7f4e0f7fe700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4e1006c2a0 0x7f4e101478b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.192:52904/0 (socket says 172.21.15.192:52904) 2023-10-06T03:29:11.610 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:11.602+0000 7f4e0f7fe700 1 -- 172.21.15.192:0/2535687379 learned_addr learned my addr 172.21.15.192:0/2535687379 (peer_addr_for_me v2:172.21.15.192:0/0) 2023-10-06T03:29:11.610 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:11.602+0000 7f4e0ffff700 1 --2- 172.21.15.192:0/2535687379 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4e1006b7c0 0x7f4e1016c8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:11.610 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:11.602+0000 7f4e0f7fe700 1 -- 172.21.15.192:0/2535687379 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4e100ce6b0 msgr2=0x7f4e1015b310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:11.610 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:11.602+0000 7f4e0f7fe700 1 --2- 172.21.15.192:0/2535687379 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4e100ce6b0 0x7f4e1015b310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:11.610 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:11.602+0000 7f4e0f7fe700 1 -- 172.21.15.192:0/2535687379 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4e1006b7c0 msgr2=0x7f4e1016c8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:11.610 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:11.602+0000 7f4e0f7fe700 1 --2- 172.21.15.192:0/2535687379 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4e1006b7c0 0x7f4e1016c8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:11.610 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:11.602+0000 7f4e0f7fe700 1 -- 172.21.15.192:0/2535687379 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e000094f0 con 0x7f4e1006c2a0 2023-10-06T03:29:11.610 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:11.602+0000 7f4e0f7fe700 1 --2- 172.21.15.192:0/2535687379 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4e1006c2a0 0x7f4e101478b0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f4e0400c530 tx=0x7f4e0400c840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:29:11.610 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:11.602+0000 7f4df7fff700 1 -- 172.21.15.192:0/2535687379 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e04011590 con 0x7f4e1006c2a0 2023-10-06T03:29:11.610 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:11.602+0000 7f4df7fff700 1 -- 172.21.15.192:0/2535687379 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e04011710 con 0x7f4e1006c2a0 2023-10-06T03:29:11.611 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:11.602+0000 7f4e15031700 1 -- 172.21.15.192:0/2535687379 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e1016cfc0 con 0x7f4e1006c2a0 2023-10-06T03:29:11.611 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:11.602+0000 7f4df7fff700 1 -- 172.21.15.192:0/2535687379 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e04011ad0 con 0x7f4e1006c2a0 2023-10-06T03:29:11.611 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:11.602+0000 7f4e15031700 1 -- 172.21.15.192:0/2535687379 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e1016d580 con 0x7f4e1006c2a0 2023-10-06T03:29:11.611 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:11.602+0000 7f4df7fff700 1 -- 172.21.15.192:0/2535687379 <== mon.1 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7f4e0401c6c0 con 0x7f4e1006c2a0 2023-10-06T03:29:11.611 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:11.602+0000 7f4df7fff700 1 -- 172.21.15.192:0/2535687379 <== mon.1 v2:172.21.15.192:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1066+0+0 (secure 0 0 0) 0x7f4e040128c0 con 0x7f4e1006c2a0 2023-10-06T03:29:11.612 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:11.602+0000 7f4e15031700 1 -- 172.21.15.192:0/2535687379 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e10070ff0 con 0x7f4e1006c2a0 2023-10-06T03:29:11.612 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:11.606+0000 7f4df7fff700 1 -- 172.21.15.192:0/2535687379 <== mon.1 v2:172.21.15.192:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7f4e04017070 con 0x7f4e1006c2a0 2023-10-06T03:29:11.686 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:11.678+0000 7f4e15031700 1 -- 172.21.15.192:0/2535687379 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "osd new", "uuid": "01c82d13-c1d4-4bde-b55e-3e0089b40115", "id": 3} v 0) v1 -- 0x7f4e100cf3b0 con 0x7f4e1006c2a0 2023-10-06T03:29:11.693 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:11.686+0000 7f4df7fff700 1 -- 172.21.15.192:0/2535687379 <== mon.1 v2:172.21.15.192:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "01c82d13-c1d4-4bde-b55e-3e0089b40115", "id": 3}]=0 v6) v1 ==== 112+0+2 (secure 0 0 0) 0x7f4e0401cb50 con 0x7f4e1006c2a0 2023-10-06T03:29:11.693 INFO:teuthology.orchestra.run.smithi192.stdout:3 2023-10-06T03:29:11.694 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:11.690+0000 7f4e15031700 1 -- 172.21.15.192:0/2535687379 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4e1006c2a0 msgr2=0x7f4e101478b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:11.694 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:11.690+0000 7f4e15031700 1 --2- 172.21.15.192:0/2535687379 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4e1006c2a0 0x7f4e101478b0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f4e0400c530 tx=0x7f4e0400c840 comp rx=0 tx=0).stop 2023-10-06T03:29:11.695 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:11.690+0000 7f4e15031700 1 -- 172.21.15.192:0/2535687379 shutdown_connections 2023-10-06T03:29:11.695 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:11.690+0000 7f4e15031700 1 --2- 172.21.15.192:0/2535687379 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4e100ce6b0 0x7f4e1015b310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:11.695 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:11.690+0000 7f4e15031700 1 --2- 172.21.15.192:0/2535687379 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4e1006c2a0 0x7f4e101478b0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:11.695 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:11.690+0000 7f4e15031700 1 --2- 172.21.15.192:0/2535687379 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4e1006b7c0 0x7f4e1016c8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:11.695 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:11.690+0000 7f4e15031700 1 -- 172.21.15.192:0/2535687379 >> 172.21.15.192:0/2535687379 conn(0x7f4e100cfea0 msgr2=0x7f4e100d0270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:29:11.695 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:11.690+0000 7f4e15031700 1 -- 172.21.15.192:0/2535687379 shutdown_connections 2023-10-06T03:29:11.695 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:11.690+0000 7f4e15031700 1 -- 172.21.15.192:0/2535687379 wait complete. 2023-10-06T03:29:11.705 DEBUG:teuthology.orchestra.run.smithi192:> sudo ceph --cluster ceph osd new 86983430-5fa3-4d17-998f-03e2cd843237 4 2023-10-06T03:29:11.742 INFO:tasks.ceph.mgr.x.smithi192.stderr:2023-10-06T03:29:11.734+0000 7f0930332000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-06T03:29:11.765 INFO:tasks.ceph.mgr.y.smithi110.stderr:2023-10-06T03:29:11.758+0000 7f5523871000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-06T03:29:11.792 INFO:tasks.ceph.mgr.x.smithi192.stderr:2023-10-06T03:29:11.786+0000 7f0930332000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-06T03:29:11.807 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:11.802+0000 7f82a4e38700 1 Processor -- start 2023-10-06T03:29:11.808 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:11.802+0000 7f82a4e38700 1 -- start start 2023-10-06T03:29:11.810 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:11.802+0000 7f82a4e38700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f82a0155dc0 0x7f82a00afb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:11.810 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:11.802+0000 7f82a4e38700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f82a00677b0 0x7f82a00ae190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:11.810 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:11.802+0000 7f82a4e38700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82a00af1a0 con 0x7f82a0156190 2023-10-06T03:29:11.810 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:11.802+0000 7f82a4e38700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f82a006d050 con 0x7f82a0155dc0 2023-10-06T03:29:11.810 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:11.802+0000 7f82a4e38700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82a0156610 con 0x7f82a00677b0 2023-10-06T03:29:11.810 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:11.802+0000 7f829ed9d700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f82a00677b0 0x7f82a00ae190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:11.810 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:11.802+0000 7f829dd9b700 1 --1- >> v1:172.21.15.192:6789/0 conn(0x7f82a0156190 0x7f82a00670a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.192:49218/0 (socket says 172.21.15.192:49218) 2023-10-06T03:29:11.810 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:11.802+0000 7f829dd9b700 1 -- 172.21.15.192:0/2697643851 learned_addr learned my addr 172.21.15.192:0/2697643851 (peer_addr_for_me v1:172.21.15.192:0/0) 2023-10-06T03:29:11.811 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:11.802+0000 7f829e59c700 1 --2- 172.21.15.192:0/2697643851 >> v2:172.21.15.110:3300/0 conn(0x7f82a0155dc0 0x7f82a00afb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:11.811 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:11.802+0000 7f829d59a700 1 -- 172.21.15.192:0/2697643851 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2290525944 0 0) 0x7f82a00af1a0 con 0x7f82a0156190 2023-10-06T03:29:11.811 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:11.802+0000 7f829ed9d700 1 -- 172.21.15.192:0/2697643851 >> v2:172.21.15.110:3300/0 conn(0x7f82a0155dc0 msgr2=0x7f82a00afb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:29:11.811 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:11.802+0000 7f829ed9d700 1 --2- 172.21.15.192:0/2697643851 >> v2:172.21.15.110:3300/0 conn(0x7f82a0155dc0 0x7f82a00afb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:11.811 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:11.802+0000 7f829ed9d700 1 -- 172.21.15.192:0/2697643851 >> v1:172.21.15.192:6789/0 conn(0x7f82a0156190 legacy=0x7f82a00670a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:11.811 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:11.802+0000 7f829ed9d700 1 -- 172.21.15.192:0/2697643851 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82a00aeec0 con 0x7f82a00677b0 2023-10-06T03:29:11.811 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:11.802+0000 7f829ed9d700 1 --2- 172.21.15.192:0/2697643851 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f82a00677b0 0x7f82a00ae190 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f82980098e0 tx=0x7f82980091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1327a504d2e797bf server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:29:11.811 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:11.802+0000 7f829d59a700 1 -- 172.21.15.192:0/2697643851 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f829800d850 con 0x7f82a00677b0 2023-10-06T03:29:11.812 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:11.802+0000 7f829d59a700 1 -- 172.21.15.192:0/2697643851 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f829800d9b0 con 0x7f82a00677b0 2023-10-06T03:29:11.812 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:11.802+0000 7f829d59a700 1 -- 172.21.15.192:0/2697643851 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f829800dd50 con 0x7f82a00677b0 2023-10-06T03:29:11.812 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:11.806+0000 7f82a4e38700 1 -- 172.21.15.192:0/2697643851 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f82a00677b0 msgr2=0x7f82a00ae190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:11.813 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:11.806+0000 7f82a4e38700 1 --2- 172.21.15.192:0/2697643851 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f82a00677b0 0x7f82a00ae190 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f82980098e0 tx=0x7f82980091a0 comp rx=0 tx=0).stop 2023-10-06T03:29:11.813 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:11.806+0000 7f82a4e38700 1 -- 172.21.15.192:0/2697643851 shutdown_connections 2023-10-06T03:29:11.813 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:11.806+0000 7f82a4e38700 1 --2- 172.21.15.192:0/2697643851 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f82a00677b0 0x7f82a00ae190 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:11.813 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:11.806+0000 7f82a4e38700 1 --2- 172.21.15.192:0/2697643851 >> v2:172.21.15.110:3300/0 conn(0x7f82a0155dc0 0x7f82a00afb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:11.813 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:11.806+0000 7f82a4e38700 1 -- 172.21.15.192:0/2697643851 >> 172.21.15.192:0/2697643851 conn(0x7f82a00ccd30 msgr2=0x7f82a00cd130 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:29:11.813 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:11.806+0000 7f82a4e38700 1 -- 172.21.15.192:0/2697643851 shutdown_connections 2023-10-06T03:29:11.813 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:11.806+0000 7f82a4e38700 1 -- 172.21.15.192:0/2697643851 wait complete. 2023-10-06T03:29:11.813 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:11.806+0000 7f82a4e38700 1 Processor -- start 2023-10-06T03:29:11.813 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:11.806+0000 7f82a4e38700 1 -- start start 2023-10-06T03:29:11.814 INFO:tasks.ceph.mgr.y.smithi110.stderr:2023-10-06T03:29:11.806+0000 7f5523871000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-06T03:29:11.816 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:11.806+0000 7f82a4e38700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f82a00677b0 0x7f82a00c07d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:11.816 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:11.806+0000 7f82a4e38700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f82a0155dc0 0x7f82a00c7a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:11.816 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:11.806+0000 7f82a4e38700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f82a0156190 0x7f82a00c7f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:11.816 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:11.806+0000 7f82a4e38700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82a0159670 con 0x7f82a0156190 2023-10-06T03:29:11.816 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:11.806+0000 7f82a4e38700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82a00c2fb0 con 0x7f82a00677b0 2023-10-06T03:29:11.816 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:11.806+0000 7f82a4e38700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82a01591e0 con 0x7f82a0155dc0 2023-10-06T03:29:11.816 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:11.806+0000 7f829dd9b700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f82a0155dc0 0x7f82a00c7a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:11.816 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:11.806+0000 7f829e59c700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f82a00677b0 0x7f82a00c07d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:11.817 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:11.806+0000 7f829e59c700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f82a00677b0 0x7f82a00c07d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.192:52912/0 (socket says 172.21.15.192:52912) 2023-10-06T03:29:11.817 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:11.806+0000 7f829e59c700 1 -- 172.21.15.192:0/4011983458 learned_addr learned my addr 172.21.15.192:0/4011983458 (peer_addr_for_me v2:172.21.15.192:0/0) 2023-10-06T03:29:11.817 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:11.810+0000 7f829ed9d700 1 --2- 172.21.15.192:0/4011983458 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f82a0156190 0x7f82a00c7f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:11.817 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:11.810+0000 7f829e59c700 1 -- 172.21.15.192:0/4011983458 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f82a0155dc0 msgr2=0x7f82a00c7a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:11.817 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:11.810+0000 7f829e59c700 1 --2- 172.21.15.192:0/4011983458 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f82a0155dc0 0x7f82a00c7a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:11.817 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:11.810+0000 7f829e59c700 1 -- 172.21.15.192:0/4011983458 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f82a0156190 msgr2=0x7f82a00c7f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:11.818 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:11.810+0000 7f829e59c700 1 --2- 172.21.15.192:0/4011983458 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f82a0156190 0x7f82a00c7f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:11.818 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:11.810+0000 7f829e59c700 1 -- 172.21.15.192:0/4011983458 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8298009420 con 0x7f82a00677b0 2023-10-06T03:29:11.818 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:11.810+0000 7f829e59c700 1 --2- 172.21.15.192:0/4011983458 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f82a00677b0 0x7f82a00c07d0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f8290009f50 tx=0x7f829000b040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:29:11.818 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:11.810+0000 7f82967fc700 1 -- 172.21.15.192:0/4011983458 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f829000cad0 con 0x7f82a00677b0 2023-10-06T03:29:11.818 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:11.810+0000 7f82967fc700 1 -- 172.21.15.192:0/4011983458 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f829000f040 con 0x7f82a00677b0 2023-10-06T03:29:11.818 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:11.810+0000 7f82a4e38700 1 -- 172.21.15.192:0/4011983458 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82a0158bd0 con 0x7f82a00677b0 2023-10-06T03:29:11.818 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:11.810+0000 7f82967fc700 1 -- 172.21.15.192:0/4011983458 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f829000ce60 con 0x7f82a00677b0 2023-10-06T03:29:11.818 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:11.810+0000 7f82a4e38700 1 -- 172.21.15.192:0/4011983458 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82a00c0d10 con 0x7f82a00677b0 2023-10-06T03:29:11.818 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:11.810+0000 7f82967fc700 1 -- 172.21.15.192:0/4011983458 <== mon.1 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7f8290014910 con 0x7f82a00677b0 2023-10-06T03:29:11.818 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:11.810+0000 7f82967fc700 1 -- 172.21.15.192:0/4011983458 <== mon.1 v2:172.21.15.192:3300/0 5 ==== osd_map(6..6 src has 1..6) v4 ==== 1181+0+0 (secure 0 0 0) 0x7f829001dc30 con 0x7f82a00677b0 2023-10-06T03:29:11.818 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:11.810+0000 7f82a4e38700 1 -- 172.21.15.192:0/4011983458 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82a006ff60 con 0x7f82a00677b0 2023-10-06T03:29:11.820 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:11.814+0000 7f82967fc700 1 -- 172.21.15.192:0/4011983458 <== mon.1 v2:172.21.15.192:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7f8290018020 con 0x7f82a00677b0 2023-10-06T03:29:11.893 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:11.886+0000 7f82a4e38700 1 -- 172.21.15.192:0/4011983458 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "osd new", "uuid": "86983430-5fa3-4d17-998f-03e2cd843237", "id": 4} v 0) v1 -- 0x7f82a006ca00 con 0x7f82a00677b0 2023-10-06T03:29:11.901 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:11.894+0000 7f82967fc700 1 -- 172.21.15.192:0/4011983458 <== mon.1 v2:172.21.15.192:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "86983430-5fa3-4d17-998f-03e2cd843237", "id": 4}]=0 v7) v1 ==== 112+0+2 (secure 0 0 0) 0x7f8290014d50 con 0x7f82a00677b0 2023-10-06T03:29:11.901 INFO:teuthology.orchestra.run.smithi192.stdout:4 2023-10-06T03:29:11.902 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:11.898+0000 7f82a4e38700 1 -- 172.21.15.192:0/4011983458 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f82a00677b0 msgr2=0x7f82a00c07d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:11.903 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:11.898+0000 7f82a4e38700 1 --2- 172.21.15.192:0/4011983458 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f82a00677b0 0x7f82a00c07d0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f8290009f50 tx=0x7f829000b040 comp rx=0 tx=0).stop 2023-10-06T03:29:11.903 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:11.898+0000 7f82a4e38700 1 -- 172.21.15.192:0/4011983458 shutdown_connections 2023-10-06T03:29:11.903 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:11.898+0000 7f82a4e38700 1 --2- 172.21.15.192:0/4011983458 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f82a0156190 0x7f82a00c7f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:11.903 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:11.898+0000 7f82a4e38700 1 --2- 172.21.15.192:0/4011983458 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f82a0155dc0 0x7f82a00c7a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:11.903 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:11.898+0000 7f82a4e38700 1 --2- 172.21.15.192:0/4011983458 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f82a00677b0 0x7f82a00c07d0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:11.903 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:11.898+0000 7f82a4e38700 1 -- 172.21.15.192:0/4011983458 >> 172.21.15.192:0/4011983458 conn(0x7f82a00ccd30 msgr2=0x7f82a00b0790 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:29:11.903 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:11.898+0000 7f82a4e38700 1 -- 172.21.15.192:0/4011983458 shutdown_connections 2023-10-06T03:29:11.903 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:11.898+0000 7f82a4e38700 1 -- 172.21.15.192:0/4011983458 wait complete. 2023-10-06T03:29:11.916 DEBUG:teuthology.orchestra.run.smithi192:> sudo ceph --cluster ceph osd new 6945257f-821d-4619-a63f-ebe857eba6f4 5 2023-10-06T03:29:11.988 INFO:tasks.ceph.mgr.y.smithi110.stderr:2023-10-06T03:29:11.982+0000 7f5523871000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-06T03:29:11.991 INFO:tasks.ceph.mgr.x.smithi192.stderr:2023-10-06T03:29:11.986+0000 7f0930332000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-06T03:29:12.049 INFO:tasks.ceph.mgr.y.smithi110.stderr:2023-10-06T03:29:12.042+0000 7f5523871000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-06T03:29:12.054 INFO:tasks.ceph.mgr.x.smithi192.stderr:2023-10-06T03:29:12.050+0000 7f0930332000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-06T03:29:12.058 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:12.050+0000 7fbf7dc8e700 1 Processor -- start 2023-10-06T03:29:12.058 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:12.050+0000 7fbf7dc8e700 1 -- start start 2023-10-06T03:29:12.060 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:12.054+0000 7fbf7dc8e700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fbf78156b50 0x7fbf780c74e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:12.060 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:12.054+0000 7fbf7dc8e700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fbf7806ce90 0x7fbf78156f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:12.060 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:12.054+0000 7fbf7dc8e700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf7806b9f0 con 0x7fbf780c7a20 2023-10-06T03:29:12.060 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:12.054+0000 7fbf7dc8e700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf780c7e50 con 0x7fbf7806ce90 2023-10-06T03:29:12.060 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:12.054+0000 7fbf7dc8e700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf78068100 con 0x7fbf78156b50 2023-10-06T03:29:12.060 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:12.054+0000 7fbf777fe700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fbf78156b50 0x7fbf780c74e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:12.060 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:12.054+0000 7fbf777fe700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fbf78156b50 0x7fbf780c74e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.192:52918/0 (socket says 172.21.15.192:52918) 2023-10-06T03:29:12.061 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:12.054+0000 7fbf777fe700 1 -- 172.21.15.192:0/744181896 learned_addr learned my addr 172.21.15.192:0/744181896 (peer_addr_for_me v2:172.21.15.192:0/0) 2023-10-06T03:29:12.061 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:12.054+0000 7fbf77fff700 1 --2- 172.21.15.192:0/744181896 >> v2:172.21.15.110:3300/0 conn(0x7fbf7806ce90 0x7fbf78156f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:12.061 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:12.054+0000 7fbf777fe700 1 -- 172.21.15.192:0/744181896 >> v2:172.21.15.110:3300/0 conn(0x7fbf7806ce90 msgr2=0x7fbf78156f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:29:12.061 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:12.054+0000 7fbf777fe700 1 --2- 172.21.15.192:0/744181896 >> v2:172.21.15.110:3300/0 conn(0x7fbf7806ce90 0x7fbf78156f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:12.061 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:12.054+0000 7fbf777fe700 1 -- 172.21.15.192:0/744181896 >> v1:172.21.15.110:6789/0 conn(0x7fbf780c7a20 legacy=0x7fbf78067440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:12.061 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:12.054+0000 7fbf777fe700 1 -- 172.21.15.192:0/744181896 --> v2:172.21.15.192:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf780683e0 con 0x7fbf78156b50 2023-10-06T03:29:12.061 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:12.054+0000 7fbf777fe700 1 --2- 172.21.15.192:0/744181896 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbf78156b50 0x7fbf780c74e0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fbf78067100 tx=0x7fbf68009250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a9f543aaa2888a4f server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:29:12.063 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:12.054+0000 7fbf767fc700 1 -- 172.21.15.192:0/744181896 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf68009e70 con 0x7fbf78156b50 2023-10-06T03:29:12.063 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:12.054+0000 7fbf767fc700 1 -- 172.21.15.192:0/744181896 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf6800b860 con 0x7fbf78156b50 2023-10-06T03:29:12.063 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:12.054+0000 7fbf767fc700 1 -- 172.21.15.192:0/744181896 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf6800bc00 con 0x7fbf78156b50 2023-10-06T03:29:12.063 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:12.054+0000 7fbf7dc8e700 1 -- 172.21.15.192:0/744181896 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbf78156b50 msgr2=0x7fbf780c74e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:12.063 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:12.054+0000 7fbf7dc8e700 1 --2- 172.21.15.192:0/744181896 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbf78156b50 0x7fbf780c74e0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fbf78067100 tx=0x7fbf68009250 comp rx=0 tx=0).stop 2023-10-06T03:29:12.063 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:12.054+0000 7fbf7dc8e700 1 -- 172.21.15.192:0/744181896 shutdown_connections 2023-10-06T03:29:12.063 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:12.054+0000 7fbf7dc8e700 1 --2- 172.21.15.192:0/744181896 >> v2:172.21.15.110:3300/0 conn(0x7fbf7806ce90 0x7fbf78156f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:12.063 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:12.054+0000 7fbf7dc8e700 1 --2- 172.21.15.192:0/744181896 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbf78156b50 0x7fbf780c74e0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:12.063 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:12.054+0000 7fbf7dc8e700 1 -- 172.21.15.192:0/744181896 >> 172.21.15.192:0/744181896 conn(0x7fbf780c8f00 msgr2=0x7fbf780c9300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:29:12.064 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:12.054+0000 7fbf7dc8e700 1 -- 172.21.15.192:0/744181896 shutdown_connections 2023-10-06T03:29:12.064 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:12.054+0000 7fbf7dc8e700 1 -- 172.21.15.192:0/744181896 wait complete. 2023-10-06T03:29:12.064 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:12.058+0000 7fbf7dc8e700 1 Processor -- start 2023-10-06T03:29:12.064 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:12.058+0000 7fbf7dc8e700 1 -- start start 2023-10-06T03:29:12.067 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:12.058+0000 7fbf7dc8e700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbf7806ce90 0x7fbf78153090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:12.067 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:12.058+0000 7fbf7dc8e700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbf780c7a20 0x7fbf78167cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:12.067 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:12.058+0000 7fbf7dc8e700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbf78156b50 0x7fbf780c3540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:12.067 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:12.058+0000 7fbf7dc8e700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf780ad410 con 0x7fbf780c7a20 2023-10-06T03:29:12.067 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:12.058+0000 7fbf7dc8e700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf781539e0 con 0x7fbf7806ce90 2023-10-06T03:29:12.067 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:12.058+0000 7fbf7dc8e700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf78168210 con 0x7fbf78156b50 2023-10-06T03:29:12.067 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:12.058+0000 7fbf777fe700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbf7806ce90 0x7fbf78153090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:12.068 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:12.058+0000 7fbf77fff700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbf78156b50 0x7fbf780c3540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:12.068 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:12.058+0000 7fbf777fe700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbf7806ce90 0x7fbf78153090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.192:52934/0 (socket says 172.21.15.192:52934) 2023-10-06T03:29:12.068 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:12.058+0000 7fbf777fe700 1 -- 172.21.15.192:0/3742500260 learned_addr learned my addr 172.21.15.192:0/3742500260 (peer_addr_for_me v2:172.21.15.192:0/0) 2023-10-06T03:29:12.068 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:12.058+0000 7fbf76ffd700 1 --2- 172.21.15.192:0/3742500260 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbf780c7a20 0x7fbf78167cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:12.068 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:12.058+0000 7fbf777fe700 1 -- 172.21.15.192:0/3742500260 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbf78156b50 msgr2=0x7fbf780c3540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:12.068 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:12.058+0000 7fbf777fe700 1 --2- 172.21.15.192:0/3742500260 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbf78156b50 0x7fbf780c3540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:12.068 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:12.058+0000 7fbf777fe700 1 -- 172.21.15.192:0/3742500260 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbf780c7a20 msgr2=0x7fbf78167cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:12.068 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:12.058+0000 7fbf777fe700 1 --2- 172.21.15.192:0/3742500260 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbf780c7a20 0x7fbf78167cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:12.068 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:12.058+0000 7fbf777fe700 1 -- 172.21.15.192:0/3742500260 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf680094f0 con 0x7fbf7806ce90 2023-10-06T03:29:12.069 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:12.058+0000 7fbf77fff700 1 --2- 172.21.15.192:0/3742500260 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbf78156b50 0x7fbf780c3540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-06T03:29:12.069 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:12.058+0000 7fbf777fe700 1 --2- 172.21.15.192:0/3742500260 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbf7806ce90 0x7fbf78153090 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fbf78068240 tx=0x7fbf68017000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:29:12.069 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:12.062+0000 7fbf5f7fe700 1 -- 172.21.15.192:0/3742500260 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf6800bbe0 con 0x7fbf7806ce90 2023-10-06T03:29:12.069 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:12.062+0000 7fbf5f7fe700 1 -- 172.21.15.192:0/3742500260 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf68011420 con 0x7fbf7806ce90 2023-10-06T03:29:12.069 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:12.062+0000 7fbf7dc8e700 1 -- 172.21.15.192:0/3742500260 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf780ac800 con 0x7fbf7806ce90 2023-10-06T03:29:12.069 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:12.062+0000 7fbf5f7fe700 1 -- 172.21.15.192:0/3742500260 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf680117c0 con 0x7fbf7806ce90 2023-10-06T03:29:12.069 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:12.062+0000 7fbf7dc8e700 1 -- 172.21.15.192:0/3742500260 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf780acdc0 con 0x7fbf7806ce90 2023-10-06T03:29:12.069 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:12.062+0000 7fbf5f7fe700 1 -- 172.21.15.192:0/3742500260 <== mon.1 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7fbf68011bf0 con 0x7fbf7806ce90 2023-10-06T03:29:12.069 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:12.062+0000 7fbf5f7fe700 1 -- 172.21.15.192:0/3742500260 <== mon.1 v2:172.21.15.192:3300/0 5 ==== osd_map(7..7 src has 1..7) v4 ==== 1296+0+0 (secure 0 0 0) 0x7fbf68023970 con 0x7fbf7806ce90 2023-10-06T03:29:12.070 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:12.062+0000 7fbf7dc8e700 1 -- 172.21.15.192:0/3742500260 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf7806ff60 con 0x7fbf7806ce90 2023-10-06T03:29:12.071 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:12.066+0000 7fbf5f7fe700 1 -- 172.21.15.192:0/3742500260 <== mon.1 v2:172.21.15.192:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7fbf68028070 con 0x7fbf7806ce90 2023-10-06T03:29:12.110 INFO:tasks.ceph.mgr.y.smithi110.stderr:2023-10-06T03:29:12.102+0000 7f5523871000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-06T03:29:12.118 INFO:tasks.ceph.mgr.x.smithi192.stderr:2023-10-06T03:29:12.114+0000 7f0930332000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-06T03:29:12.144 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:12.138+0000 7fbf7dc8e700 1 -- 172.21.15.192:0/3742500260 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "osd new", "uuid": "6945257f-821d-4619-a63f-ebe857eba6f4", "id": 5} v 0) v1 -- 0x7fbf780686c0 con 0x7fbf7806ce90 2023-10-06T03:29:12.151 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:12.146+0000 7fbf5f7fe700 1 -- 172.21.15.192:0/3742500260 <== mon.1 v2:172.21.15.192:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "6945257f-821d-4619-a63f-ebe857eba6f4", "id": 5}]=0 v8) v1 ==== 112+0+2 (secure 0 0 0) 0x7fbf68015020 con 0x7fbf7806ce90 2023-10-06T03:29:12.152 INFO:teuthology.orchestra.run.smithi192.stdout:5 2023-10-06T03:29:12.153 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:12.146+0000 7fbf7dc8e700 1 -- 172.21.15.192:0/3742500260 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbf7806ce90 msgr2=0x7fbf78153090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:12.153 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:12.146+0000 7fbf7dc8e700 1 --2- 172.21.15.192:0/3742500260 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbf7806ce90 0x7fbf78153090 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fbf78068240 tx=0x7fbf68017000 comp rx=0 tx=0).stop 2023-10-06T03:29:12.153 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:12.146+0000 7fbf7dc8e700 1 -- 172.21.15.192:0/3742500260 shutdown_connections 2023-10-06T03:29:12.153 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:12.146+0000 7fbf7dc8e700 1 --2- 172.21.15.192:0/3742500260 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fbf78156b50 0x7fbf780c3540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:12.153 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:12.146+0000 7fbf7dc8e700 1 --2- 172.21.15.192:0/3742500260 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbf780c7a20 0x7fbf78167cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:12.153 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:12.146+0000 7fbf7dc8e700 1 --2- 172.21.15.192:0/3742500260 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbf7806ce90 0x7fbf78153090 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:12.154 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:12.146+0000 7fbf7dc8e700 1 -- 172.21.15.192:0/3742500260 >> 172.21.15.192:0/3742500260 conn(0x7fbf780c8f00 msgr2=0x7fbf780c20e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:29:12.154 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:12.146+0000 7fbf7dc8e700 1 -- 172.21.15.192:0/3742500260 shutdown_connections 2023-10-06T03:29:12.154 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:12.146+0000 7fbf7dc8e700 1 -- 172.21.15.192:0/3742500260 wait complete. 2023-10-06T03:29:12.164 DEBUG:teuthology.orchestra.run.smithi192:> sudo ceph --cluster ceph osd new 7854d1ae-7060-4780-8f97-73cea5e7a12f 6 2023-10-06T03:29:12.263 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:12.258+0000 7f9dbe07e700 1 Processor -- start 2023-10-06T03:29:12.264 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:12.258+0000 7f9dbe07e700 1 -- start start 2023-10-06T03:29:12.265 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:12.258+0000 7f9dbe07e700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f9db80c8a10 0x7f9db8157090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:12.265 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:12.258+0000 7f9dbe07e700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9db80672b0 con 0x7f9db8156cc0 2023-10-06T03:29:12.265 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:12.258+0000 7f9dbe07e700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9db815b2b0 con 0x7f9db815ab40 2023-10-06T03:29:12.265 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:12.258+0000 7f9dbe07e700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9db815b590 con 0x7f9db80c8a10 2023-10-06T03:29:12.265 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:12.258+0000 7f9dbd87d700 1 --1- >> v1:172.21.15.192:6789/0 conn(0x7f9db815ab40 0x7f9db806c7d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.192:49224/0 (socket says 172.21.15.192:49224) 2023-10-06T03:29:12.265 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:12.258+0000 7f9dbd87d700 1 -- 172.21.15.192:0/2781435470 learned_addr learned my addr 172.21.15.192:0/2781435470 (peer_addr_for_me v1:172.21.15.192:0/0) 2023-10-06T03:29:12.266 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:12.258+0000 7f9dbc87b700 1 --2- 172.21.15.192:0/2781435470 >> v2:172.21.15.110:3300/0 conn(0x7f9db80c8a10 0x7f9db8157090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:12.267 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:12.258+0000 7f9daffff700 1 -- 172.21.15.192:0/2781435470 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1076149500 0 0) 0x7f9db815b2b0 con 0x7f9db815ab40 2023-10-06T03:29:12.267 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:12.258+0000 7f9daffff700 1 -- 172.21.15.192:0/2781435470 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9da4003570 con 0x7f9db815ab40 2023-10-06T03:29:12.267 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:12.258+0000 7f9daffff700 1 -- 172.21.15.192:0/2781435470 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2587492546 0 0) 0x7f9db80672b0 con 0x7f9db8156cc0 2023-10-06T03:29:12.267 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:12.258+0000 7f9daffff700 1 -- 172.21.15.192:0/2781435470 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9db815b2b0 con 0x7f9db8156cc0 2023-10-06T03:29:12.267 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:12.262+0000 7f9daffff700 1 -- 172.21.15.192:0/2781435470 <== mon.1 v1:172.21.15.192:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3671749431 0 0) 0x7f9da4003570 con 0x7f9db815ab40 2023-10-06T03:29:12.267 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:12.262+0000 7f9dbc87b700 1 -- 172.21.15.192:0/2781435470 >> v1:172.21.15.192:6789/0 conn(0x7f9db815ab40 legacy=0x7f9db806c7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:12.267 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:12.262+0000 7f9dbc87b700 1 -- 172.21.15.192:0/2781435470 >> v1:172.21.15.110:6789/0 conn(0x7f9db8156cc0 legacy=0x7f9db80caee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:12.267 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:12.262+0000 7f9dbc87b700 1 -- 172.21.15.192:0/2781435470 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9db815b870 con 0x7f9db80c8a10 2023-10-06T03:29:12.268 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:12.262+0000 7f9dbc87b700 1 --2- 172.21.15.192:0/2781435470 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9db80c8a10 0x7f9db8157090 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f9da8009860 tx=0x7f9da80091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b2916a76079a706 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:29:12.268 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:12.262+0000 7f9daffff700 1 -- 172.21.15.192:0/2781435470 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9db00017f0 con 0x7f9db80c8a10 2023-10-06T03:29:12.268 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:12.262+0000 7f9daffff700 1 -- 172.21.15.192:0/2781435470 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9da8011420 con 0x7f9db80c8a10 2023-10-06T03:29:12.268 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:12.262+0000 7f9daffff700 1 -- 172.21.15.192:0/2781435470 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9da80117a0 con 0x7f9db80c8a10 2023-10-06T03:29:12.269 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:12.262+0000 7f9dbe07e700 1 -- 172.21.15.192:0/2781435470 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9db80c8a10 msgr2=0x7f9db8157090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:12.269 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:12.262+0000 7f9dbe07e700 1 --2- 172.21.15.192:0/2781435470 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9db80c8a10 0x7f9db8157090 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f9da8009860 tx=0x7f9da80091a0 comp rx=0 tx=0).stop 2023-10-06T03:29:12.269 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:12.262+0000 7f9dbe07e700 1 -- 172.21.15.192:0/2781435470 shutdown_connections 2023-10-06T03:29:12.269 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:12.262+0000 7f9dbe07e700 1 --2- 172.21.15.192:0/2781435470 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9db80c8a10 0x7f9db8157090 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:12.269 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:12.262+0000 7f9dbe07e700 1 -- 172.21.15.192:0/2781435470 >> 172.21.15.192:0/2781435470 conn(0x7f9db806ab30 msgr2=0x7f9db80adff0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:29:12.269 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:12.262+0000 7f9dbe07e700 1 -- 172.21.15.192:0/2781435470 shutdown_connections 2023-10-06T03:29:12.269 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:12.262+0000 7f9dbe07e700 1 -- 172.21.15.192:0/2781435470 wait complete. 2023-10-06T03:29:12.270 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:12.262+0000 7f9dbe07e700 1 Processor -- start 2023-10-06T03:29:12.270 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:12.266+0000 7f9dbe07e700 1 -- start start 2023-10-06T03:29:12.273 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:12.266+0000 7f9dbe07e700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9db80c8a10 0x7f9db80ab7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:12.273 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:12.266+0000 7f9dbe07e700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9db8156cc0 0x7f9db80c70b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:12.273 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:12.266+0000 7f9dbe07e700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9db815ab40 0x7f9db80c75f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:12.273 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:12.266+0000 7f9dbe07e700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9db8143b90 con 0x7f9db80c8a10 2023-10-06T03:29:12.273 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:12.266+0000 7f9dbe07e700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9db80c7b30 con 0x7f9db8156cc0 2023-10-06T03:29:12.273 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:12.266+0000 7f9dbe07e700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9db80c4720 con 0x7f9db815ab40 2023-10-06T03:29:12.273 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:12.266+0000 7f9dbd87d700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9db815ab40 0x7f9db80c75f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:12.273 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:12.266+0000 7f9dbc87b700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9db8156cc0 0x7f9db80c70b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:12.273 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:12.266+0000 7f9dbc87b700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9db8156cc0 0x7f9db80c70b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.192:52948/0 (socket says 172.21.15.192:52948) 2023-10-06T03:29:12.273 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:12.266+0000 7f9dbc87b700 1 -- 172.21.15.192:0/396868769 learned_addr learned my addr 172.21.15.192:0/396868769 (peer_addr_for_me v2:172.21.15.192:0/0) 2023-10-06T03:29:12.273 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:12.266+0000 7f9dbd07c700 1 --2- 172.21.15.192:0/396868769 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9db80c8a10 0x7f9db80ab7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:12.274 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:12.266+0000 7f9dbc87b700 1 -- 172.21.15.192:0/396868769 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9db815ab40 msgr2=0x7f9db80c75f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:12.274 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:12.266+0000 7f9dbc87b700 1 --2- 172.21.15.192:0/396868769 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9db815ab40 0x7f9db80c75f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:12.274 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:12.266+0000 7f9dbc87b700 1 -- 172.21.15.192:0/396868769 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9db80c8a10 msgr2=0x7f9db80ab7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:12.274 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:12.266+0000 7f9dbc87b700 1 --2- 172.21.15.192:0/396868769 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9db80c8a10 0x7f9db80ab7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:12.274 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:12.266+0000 7f9dbc87b700 1 -- 172.21.15.192:0/396868769 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9da8009420 con 0x7f9db8156cc0 2023-10-06T03:29:12.274 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:12.266+0000 7f9dbd07c700 1 --2- 172.21.15.192:0/396868769 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9db80c8a10 0x7f9db80ab7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-06T03:29:12.274 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:12.266+0000 7f9dbc87b700 1 --2- 172.21.15.192:0/396868769 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9db8156cc0 0x7f9db80c70b0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f9db80691d0 tx=0x7f9da8016000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:29:12.274 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:12.266+0000 7f9dad7fa700 1 -- 172.21.15.192:0/396868769 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9da801cbe0 con 0x7f9db8156cc0 2023-10-06T03:29:12.274 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:12.266+0000 7f9dad7fa700 1 -- 172.21.15.192:0/396868769 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9da801cd80 con 0x7f9db8156cc0 2023-10-06T03:29:12.274 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:12.266+0000 7f9dbe07e700 1 -- 172.21.15.192:0/396868769 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9db80c4fc0 con 0x7f9db8156cc0 2023-10-06T03:29:12.274 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:12.266+0000 7f9dad7fa700 1 -- 172.21.15.192:0/396868769 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9da8019640 con 0x7f9db8156cc0 2023-10-06T03:29:12.275 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:12.266+0000 7f9dbe07e700 1 -- 172.21.15.192:0/396868769 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9db80c4970 con 0x7f9db8156cc0 2023-10-06T03:29:12.275 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:12.266+0000 7f9dad7fa700 1 -- 172.21.15.192:0/396868769 <== mon.1 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7f9da8019c70 con 0x7f9db8156cc0 2023-10-06T03:29:12.275 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:12.266+0000 7f9dad7fa700 1 -- 172.21.15.192:0/396868769 <== mon.1 v2:172.21.15.192:3300/0 5 ==== osd_map(8..8 src has 1..8) v4 ==== 1411+0+0 (secure 0 0 0) 0x7f9da8025d50 con 0x7f9db8156cc0 2023-10-06T03:29:12.275 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:12.266+0000 7f9dbe07e700 1 -- 172.21.15.192:0/396868769 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9db80c5150 con 0x7f9db8156cc0 2023-10-06T03:29:12.275 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:12.270+0000 7f9dad7fa700 1 -- 172.21.15.192:0/396868769 <== mon.1 v2:172.21.15.192:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7f9da802a060 con 0x7f9db8156cc0 2023-10-06T03:29:12.352 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:12.346+0000 7f9dbe07e700 1 -- 172.21.15.192:0/396868769 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "osd new", "uuid": "7854d1ae-7060-4780-8f97-73cea5e7a12f", "id": 6} v 0) v1 -- 0x7f9db8070ff0 con 0x7f9db8156cc0 2023-10-06T03:29:12.359 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:12.354+0000 7f9dad7fa700 1 -- 172.21.15.192:0/396868769 <== mon.1 v2:172.21.15.192:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "7854d1ae-7060-4780-8f97-73cea5e7a12f", "id": 6}]=0 v9) v1 ==== 112+0+2 (secure 0 0 0) 0x7f9da8025750 con 0x7f9db8156cc0 2023-10-06T03:29:12.359 INFO:teuthology.orchestra.run.smithi192.stdout:6 2023-10-06T03:29:12.360 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:12.354+0000 7f9dbe07e700 1 -- 172.21.15.192:0/396868769 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9db8156cc0 msgr2=0x7f9db80c70b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:12.360 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:12.354+0000 7f9dbe07e700 1 --2- 172.21.15.192:0/396868769 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9db8156cc0 0x7f9db80c70b0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f9db80691d0 tx=0x7f9da8016000 comp rx=0 tx=0).stop 2023-10-06T03:29:12.360 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:12.354+0000 7f9dbe07e700 1 -- 172.21.15.192:0/396868769 shutdown_connections 2023-10-06T03:29:12.360 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:12.354+0000 7f9dbe07e700 1 --2- 172.21.15.192:0/396868769 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9db815ab40 0x7f9db80c75f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:12.360 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:12.354+0000 7f9dbe07e700 1 --2- 172.21.15.192:0/396868769 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9db8156cc0 0x7f9db80c70b0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:12.361 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:12.354+0000 7f9dbe07e700 1 --2- 172.21.15.192:0/396868769 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9db80c8a10 0x7f9db80ab7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:12.361 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:12.354+0000 7f9dbe07e700 1 -- 172.21.15.192:0/396868769 >> 172.21.15.192:0/396868769 conn(0x7f9db806ab30 msgr2=0x7f9db8156080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:29:12.361 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:12.354+0000 7f9dbe07e700 1 -- 172.21.15.192:0/396868769 shutdown_connections 2023-10-06T03:29:12.361 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:12.354+0000 7f9dbe07e700 1 -- 172.21.15.192:0/396868769 wait complete. 2023-10-06T03:29:12.373 DEBUG:teuthology.orchestra.run.smithi192:> sudo ceph --cluster ceph osd new f1ebdfd5-b859-482b-9f97-c18c4a26f445 7 2023-10-06T03:29:12.476 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:12.470+0000 7f888cbe2700 1 Processor -- start 2023-10-06T03:29:12.476 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:12.470+0000 7f888cbe2700 1 -- start start 2023-10-06T03:29:12.480 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:12.470+0000 7f888cbe2700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f888806a700 0x7f888806b7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:12.480 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:12.470+0000 7f888cbe2700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f888806bd00 0x7f888806c1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:12.480 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:12.470+0000 7f888cbe2700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8888159140 con 0x7f8888155290 2023-10-06T03:29:12.481 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:12.470+0000 7f888cbe2700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f88880c7be0 con 0x7f888806bd00 2023-10-06T03:29:12.481 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:12.470+0000 7f888cbe2700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88881599e0 con 0x7f888806a700 2023-10-06T03:29:12.481 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:12.470+0000 7f8886ffd700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f888806a700 0x7f888806b7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:12.481 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:12.470+0000 7f8886ffd700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f888806a700 0x7f888806b7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3301/0 says I am v2:172.21.15.192:60592/0 (socket says 172.21.15.192:60592) 2023-10-06T03:29:12.481 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:12.470+0000 7f8886ffd700 1 -- 172.21.15.192:0/3490146904 learned_addr learned my addr 172.21.15.192:0/3490146904 (peer_addr_for_me v2:172.21.15.192:0/0) 2023-10-06T03:29:12.482 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:12.470+0000 7f8887fff700 1 --2- 172.21.15.192:0/3490146904 >> v2:172.21.15.110:3300/0 conn(0x7f888806bd00 0x7f888806c1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:12.482 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:12.470+0000 7f8886ffd700 1 -- 172.21.15.192:0/3490146904 >> v2:172.21.15.110:3300/0 conn(0x7f888806bd00 msgr2=0x7f888806c1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:29:12.482 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:12.470+0000 7f8886ffd700 1 --2- 172.21.15.192:0/3490146904 >> v2:172.21.15.110:3300/0 conn(0x7f888806bd00 0x7f888806c1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:12.482 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:12.470+0000 7f8886ffd700 1 -- 172.21.15.192:0/3490146904 >> v1:172.21.15.110:6789/0 conn(0x7f8888155290 legacy=0x7f8888069ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:12.482 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:12.470+0000 7f8886ffd700 1 -- 172.21.15.192:0/3490146904 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f888806c6e0 con 0x7f888806a700 2023-10-06T03:29:12.482 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:12.470+0000 7f8886ffd700 1 --2- 172.21.15.192:0/3490146904 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f888806a700 0x7f888806b7c0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f887c009990 tx=0x7f887c009220 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b6a581ab7d5e51 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:29:12.483 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:12.470+0000 7f88867fc700 1 -- 172.21.15.192:0/3490146904 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f887c009e80 con 0x7f888806a700 2023-10-06T03:29:12.483 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:12.470+0000 7f88867fc700 1 -- 172.21.15.192:0/3490146904 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f887c005c10 con 0x7f888806a700 2023-10-06T03:29:12.483 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:12.470+0000 7f88867fc700 1 -- 172.21.15.192:0/3490146904 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f887c00b650 con 0x7f888806a700 2023-10-06T03:29:12.483 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:12.474+0000 7f888cbe2700 1 -- 172.21.15.192:0/3490146904 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f888806a700 msgr2=0x7f888806b7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:12.483 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:12.474+0000 7f888cbe2700 1 --2- 172.21.15.192:0/3490146904 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f888806a700 0x7f888806b7c0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f887c009990 tx=0x7f887c009220 comp rx=0 tx=0).stop 2023-10-06T03:29:12.483 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:12.474+0000 7f888cbe2700 1 -- 172.21.15.192:0/3490146904 shutdown_connections 2023-10-06T03:29:12.484 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:12.474+0000 7f888cbe2700 1 --2- 172.21.15.192:0/3490146904 >> v2:172.21.15.110:3300/0 conn(0x7f888806bd00 0x7f888806c1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:12.484 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:12.474+0000 7f888cbe2700 1 --2- 172.21.15.192:0/3490146904 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f888806a700 0x7f888806b7c0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:12.484 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:12.474+0000 7f888cbe2700 1 -- 172.21.15.192:0/3490146904 >> 172.21.15.192:0/3490146904 conn(0x7f8888067b70 msgr2=0x7f88880cc560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:29:12.484 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:12.474+0000 7f888cbe2700 1 -- 172.21.15.192:0/3490146904 shutdown_connections 2023-10-06T03:29:12.485 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:12.474+0000 7f888cbe2700 1 -- 172.21.15.192:0/3490146904 wait complete. 2023-10-06T03:29:12.485 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:12.474+0000 7f888cbe2700 1 Processor -- start 2023-10-06T03:29:12.485 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:12.474+0000 7f888cbe2700 1 -- start start 2023-10-06T03:29:12.489 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:12.474+0000 7f888cbe2700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f888806a700 0x7f888814c550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:12.491 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:12.474+0000 7f888cbe2700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f888806bd00 0x7f88880b1590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:12.491 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:12.474+0000 7f888cbe2700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8888155290 0x7f88880b1ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:12.491 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:12.474+0000 7f888cbe2700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f888816b0f0 con 0x7f888806a700 2023-10-06T03:29:12.491 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:12.474+0000 7f888cbe2700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88881587a0 con 0x7f888806bd00 2023-10-06T03:29:12.491 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:12.474+0000 7f888cbe2700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8888157e70 con 0x7f8888155290 2023-10-06T03:29:12.492 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:12.474+0000 7f8887fff700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8888155290 0x7f88880b1ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:12.492 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:12.474+0000 7f8886ffd700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f888806bd00 0x7f88880b1590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:12.492 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:12.474+0000 7f8887fff700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8888155290 0x7f88880b1ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3301/0 says I am v2:172.21.15.192:60604/0 (socket says 172.21.15.192:60604) 2023-10-06T03:29:12.492 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:12.474+0000 7f8887fff700 1 -- 172.21.15.192:0/3946331568 learned_addr learned my addr 172.21.15.192:0/3946331568 (peer_addr_for_me v2:172.21.15.192:0/0) 2023-10-06T03:29:12.492 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:12.474+0000 7f88877fe700 1 --2- 172.21.15.192:0/3946331568 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f888806a700 0x7f888814c550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:12.492 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:12.474+0000 7f8887fff700 1 -- 172.21.15.192:0/3946331568 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f888806bd00 msgr2=0x7f88880b1590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:12.492 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:12.474+0000 7f8887fff700 1 --2- 172.21.15.192:0/3946331568 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f888806bd00 0x7f88880b1590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:12.492 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:12.474+0000 7f8887fff700 1 -- 172.21.15.192:0/3946331568 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f888806a700 msgr2=0x7f888814c550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:12.492 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:12.474+0000 7f8887fff700 1 --2- 172.21.15.192:0/3946331568 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f888806a700 0x7f888814c550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:12.492 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:12.474+0000 7f8887fff700 1 -- 172.21.15.192:0/3946331568 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f887c009480 con 0x7f8888155290 2023-10-06T03:29:12.492 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:12.474+0000 7f8887fff700 1 --2- 172.21.15.192:0/3946331568 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8888155290 0x7f88880b1ad0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f8880009ec0 tx=0x7f888000b040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:29:12.493 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:12.474+0000 7f886f7fe700 1 -- 172.21.15.192:0/3946331568 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f888000cc20 con 0x7f8888155290 2023-10-06T03:29:12.493 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:12.474+0000 7f886f7fe700 1 -- 172.21.15.192:0/3946331568 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f888000f040 con 0x7f8888155290 2023-10-06T03:29:12.493 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:12.474+0000 7f888cbe2700 1 -- 172.21.15.192:0/3946331568 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8888158150 con 0x7f8888155290 2023-10-06T03:29:12.493 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:12.474+0000 7f886f7fe700 1 -- 172.21.15.192:0/3946331568 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8880014480 con 0x7f8888155290 2023-10-06T03:29:12.493 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:12.474+0000 7f888cbe2700 1 -- 172.21.15.192:0/3946331568 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f888814ca90 con 0x7f8888155290 2023-10-06T03:29:12.493 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:12.478+0000 7f886f7fe700 1 -- 172.21.15.192:0/3946331568 <== mon.2 v2:172.21.15.192:3301/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7f8880014830 con 0x7f8888155290 2023-10-06T03:29:12.493 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:12.478+0000 7f886f7fe700 1 -- 172.21.15.192:0/3946331568 <== mon.2 v2:172.21.15.192:3301/0 5 ==== osd_map(9..9 src has 1..9) v4 ==== 1526+0+0 (secure 0 0 0) 0x7f8880013a50 con 0x7f8888155290 2023-10-06T03:29:12.493 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:12.478+0000 7f888cbe2700 1 -- 172.21.15.192:0/3946331568 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8888070ff0 con 0x7f8888155290 2023-10-06T03:29:12.493 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:12.482+0000 7f886f7fe700 1 -- 172.21.15.192:0/3946331568 <== mon.2 v2:172.21.15.192:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7f8880018020 con 0x7f8888155290 2023-10-06T03:29:12.512 INFO:tasks.ceph.mgr.y.smithi110.stderr:2023-10-06T03:29:12.506+0000 7f5523871000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-06T03:29:12.517 INFO:tasks.ceph.mgr.x.smithi192.stderr:2023-10-06T03:29:12.510+0000 7f0930332000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-06T03:29:12.560 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:12.554+0000 7f888cbe2700 1 -- 172.21.15.192:0/3946331568 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_command({"prefix": "osd new", "uuid": "f1ebdfd5-b859-482b-9f97-c18c4a26f445", "id": 7} v 0) v1 -- 0x7f88881568e0 con 0x7f8888155290 2023-10-06T03:29:12.567 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:12.562+0000 7f886f7fe700 1 -- 172.21.15.192:0/3946331568 <== mon.2 v2:172.21.15.192:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "f1ebdfd5-b859-482b-9f97-c18c4a26f445", "id": 7}]=0 v10) v1 ==== 112+0+2 (secure 0 0 0) 0x7f8880014c70 con 0x7f8888155290 2023-10-06T03:29:12.567 INFO:teuthology.orchestra.run.smithi192.stdout:7 2023-10-06T03:29:12.568 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:12.562+0000 7f888cbe2700 1 -- 172.21.15.192:0/3946331568 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8888155290 msgr2=0x7f88880b1ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:12.568 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:12.562+0000 7f888cbe2700 1 --2- 172.21.15.192:0/3946331568 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8888155290 0x7f88880b1ad0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f8880009ec0 tx=0x7f888000b040 comp rx=0 tx=0).stop 2023-10-06T03:29:12.568 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:12.562+0000 7f888cbe2700 1 -- 172.21.15.192:0/3946331568 shutdown_connections 2023-10-06T03:29:12.568 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:12.562+0000 7f888cbe2700 1 --2- 172.21.15.192:0/3946331568 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8888155290 0x7f88880b1ad0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:12.569 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:12.562+0000 7f888cbe2700 1 --2- 172.21.15.192:0/3946331568 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f888806bd00 0x7f88880b1590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:12.569 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:12.562+0000 7f888cbe2700 1 --2- 172.21.15.192:0/3946331568 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f888806a700 0x7f888814c550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:12.569 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:12.562+0000 7f888cbe2700 1 -- 172.21.15.192:0/3946331568 >> 172.21.15.192:0/3946331568 conn(0x7f8888067b70 msgr2=0x7f88880c2540 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:29:12.569 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:12.562+0000 7f888cbe2700 1 -- 172.21.15.192:0/3946331568 shutdown_connections 2023-10-06T03:29:12.569 INFO:teuthology.orchestra.run.smithi192.stderr:2023-10-06T03:29:12.562+0000 7f888cbe2700 1 -- 172.21.15.192:0/3946331568 wait complete. 2023-10-06T03:29:12.580 INFO:tasks.ceph.osd.0:Restarting daemon 2023-10-06T03:29:12.581 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2023-10-06T03:29:12.583 INFO:tasks.ceph.osd.0:Started 2023-10-06T03:29:12.583 INFO:tasks.ceph.osd.1:Restarting daemon 2023-10-06T03:29:12.583 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2023-10-06T03:29:12.585 INFO:tasks.ceph.osd.1:Started 2023-10-06T03:29:12.585 INFO:tasks.ceph.osd.2:Restarting daemon 2023-10-06T03:29:12.585 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2023-10-06T03:29:12.587 INFO:tasks.ceph.osd.2:Started 2023-10-06T03:29:12.587 INFO:tasks.ceph.osd.3:Restarting daemon 2023-10-06T03:29:12.588 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2023-10-06T03:29:12.589 INFO:tasks.ceph.osd.3:Started 2023-10-06T03:29:12.590 INFO:tasks.ceph.osd.4:Restarting daemon 2023-10-06T03:29:12.590 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2023-10-06T03:29:12.592 INFO:tasks.ceph.osd.4:Started 2023-10-06T03:29:12.592 INFO:tasks.ceph.osd.5:Restarting daemon 2023-10-06T03:29:12.592 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2023-10-06T03:29:12.594 INFO:tasks.ceph.osd.5:Started 2023-10-06T03:29:12.594 INFO:tasks.ceph.osd.6:Restarting daemon 2023-10-06T03:29:12.595 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2023-10-06T03:29:12.597 INFO:tasks.ceph.osd.6:Started 2023-10-06T03:29:12.597 INFO:tasks.ceph.osd.7:Restarting daemon 2023-10-06T03:29:12.597 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2023-10-06T03:29:12.599 INFO:tasks.ceph.osd.7:Started 2023-10-06T03:29:12.600 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-06T03:29:12.630 INFO:tasks.ceph.mgr.x.smithi192.stderr:2023-10-06T03:29:12.622+0000 7f0930332000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-06T03:29:12.637 INFO:tasks.ceph.mgr.y.smithi110.stderr:2023-10-06T03:29:12.630+0000 7f5523871000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-06T03:29:12.692 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:12.686+0000 7f19a88f3700 1 Processor -- start 2023-10-06T03:29:12.692 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:12.686+0000 7f19a88f3700 1 -- start start 2023-10-06T03:29:12.693 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:12.686+0000 7f19a88f3700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f19a4159d30 0x7f19a40ca670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:12.693 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:12.686+0000 7f19a88f3700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f19a415a810 0x7f19a406aa60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:12.694 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:12.686+0000 7f19a88f3700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19a40cba20 con 0x7f19a40cb390 2023-10-06T03:29:12.694 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:12.686+0000 7f19a88f3700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f19a406bb90 con 0x7f19a415a810 2023-10-06T03:29:12.694 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:12.686+0000 7f19a88f3700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f19a40cb830 con 0x7f19a4159d30 2023-10-06T03:29:12.694 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:12.686+0000 7f19a2d9d700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f19a415a810 0x7f19a406aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:12.694 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:12.686+0000 7f19a1d9b700 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f19a40cb390 0x7f19a415a100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:35618/0 (socket says 172.21.15.110:35618) 2023-10-06T03:29:12.694 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:12.686+0000 7f19a1d9b700 1 -- 172.21.15.110:0/2898439552 learned_addr learned my addr 172.21.15.110:0/2898439552 (peer_addr_for_me v1:172.21.15.110:0/0) 2023-10-06T03:29:12.694 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:12.686+0000 7f19a259c700 1 --2- 172.21.15.110:0/2898439552 >> v2:172.21.15.192:3300/0 conn(0x7f19a4159d30 0x7f19a40ca670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:12.694 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:12.686+0000 7f19a2d9d700 1 -- 172.21.15.110:0/2898439552 >> v2:172.21.15.192:3300/0 conn(0x7f19a4159d30 msgr2=0x7f19a40ca670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:29:12.694 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:12.686+0000 7f19a2d9d700 1 --2- 172.21.15.110:0/2898439552 >> v2:172.21.15.192:3300/0 conn(0x7f19a4159d30 0x7f19a40ca670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:12.694 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:12.686+0000 7f19a2d9d700 1 -- 172.21.15.110:0/2898439552 >> v1:172.21.15.110:6789/0 conn(0x7f19a40cb390 legacy=0x7f19a415a100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:12.694 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:12.686+0000 7f19a159a700 1 -- 172.21.15.110:0/2898439552 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1023942995 0 0) 0x7f19a40cba20 con 0x7f19a40cb390 2023-10-06T03:29:12.695 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:12.686+0000 7f19a2d9d700 1 -- 172.21.15.110:0/2898439552 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19a40c98d0 con 0x7f19a415a810 2023-10-06T03:29:12.695 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:12.686+0000 7f19a2d9d700 1 --2- 172.21.15.110:0/2898439552 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f19a415a810 0x7f19a406aa60 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f19a40640e0 tx=0x7f199400b430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=15cd80e1765afdb3 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:29:12.696 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:12.686+0000 7f19a159a700 1 -- 172.21.15.110:0/2898439552 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f199400db40 con 0x7f19a415a810 2023-10-06T03:29:12.696 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:12.686+0000 7f19a159a700 1 -- 172.21.15.110:0/2898439552 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f199400dca0 con 0x7f19a415a810 2023-10-06T03:29:12.696 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:12.686+0000 7f19a159a700 1 -- 172.21.15.110:0/2898439552 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19940196b0 con 0x7f19a415a810 2023-10-06T03:29:12.696 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:12.690+0000 7f19a88f3700 1 -- 172.21.15.110:0/2898439552 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f19a415a810 msgr2=0x7f19a406aa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:12.696 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:12.690+0000 7f19a88f3700 1 --2- 172.21.15.110:0/2898439552 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f19a415a810 0x7f19a406aa60 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f19a40640e0 tx=0x7f199400b430 comp rx=0 tx=0).stop 2023-10-06T03:29:12.696 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:12.690+0000 7f19a88f3700 1 -- 172.21.15.110:0/2898439552 shutdown_connections 2023-10-06T03:29:12.696 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:12.690+0000 7f19a88f3700 1 --2- 172.21.15.110:0/2898439552 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f19a415a810 0x7f19a406aa60 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:12.696 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:12.690+0000 7f19a88f3700 1 --2- 172.21.15.110:0/2898439552 >> v2:172.21.15.192:3300/0 conn(0x7f19a4159d30 0x7f19a40ca670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:12.696 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:12.690+0000 7f19a88f3700 1 -- 172.21.15.110:0/2898439552 >> 172.21.15.110:0/2898439552 conn(0x7f19a40cbd00 msgr2=0x7f19a40cc100 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:29:12.697 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:12.690+0000 7f19a88f3700 1 -- 172.21.15.110:0/2898439552 shutdown_connections 2023-10-06T03:29:12.697 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:12.690+0000 7f19a88f3700 1 -- 172.21.15.110:0/2898439552 wait complete. 2023-10-06T03:29:12.697 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:12.690+0000 7f19a88f3700 1 Processor -- start 2023-10-06T03:29:12.697 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:12.690+0000 7f19a88f3700 1 -- start start 2023-10-06T03:29:12.699 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:12.690+0000 7f19a88f3700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f19a40cb390 0x7f19a414c0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:12.699 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:12.690+0000 7f19a88f3700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f19a4159d30 0x7f19a414ccb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:12.700 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:12.690+0000 7f19a88f3700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f19a415a810 0x7f19a413d540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:12.700 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:12.690+0000 7f19a88f3700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19a4164e60 con 0x7f19a4159d30 2023-10-06T03:29:12.700 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:12.690+0000 7f19a88f3700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19a414d6e0 con 0x7f19a415a810 2023-10-06T03:29:12.700 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:12.690+0000 7f19a88f3700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19a414d9c0 con 0x7f19a40cb390 2023-10-06T03:29:12.700 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:12.690+0000 7f19a1d9b700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f19a4159d30 0x7f19a414ccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:12.700 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:12.690+0000 7f19a1d9b700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f19a4159d30 0x7f19a414ccb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:53652/0 (socket says 172.21.15.110:53652) 2023-10-06T03:29:12.700 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:12.690+0000 7f19a1d9b700 1 -- 172.21.15.110:0/2234622350 learned_addr learned my addr 172.21.15.110:0/2234622350 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:29:12.700 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:12.690+0000 7f19a259c700 1 --2- 172.21.15.110:0/2234622350 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f19a40cb390 0x7f19a414c0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:12.700 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:12.690+0000 7f19a2d9d700 1 --2- 172.21.15.110:0/2234622350 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f19a415a810 0x7f19a413d540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:12.700 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:12.690+0000 7f19a1d9b700 1 -- 172.21.15.110:0/2234622350 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f19a40cb390 msgr2=0x7f19a414c0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:12.700 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:12.690+0000 7f19a1d9b700 1 --2- 172.21.15.110:0/2234622350 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f19a40cb390 0x7f19a414c0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:12.701 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:12.690+0000 7f19a1d9b700 1 -- 172.21.15.110:0/2234622350 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f19a415a810 msgr2=0x7f19a413d540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:12.701 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:12.690+0000 7f19a1d9b700 1 --2- 172.21.15.110:0/2234622350 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f19a415a810 0x7f19a413d540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:12.701 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:12.690+0000 7f19a1d9b700 1 -- 172.21.15.110:0/2234622350 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f199400b6d0 con 0x7f19a4159d30 2023-10-06T03:29:12.701 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:12.690+0000 7f19a259c700 1 --2- 172.21.15.110:0/2234622350 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f19a40cb390 0x7f19a414c0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-06T03:29:12.701 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:12.690+0000 7f19a1d9b700 1 --2- 172.21.15.110:0/2234622350 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f19a4159d30 0x7f19a414ccb0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f198c00c7e0 tx=0x7f198c007df0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:29:12.701 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:12.694+0000 7f199a7fc700 1 -- 172.21.15.110:0/2234622350 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f198c005b30 con 0x7f19a4159d30 2023-10-06T03:29:12.701 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:12.694+0000 7f199a7fc700 1 -- 172.21.15.110:0/2234622350 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f198c005c90 con 0x7f19a4159d30 2023-10-06T03:29:12.701 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:12.694+0000 7f19a88f3700 1 -- 172.21.15.110:0/2234622350 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19a414dca0 con 0x7f19a4159d30 2023-10-06T03:29:12.701 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:12.694+0000 7f199a7fc700 1 -- 172.21.15.110:0/2234622350 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f198c007300 con 0x7f19a4159d30 2023-10-06T03:29:12.701 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:12.694+0000 7f19a88f3700 1 -- 172.21.15.110:0/2234622350 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19a40b0f30 con 0x7f19a4159d30 2023-10-06T03:29:12.701 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:12.694+0000 7f199a7fc700 1 -- 172.21.15.110:0/2234622350 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7f198c01c440 con 0x7f19a4159d30 2023-10-06T03:29:12.702 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:12.694+0000 7f199a7fc700 1 -- 172.21.15.110:0/2234622350 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1641+0+0 (secure 0 0 0) 0x7f198c005e00 con 0x7f19a4159d30 2023-10-06T03:29:12.702 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:12.694+0000 7f19a88f3700 1 -- 172.21.15.110:0/2234622350 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19a4164fb0 con 0x7f19a4159d30 2023-10-06T03:29:12.703 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:12.698+0000 7f199a7fc700 1 -- 172.21.15.110:0/2234622350 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7f19a4164fb0 con 0x7f19a4159d30 2023-10-06T03:29:12.769 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:12.762+0000 7f19a88f3700 1 -- 172.21.15.110:0/2234622350 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f19a406ff60 con 0x7f19a4159d30 2023-10-06T03:29:12.770 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:12.762+0000 7f199a7fc700 1 -- 172.21.15.110:0/2234622350 <== mon.0 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7013 (secure 0 0 0) 0x7f198c017070 con 0x7f19a4159d30 2023-10-06T03:29:12.771 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-10-06T03:29:12.771 INFO:teuthology.orchestra.run.smithi110.stdout:{"epoch":10,"fsid":"93b8f6ac-4c92-4b45-b8e4-b59617024aa2","created":"2023-10-06T03:29:10.519583+0000","modified":"2023-10-06T03:29:12.560436+0000","last_up_change":"0.000000","last_in_change":"2023-10-06T03:29:12.560436+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"5866eb6c-2adc-46e4-be7a-e5bcda0f4065","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"a38de118-062b-4a05-9782-2ceddf6ff4df","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"56e39be2-3407-4b69-877e-cf16617b06d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"01c82d13-c1d4-4bde-b55e-3e0089b40115","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"86983430-5fa3-4d17-998f-03e2cd843237","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"6945257f-821d-4619-a63f-ebe857eba6f4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7854d1ae-7060-4780-8f97-73cea5e7a12f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f1ebdfd5-b859-482b-9f97-c18c4a26f445","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-06T03:29:12.772 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:12.766+0000 7f19a88f3700 1 -- 172.21.15.110:0/2234622350 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f19a4159d30 msgr2=0x7f19a414ccb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:12.773 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:12.766+0000 7f19a88f3700 1 --2- 172.21.15.110:0/2234622350 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f19a4159d30 0x7f19a414ccb0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f198c00c7e0 tx=0x7f198c007df0 comp rx=0 tx=0).stop 2023-10-06T03:29:12.773 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:12.766+0000 7f19a88f3700 1 -- 172.21.15.110:0/2234622350 shutdown_connections 2023-10-06T03:29:12.774 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:12.766+0000 7f19a88f3700 1 --2- 172.21.15.110:0/2234622350 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f19a415a810 0x7f19a413d540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:12.774 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:12.766+0000 7f19a88f3700 1 --2- 172.21.15.110:0/2234622350 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f19a4159d30 0x7f19a414ccb0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:12.774 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:12.766+0000 7f19a88f3700 1 --2- 172.21.15.110:0/2234622350 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f19a40cb390 0x7f19a414c0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:12.774 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:12.766+0000 7f19a88f3700 1 -- 172.21.15.110:0/2234622350 >> 172.21.15.110:0/2234622350 conn(0x7f19a40cbd00 msgr2=0x7f19a406a9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:29:12.774 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:12.766+0000 7f19a88f3700 1 -- 172.21.15.110:0/2234622350 shutdown_connections 2023-10-06T03:29:12.775 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:12.766+0000 7f19a88f3700 1 -- 172.21.15.110:0/2234622350 wait complete. 2023-10-06T03:29:12.788 INFO:tasks.ceph.ceph_manager.ceph:[] 2023-10-06T03:29:12.788 INFO:tasks.ceph:Waiting for OSDs to come up 2023-10-06T03:29:12.932 INFO:tasks.ceph.mgr.x.smithi192.stderr:2023-10-06T03:29:12.926+0000 7f0930332000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-06T03:29:12.949 INFO:tasks.ceph.osd.1.smithi110.stderr:2023-10-06T03:29:12.942+0000 7eff1e7b63c0 -1 Falling back to public interface 2023-10-06T03:29:12.955 INFO:tasks.ceph.mgr.y.smithi110.stderr:2023-10-06T03:29:12.950+0000 7f5523871000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-06T03:29:12.964 INFO:tasks.ceph.osd.5.smithi192.stderr:2023-10-06T03:29:12.958+0000 7f4ae2a6d3c0 -1 Falling back to public interface 2023-10-06T03:29:12.964 INFO:tasks.ceph.osd.3.smithi110.stderr:2023-10-06T03:29:12.958+0000 7fc032eff3c0 -1 Falling back to public interface 2023-10-06T03:29:13.089 INFO:tasks.ceph.mgr.x.smithi192.stderr:2023-10-06T03:29:13.082+0000 7f0930332000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-06T03:29:13.112 INFO:tasks.ceph.mgr.y.smithi110.stderr:2023-10-06T03:29:13.106+0000 7f5523871000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-06T03:29:13.154 INFO:tasks.ceph.mgr.x.smithi192.stderr:2023-10-06T03:29:13.150+0000 7f0930332000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-06T03:29:13.177 INFO:tasks.ceph.mgr.y.smithi110.stderr:2023-10-06T03:29:13.170+0000 7f5523871000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-06T03:29:13.206 INFO:tasks.ceph.mgr.x.smithi192.stderr:2023-10-06T03:29:13.198+0000 7f0930332000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-06T03:29:13.217 INFO:tasks.ceph.osd.2.smithi110.stderr:2023-10-06T03:29:13.210+0000 7efd385053c0 -1 Falling back to public interface 2023-10-06T03:29:13.219 INFO:tasks.ceph.osd.7.smithi192.stderr:2023-10-06T03:29:13.214+0000 7f6b5e6183c0 -1 Falling back to public interface 2023-10-06T03:29:13.229 INFO:tasks.ceph.mgr.y.smithi110.stderr:2023-10-06T03:29:13.222+0000 7f5523871000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-06T03:29:13.231 INFO:tasks.ceph.osd.4.smithi192.stderr:2023-10-06T03:29:13.226+0000 7fda11a843c0 -1 Falling back to public interface 2023-10-06T03:29:13.237 INFO:tasks.ceph.osd.0.smithi110.stderr:2023-10-06T03:29:13.230+0000 7f0f23f813c0 -1 Falling back to public interface 2023-10-06T03:29:13.243 INFO:tasks.ceph.osd.6.smithi192.stderr:2023-10-06T03:29:13.238+0000 7fa560b263c0 -1 Falling back to public interface 2023-10-06T03:29:13.260 INFO:tasks.ceph.mgr.x.smithi192.stderr:2023-10-06T03:29:13.254+0000 7f0930332000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-06T03:29:13.288 INFO:tasks.ceph.mgr.y.smithi110.stderr:2023-10-06T03:29:13.282+0000 7f5523871000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-06T03:29:13.565 INFO:tasks.ceph.mgr.y.smithi110.stderr:2023-10-06T03:29:13.558+0000 7f5523871000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-06T03:29:13.566 INFO:tasks.ceph.mgr.x.smithi192.stderr:2023-10-06T03:29:13.558+0000 7f0930332000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-06T03:29:13.594 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-10-06T03:29:13.621 INFO:tasks.ceph.mgr.x.smithi192.stderr:2023-10-06T03:29:13.614+0000 7f0930332000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-06T03:29:13.623 INFO:tasks.ceph.mgr.y.smithi110.stderr:2023-10-06T03:29:13.614+0000 7f5523871000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-06T03:29:13.684 INFO:teuthology.misc.health.smithi110.stderr:2023-10-06T03:29:13.678+0000 7f0159e6e700 1 Processor -- start 2023-10-06T03:29:13.685 INFO:teuthology.misc.health.smithi110.stderr:2023-10-06T03:29:13.678+0000 7f0159e6e700 1 -- start start 2023-10-06T03:29:13.687 INFO:teuthology.misc.health.smithi110.stderr:2023-10-06T03:29:13.678+0000 7f0159e6e700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0154155770 0x7f0154069bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:13.687 INFO:teuthology.misc.health.smithi110.stderr:2023-10-06T03:29:13.678+0000 7f0159e6e700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f01540cc3a0 0x7f0154155b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:13.687 INFO:teuthology.misc.health.smithi110.stderr:2023-10-06T03:29:13.678+0000 7f0159e6e700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01540cb350 con 0x7f0154068cb0 2023-10-06T03:29:13.687 INFO:teuthology.misc.health.smithi110.stderr:2023-10-06T03:29:13.678+0000 7f0159e6e700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0154067f50 con 0x7f01540cc3a0 2023-10-06T03:29:13.687 INFO:teuthology.misc.health.smithi110.stderr:2023-10-06T03:29:13.678+0000 7f0159e6e700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01540c94b0 con 0x7f0154155770 2023-10-06T03:29:13.687 INFO:teuthology.misc.health.smithi110.stderr:2023-10-06T03:29:13.678+0000 7f0153fff700 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f0154068cb0 0x7f01540cbc90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:35622/0 (socket says 172.21.15.110:35622) 2023-10-06T03:29:13.687 INFO:teuthology.misc.health.smithi110.stderr:2023-10-06T03:29:13.678+0000 7f0153fff700 1 -- 172.21.15.110:0/1855794914 learned_addr learned my addr 172.21.15.110:0/1855794914 (peer_addr_for_me v1:172.21.15.110:0/0) 2023-10-06T03:29:13.687 INFO:teuthology.misc.health.smithi110.stderr:2023-10-06T03:29:13.678+0000 7f0158e6c700 1 --2- 172.21.15.110:0/1855794914 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0154155770 0x7f0154069bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:13.687 INFO:teuthology.misc.health.smithi110.stderr:2023-10-06T03:29:13.678+0000 7f015966d700 1 --2- 172.21.15.110:0/1855794914 >> v2:172.21.15.192:3300/0 conn(0x7f01540cc3a0 0x7f0154155b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:13.688 INFO:teuthology.misc.health.smithi110.stderr:2023-10-06T03:29:13.678+0000 7f01537fe700 1 -- 172.21.15.110:0/1855794914 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2288776503 0 0) 0x7f01540cb350 con 0x7f0154068cb0 2023-10-06T03:29:13.688 INFO:teuthology.misc.health.smithi110.stderr:2023-10-06T03:29:13.678+0000 7f01537fe700 1 -- 172.21.15.110:0/1855794914 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0144003570 con 0x7f0154068cb0 2023-10-06T03:29:13.688 INFO:tasks.ceph.mgr.x.smithi192.stderr:2023-10-06T03:29:13.682+0000 7f0930332000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-06T03:29:13.689 INFO:teuthology.misc.health.smithi110.stderr:2023-10-06T03:29:13.682+0000 7f015966d700 1 -- 172.21.15.110:0/1855794914 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0154155770 msgr2=0x7f0154069bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:29:13.690 INFO:teuthology.misc.health.smithi110.stderr:2023-10-06T03:29:13.682+0000 7f015966d700 1 --2- 172.21.15.110:0/1855794914 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0154155770 0x7f0154069bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:13.690 INFO:teuthology.misc.health.smithi110.stderr:2023-10-06T03:29:13.682+0000 7f015966d700 1 -- 172.21.15.110:0/1855794914 >> v1:172.21.15.110:6789/0 conn(0x7f0154068cb0 legacy=0x7f01540cbc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:13.690 INFO:teuthology.misc.health.smithi110.stderr:2023-10-06T03:29:13.682+0000 7f015966d700 1 -- 172.21.15.110:0/1855794914 --> v2:172.21.15.192:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01540ca740 con 0x7f01540cc3a0 2023-10-06T03:29:13.690 INFO:teuthology.misc.health.smithi110.stderr:2023-10-06T03:29:13.682+0000 7f01537fe700 1 -- 172.21.15.110:0/1855794914 <== mon.0 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1753138420 0 0) 0x7f0144003570 con 0x7f0154068cb0 2023-10-06T03:29:13.690 INFO:teuthology.misc.health.smithi110.stderr:2023-10-06T03:29:13.682+0000 7f015966d700 1 --2- 172.21.15.110:0/1855794914 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f01540cc3a0 0x7f0154155b40 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f014c009e60 tx=0x7f014c009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f0e52b88e78979d6 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:29:13.690 INFO:teuthology.misc.health.smithi110.stderr:2023-10-06T03:29:13.682+0000 7f01537fe700 1 -- 172.21.15.110:0/1855794914 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f014c00e070 con 0x7f01540cc3a0 2023-10-06T03:29:13.690 INFO:teuthology.misc.health.smithi110.stderr:2023-10-06T03:29:13.682+0000 7f01537fe700 1 -- 172.21.15.110:0/1855794914 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f014c00b7a0 con 0x7f01540cc3a0 2023-10-06T03:29:13.690 INFO:teuthology.misc.health.smithi110.stderr:2023-10-06T03:29:13.682+0000 7f01537fe700 1 -- 172.21.15.110:0/1855794914 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0140001470 con 0x7f01540cc3a0 2023-10-06T03:29:13.690 INFO:teuthology.misc.health.smithi110.stderr:2023-10-06T03:29:13.682+0000 7f0159e6e700 1 -- 172.21.15.110:0/1855794914 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f01540cc3a0 msgr2=0x7f0154155b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:13.691 INFO:teuthology.misc.health.smithi110.stderr:2023-10-06T03:29:13.682+0000 7f0159e6e700 1 --2- 172.21.15.110:0/1855794914 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f01540cc3a0 0x7f0154155b40 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f014c009e60 tx=0x7f014c009780 comp rx=0 tx=0).stop 2023-10-06T03:29:13.691 INFO:teuthology.misc.health.smithi110.stderr:2023-10-06T03:29:13.682+0000 7f0159e6e700 1 -- 172.21.15.110:0/1855794914 shutdown_connections 2023-10-06T03:29:13.691 INFO:teuthology.misc.health.smithi110.stderr:2023-10-06T03:29:13.682+0000 7f0159e6e700 1 --2- 172.21.15.110:0/1855794914 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f01540cc3a0 0x7f0154155b40 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:13.691 INFO:teuthology.misc.health.smithi110.stderr:2023-10-06T03:29:13.682+0000 7f0159e6e700 1 --2- 172.21.15.110:0/1855794914 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0154155770 0x7f0154069bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:13.691 INFO:teuthology.misc.health.smithi110.stderr:2023-10-06T03:29:13.682+0000 7f0159e6e700 1 -- 172.21.15.110:0/1855794914 >> 172.21.15.110:0/1855794914 conn(0x7f015406ebc0 msgr2=0x7f0154067580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:29:13.691 INFO:teuthology.misc.health.smithi110.stderr:2023-10-06T03:29:13.682+0000 7f0159e6e700 1 -- 172.21.15.110:0/1855794914 shutdown_connections 2023-10-06T03:29:13.691 INFO:teuthology.misc.health.smithi110.stderr:2023-10-06T03:29:13.682+0000 7f0159e6e700 1 -- 172.21.15.110:0/1855794914 wait complete. 2023-10-06T03:29:13.692 INFO:tasks.ceph.mgr.y.smithi110.stderr:2023-10-06T03:29:13.682+0000 7f5523871000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-06T03:29:13.694 INFO:teuthology.misc.health.smithi110.stderr:2023-10-06T03:29:13.682+0000 7f0159e6e700 1 Processor -- start 2023-10-06T03:29:13.694 INFO:teuthology.misc.health.smithi110.stderr:2023-10-06T03:29:13.686+0000 7f0159e6e700 1 -- start start 2023-10-06T03:29:13.694 INFO:teuthology.misc.health.smithi110.stderr:2023-10-06T03:29:13.686+0000 7f0159e6e700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0154068cb0 0x7f01540c3af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:13.694 INFO:teuthology.misc.health.smithi110.stderr:2023-10-06T03:29:13.686+0000 7f0159e6e700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f01540cc3a0 0x7f015414f7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:13.694 INFO:teuthology.misc.health.smithi110.stderr:2023-10-06T03:29:13.686+0000 7f0159e6e700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0154155770 0x7f0154150340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:13.694 INFO:teuthology.misc.health.smithi110.stderr:2023-10-06T03:29:13.686+0000 7f0159e6e700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f015414fce0 con 0x7f01540cc3a0 2023-10-06T03:29:13.695 INFO:teuthology.misc.health.smithi110.stderr:2023-10-06T03:29:13.686+0000 7f0159e6e700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01540b0d70 con 0x7f0154068cb0 2023-10-06T03:29:13.695 INFO:teuthology.misc.health.smithi110.stderr:2023-10-06T03:29:13.686+0000 7f0159e6e700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0154159660 con 0x7f0154155770 2023-10-06T03:29:13.695 INFO:teuthology.misc.health.smithi110.stderr:2023-10-06T03:29:13.686+0000 7f0153fff700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f01540cc3a0 0x7f015414f7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:13.695 INFO:teuthology.misc.health.smithi110.stderr:2023-10-06T03:29:13.686+0000 7f0153fff700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f01540cc3a0 0x7f015414f7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:53662/0 (socket says 172.21.15.110:53662) 2023-10-06T03:29:13.695 INFO:teuthology.misc.health.smithi110.stderr:2023-10-06T03:29:13.686+0000 7f0153fff700 1 -- 172.21.15.110:0/1449325023 learned_addr learned my addr 172.21.15.110:0/1449325023 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:29:13.695 INFO:teuthology.misc.health.smithi110.stderr:2023-10-06T03:29:13.686+0000 7f0158e6c700 1 --2- 172.21.15.110:0/1449325023 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0154068cb0 0x7f01540c3af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:13.695 INFO:teuthology.misc.health.smithi110.stderr:2023-10-06T03:29:13.686+0000 7f015966d700 1 --2- 172.21.15.110:0/1449325023 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0154155770 0x7f0154150340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:13.695 INFO:teuthology.misc.health.smithi110.stderr:2023-10-06T03:29:13.686+0000 7f0153fff700 1 -- 172.21.15.110:0/1449325023 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0154155770 msgr2=0x7f0154150340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:13.695 INFO:teuthology.misc.health.smithi110.stderr:2023-10-06T03:29:13.686+0000 7f0153fff700 1 --2- 172.21.15.110:0/1449325023 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0154155770 0x7f0154150340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:13.695 INFO:teuthology.misc.health.smithi110.stderr:2023-10-06T03:29:13.686+0000 7f0153fff700 1 -- 172.21.15.110:0/1449325023 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0154068cb0 msgr2=0x7f01540c3af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:13.695 INFO:teuthology.misc.health.smithi110.stderr:2023-10-06T03:29:13.686+0000 7f0153fff700 1 --2- 172.21.15.110:0/1449325023 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0154068cb0 0x7f01540c3af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:13.696 INFO:teuthology.misc.health.smithi110.stderr:2023-10-06T03:29:13.686+0000 7f0153fff700 1 -- 172.21.15.110:0/1449325023 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f014c009a20 con 0x7f01540cc3a0 2023-10-06T03:29:13.696 INFO:teuthology.misc.health.smithi110.stderr:2023-10-06T03:29:13.686+0000 7f0153fff700 1 --2- 172.21.15.110:0/1449325023 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f01540cc3a0 0x7f015414f7a0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f0140001660 tx=0x7f0140001a80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:29:13.696 INFO:teuthology.misc.health.smithi110.stderr:2023-10-06T03:29:13.686+0000 7f0150ff9700 1 -- 172.21.15.110:0/1449325023 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0140007150 con 0x7f01540cc3a0 2023-10-06T03:29:13.696 INFO:teuthology.misc.health.smithi110.stderr:2023-10-06T03:29:13.686+0000 7f0150ff9700 1 -- 172.21.15.110:0/1449325023 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01400072b0 con 0x7f01540cc3a0 2023-10-06T03:29:13.696 INFO:teuthology.misc.health.smithi110.stderr:2023-10-06T03:29:13.686+0000 7f0159e6e700 1 -- 172.21.15.110:0/1449325023 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0154159940 con 0x7f01540cc3a0 2023-10-06T03:29:13.696 INFO:teuthology.misc.health.smithi110.stderr:2023-10-06T03:29:13.686+0000 7f0150ff9700 1 -- 172.21.15.110:0/1449325023 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0140007670 con 0x7f01540cc3a0 2023-10-06T03:29:13.697 INFO:teuthology.misc.health.smithi110.stderr:2023-10-06T03:29:13.686+0000 7f0159e6e700 1 -- 172.21.15.110:0/1449325023 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0154159f00 con 0x7f01540cc3a0 2023-10-06T03:29:13.697 INFO:teuthology.misc.health.smithi110.stderr:2023-10-06T03:29:13.686+0000 7f0150ff9700 1 -- 172.21.15.110:0/1449325023 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7f0140004cc0 con 0x7f01540cc3a0 2023-10-06T03:29:13.697 INFO:teuthology.misc.health.smithi110.stderr:2023-10-06T03:29:13.686+0000 7f0150ff9700 1 -- 172.21.15.110:0/1449325023 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1641+0+0 (secure 0 0 0) 0x7f014000ab20 con 0x7f01540cc3a0 2023-10-06T03:29:13.697 INFO:teuthology.misc.health.smithi110.stderr:2023-10-06T03:29:13.686+0000 7f0159e6e700 1 -- 172.21.15.110:0/1449325023 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0154064150 con 0x7f01540cc3a0 2023-10-06T03:29:13.698 INFO:teuthology.misc.health.smithi110.stderr:2023-10-06T03:29:13.690+0000 7f0150ff9700 1 -- 172.21.15.110:0/1449325023 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7f0154064150 con 0x7f01540cc3a0 2023-10-06T03:29:13.764 INFO:teuthology.misc.health.smithi110.stderr:2023-10-06T03:29:13.754+0000 7f0159e6e700 1 -- 172.21.15.110:0/1449325023 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f01540caa20 con 0x7f01540cc3a0 2023-10-06T03:29:13.764 INFO:teuthology.misc.health.smithi110.stderr:2023-10-06T03:29:13.758+0000 7f0150ff9700 1 -- 172.21.15.110:0/1449325023 <== mon.0 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7013 (secure 0 0 0) 0x7f0140018070 con 0x7f01540cc3a0 2023-10-06T03:29:13.765 INFO:teuthology.misc.health.smithi110.stdout: 2023-10-06T03:29:13.765 INFO:teuthology.misc.health.smithi110.stdout:{"epoch":10,"fsid":"93b8f6ac-4c92-4b45-b8e4-b59617024aa2","created":"2023-10-06T03:29:10.519583+0000","modified":"2023-10-06T03:29:12.560436+0000","last_up_change":"0.000000","last_in_change":"2023-10-06T03:29:12.560436+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"5866eb6c-2adc-46e4-be7a-e5bcda0f4065","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"a38de118-062b-4a05-9782-2ceddf6ff4df","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"56e39be2-3407-4b69-877e-cf16617b06d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"01c82d13-c1d4-4bde-b55e-3e0089b40115","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"86983430-5fa3-4d17-998f-03e2cd843237","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"6945257f-821d-4619-a63f-ebe857eba6f4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7854d1ae-7060-4780-8f97-73cea5e7a12f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f1ebdfd5-b859-482b-9f97-c18c4a26f445","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-06T03:29:13.765 INFO:teuthology.misc.health.smithi110.stderr:2023-10-06T03:29:13.758+0000 7f0159e6e700 1 -- 172.21.15.110:0/1449325023 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f01540cc3a0 msgr2=0x7f015414f7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:13.766 INFO:teuthology.misc.health.smithi110.stderr:2023-10-06T03:29:13.758+0000 7f0159e6e700 1 --2- 172.21.15.110:0/1449325023 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f01540cc3a0 0x7f015414f7a0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f0140001660 tx=0x7f0140001a80 comp rx=0 tx=0).stop 2023-10-06T03:29:13.766 INFO:teuthology.misc.health.smithi110.stderr:2023-10-06T03:29:13.758+0000 7f0159e6e700 1 -- 172.21.15.110:0/1449325023 shutdown_connections 2023-10-06T03:29:13.766 INFO:teuthology.misc.health.smithi110.stderr:2023-10-06T03:29:13.758+0000 7f0159e6e700 1 --2- 172.21.15.110:0/1449325023 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0154155770 0x7f0154150340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:13.766 INFO:teuthology.misc.health.smithi110.stderr:2023-10-06T03:29:13.758+0000 7f0159e6e700 1 --2- 172.21.15.110:0/1449325023 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f01540cc3a0 0x7f015414f7a0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:13.766 INFO:teuthology.misc.health.smithi110.stderr:2023-10-06T03:29:13.758+0000 7f0159e6e700 1 --2- 172.21.15.110:0/1449325023 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0154068cb0 0x7f01540c3af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:13.766 INFO:teuthology.misc.health.smithi110.stderr:2023-10-06T03:29:13.758+0000 7f0159e6e700 1 -- 172.21.15.110:0/1449325023 >> 172.21.15.110:0/1449325023 conn(0x7f015406ebc0 msgr2=0x7f0154158860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:29:13.766 INFO:teuthology.misc.health.smithi110.stderr:2023-10-06T03:29:13.758+0000 7f0159e6e700 1 -- 172.21.15.110:0/1449325023 shutdown_connections 2023-10-06T03:29:13.766 INFO:teuthology.misc.health.smithi110.stderr:2023-10-06T03:29:13.758+0000 7f0159e6e700 1 -- 172.21.15.110:0/1449325023 wait complete. 2023-10-06T03:29:13.779 DEBUG:teuthology.misc:0 of 8 OSDs are up 2023-10-06T03:29:13.846 INFO:tasks.ceph.mgr.x.smithi192.stderr:2023-10-06T03:29:13.842+0000 7f0930332000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-06T03:29:13.850 INFO:tasks.ceph.mgr.y.smithi110.stderr:2023-10-06T03:29:13.842+0000 7f5523871000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-06T03:29:13.913 INFO:tasks.ceph.mgr.x.smithi192.stderr:2023-10-06T03:29:13.906+0000 7f0930332000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-06T03:29:13.916 INFO:tasks.ceph.mgr.y.smithi110.stderr:2023-10-06T03:29:13.910+0000 7f5523871000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-06T03:29:14.613 INFO:tasks.ceph.mgr.y.smithi110.stderr:2023-10-06T03:29:14.606+0000 7f5523871000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-06T03:29:14.620 INFO:tasks.ceph.mgr.x.smithi192.stderr:2023-10-06T03:29:14.614+0000 7f0930332000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-06T03:29:15.656 INFO:tasks.ceph.mgr.y.smithi110.stderr:2023-10-06T03:29:15.650+0000 7f550c969700 -1 mgr.server handle_report got status from non-daemon mon.c 2023-10-06T03:29:15.658 INFO:tasks.ceph.mgr.y.smithi110.stderr:2023-10-06T03:29:15.650+0000 7f550c969700 -1 mgr.server handle_report got status from non-daemon mon.b 2023-10-06T03:29:16.489 INFO:tasks.ceph.osd.1.smithi110.stderr:2023-10-06T03:29:16.482+0000 7eff1e7b63c0 -1 osd.1 0 log_to_monitors true 2023-10-06T03:29:16.635 INFO:tasks.ceph.osd.3.smithi110.stderr:2023-10-06T03:29:16.630+0000 7fc032eff3c0 -1 osd.3 0 log_to_monitors true 2023-10-06T03:29:16.796 INFO:tasks.ceph.osd.5.smithi192.stderr:2023-10-06T03:29:16.790+0000 7f4ae2a6d3c0 -1 osd.5 0 log_to_monitors true 2023-10-06T03:29:16.868 INFO:tasks.ceph.osd.6.smithi192.stderr:2023-10-06T03:29:16.862+0000 7fa560b263c0 -1 osd.6 0 log_to_monitors true 2023-10-06T03:29:17.001 INFO:tasks.ceph.osd.4.smithi192.stderr:2023-10-06T03:29:16.994+0000 7fda11a843c0 -1 osd.4 0 log_to_monitors true 2023-10-06T03:29:17.034 INFO:tasks.ceph.osd.0.smithi110.stderr:2023-10-06T03:29:17.026+0000 7f0f23f813c0 -1 osd.0 0 log_to_monitors true 2023-10-06T03:29:17.041 INFO:tasks.ceph.osd.2.smithi110.stderr:2023-10-06T03:29:17.034+0000 7efd385053c0 -1 osd.2 0 log_to_monitors true 2023-10-06T03:29:17.148 INFO:tasks.ceph.osd.7.smithi192.stderr:2023-10-06T03:29:17.142+0000 7f6b5e6183c0 -1 osd.7 0 log_to_monitors true 2023-10-06T03:29:18.693 INFO:tasks.ceph.osd.3.smithi110.stderr:2023-10-06T03:29:18.686+0000 7fc02e69f700 -1 osd.3 0 waiting for initial osdmap 2023-10-06T03:29:18.705 INFO:tasks.ceph.osd.1.smithi110.stderr:2023-10-06T03:29:18.698+0000 7eff1b77b700 -1 osd.1 0 waiting for initial osdmap 2023-10-06T03:29:18.706 INFO:tasks.ceph.osd.3.smithi110.stderr:2023-10-06T03:29:18.698+0000 7fc027443700 -1 osd.3 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-06T03:29:18.813 INFO:tasks.ceph.osd.1.smithi110.stderr:2023-10-06T03:29:18.806+0000 7eff12cfa700 -1 osd.1 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-06T03:29:19.678 INFO:tasks.ceph.osd.2.smithi110.stderr:2023-10-06T03:29:19.670+0000 7efd33ca5700 -1 osd.2 0 waiting for initial osdmap 2023-10-06T03:29:19.685 INFO:tasks.ceph.osd.2.smithi110.stderr:2023-10-06T03:29:19.678+0000 7efd2ca49700 -1 osd.2 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-06T03:29:19.703 INFO:tasks.ceph.osd.0.smithi110.stderr:2023-10-06T03:29:19.698+0000 7f0f1f721700 -1 osd.0 0 waiting for initial osdmap 2023-10-06T03:29:19.754 INFO:tasks.ceph.osd.0.smithi110.stderr:2023-10-06T03:29:19.746+0000 7f0f184c5700 -1 osd.0 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-06T03:29:20.585 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-10-06T03:29:20.688 INFO:teuthology.misc.health.smithi110.stderr:2023-10-06T03:29:20.682+0000 7fe193ee7700 1 Processor -- start 2023-10-06T03:29:20.689 INFO:teuthology.misc.health.smithi110.stderr:2023-10-06T03:29:20.682+0000 7fe193ee7700 1 -- start start 2023-10-06T03:29:20.689 INFO:teuthology.misc.health.smithi110.stderr:2023-10-06T03:29:20.682+0000 7fe193ee7700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fe18c0c8060 0x7fe18c157330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:20.689 INFO:teuthology.misc.health.smithi110.stderr:2023-10-06T03:29:20.682+0000 7fe193ee7700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe18c0c7950 0x7fe18c06cfc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:20.689 INFO:teuthology.misc.health.smithi110.stderr:2023-10-06T03:29:20.682+0000 7fe193ee7700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe18c157870 con 0x7fe18c0c8430 2023-10-06T03:29:20.689 INFO:teuthology.misc.health.smithi110.stderr:2023-10-06T03:29:20.682+0000 7fe193ee7700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe18c0cd740 con 0x7fe18c0c8060 2023-10-06T03:29:20.689 INFO:teuthology.misc.health.smithi110.stderr:2023-10-06T03:29:20.682+0000 7fe193ee7700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe18c157ac0 con 0x7fe18c0c7950 2023-10-06T03:29:20.691 INFO:teuthology.misc.health.smithi110.stderr:2023-10-06T03:29:20.682+0000 7fe191c83700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fe18c0c8060 0x7fe18c157330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:20.691 INFO:teuthology.misc.health.smithi110.stderr:2023-10-06T03:29:20.682+0000 7fe191c83700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fe18c0c8060 0x7fe18c157330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.110:37604/0 (socket says 172.21.15.110:37604) 2023-10-06T03:29:20.691 INFO:teuthology.misc.health.smithi110.stderr:2023-10-06T03:29:20.682+0000 7fe191c83700 1 -- 172.21.15.110:0/1529648594 learned_addr learned my addr 172.21.15.110:0/1529648594 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:29:20.691 INFO:teuthology.misc.health.smithi110.stderr:2023-10-06T03:29:20.682+0000 7fe192484700 1 --1- 172.21.15.110:0/1529648594 >> v1:172.21.15.192:6789/0 conn(0x7fe18c0c8430 0x7fe18c15a900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.110:45374/0 (socket says 172.21.15.110:45374) 2023-10-06T03:29:20.691 INFO:teuthology.misc.health.smithi110.stderr:2023-10-06T03:29:20.682+0000 7fe191482700 1 --2- 172.21.15.110:0/1529648594 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe18c0c7950 0x7fe18c06cfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:20.691 INFO:teuthology.misc.health.smithi110.stderr:2023-10-06T03:29:20.682+0000 7fe190c81700 1 -- 172.21.15.110:0/1529648594 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2134766735 0 0) 0x7fe18c157870 con 0x7fe18c0c8430 2023-10-06T03:29:20.691 INFO:teuthology.misc.health.smithi110.stderr:2023-10-06T03:29:20.682+0000 7fe190c81700 1 -- 172.21.15.110:0/1529648594 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe17c003570 con 0x7fe18c0c8430 2023-10-06T03:29:20.691 INFO:teuthology.misc.health.smithi110.stderr:2023-10-06T03:29:20.682+0000 7fe191482700 1 -- 172.21.15.110:0/1529648594 >> v2:172.21.15.192:3300/0 conn(0x7fe18c0c8060 msgr2=0x7fe18c157330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:29:20.691 INFO:teuthology.misc.health.smithi110.stderr:2023-10-06T03:29:20.682+0000 7fe191482700 1 --2- 172.21.15.110:0/1529648594 >> v2:172.21.15.192:3300/0 conn(0x7fe18c0c8060 0x7fe18c157330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:20.692 INFO:teuthology.misc.health.smithi110.stderr:2023-10-06T03:29:20.682+0000 7fe191482700 1 -- 172.21.15.110:0/1529648594 >> v1:172.21.15.192:6789/0 conn(0x7fe18c0c8430 legacy=0x7fe18c15a900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:20.692 INFO:teuthology.misc.health.smithi110.stderr:2023-10-06T03:29:20.682+0000 7fe191482700 1 -- 172.21.15.110:0/1529648594 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe18c157da0 con 0x7fe18c0c7950 2023-10-06T03:29:20.692 INFO:teuthology.misc.health.smithi110.stderr:2023-10-06T03:29:20.686+0000 7fe191482700 1 --2- 172.21.15.110:0/1529648594 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe18c0c7950 0x7fe18c06cfc0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fe1780098e0 tx=0x7fe1780091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a352aeda85630f61 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:29:20.693 INFO:teuthology.misc.health.smithi110.stderr:2023-10-06T03:29:20.686+0000 7fe190c81700 1 -- 172.21.15.110:0/1529648594 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe178011630 con 0x7fe18c0c7950 2023-10-06T03:29:20.693 INFO:teuthology.misc.health.smithi110.stderr:2023-10-06T03:29:20.686+0000 7fe190c81700 1 -- 172.21.15.110:0/1529648594 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe178011790 con 0x7fe18c0c7950 2023-10-06T03:29:20.693 INFO:teuthology.misc.health.smithi110.stderr:2023-10-06T03:29:20.686+0000 7fe190c81700 1 -- 172.21.15.110:0/1529648594 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe178011b30 con 0x7fe18c0c7950 2023-10-06T03:29:20.693 INFO:teuthology.misc.health.smithi110.stderr:2023-10-06T03:29:20.686+0000 7fe193ee7700 1 -- 172.21.15.110:0/1529648594 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe18c0c7950 msgr2=0x7fe18c06cfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:20.693 INFO:teuthology.misc.health.smithi110.stderr:2023-10-06T03:29:20.686+0000 7fe193ee7700 1 --2- 172.21.15.110:0/1529648594 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe18c0c7950 0x7fe18c06cfc0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fe1780098e0 tx=0x7fe1780091a0 comp rx=0 tx=0).stop 2023-10-06T03:29:20.693 INFO:teuthology.misc.health.smithi110.stderr:2023-10-06T03:29:20.686+0000 7fe193ee7700 1 -- 172.21.15.110:0/1529648594 shutdown_connections 2023-10-06T03:29:20.693 INFO:teuthology.misc.health.smithi110.stderr:2023-10-06T03:29:20.686+0000 7fe193ee7700 1 --2- 172.21.15.110:0/1529648594 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe18c0c7950 0x7fe18c06cfc0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:20.693 INFO:teuthology.misc.health.smithi110.stderr:2023-10-06T03:29:20.686+0000 7fe193ee7700 1 --2- 172.21.15.110:0/1529648594 >> v2:172.21.15.192:3300/0 conn(0x7fe18c0c8060 0x7fe18c157330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:20.693 INFO:teuthology.misc.health.smithi110.stderr:2023-10-06T03:29:20.686+0000 7fe193ee7700 1 -- 172.21.15.110:0/1529648594 >> 172.21.15.110:0/1529648594 conn(0x7fe18c06c0e0 msgr2=0x7fe18c0ccc80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:29:20.694 INFO:teuthology.misc.health.smithi110.stderr:2023-10-06T03:29:20.686+0000 7fe193ee7700 1 -- 172.21.15.110:0/1529648594 shutdown_connections 2023-10-06T03:29:20.694 INFO:teuthology.misc.health.smithi110.stderr:2023-10-06T03:29:20.686+0000 7fe193ee7700 1 -- 172.21.15.110:0/1529648594 wait complete. 2023-10-06T03:29:20.695 INFO:teuthology.misc.health.smithi110.stderr:2023-10-06T03:29:20.686+0000 7fe193ee7700 1 Processor -- start 2023-10-06T03:29:20.695 INFO:teuthology.misc.health.smithi110.stderr:2023-10-06T03:29:20.686+0000 7fe193ee7700 1 -- start start 2023-10-06T03:29:20.695 INFO:teuthology.misc.health.smithi110.stderr:2023-10-06T03:29:20.686+0000 7fe193ee7700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe18c0c7950 0x7fe18c142670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:20.695 INFO:teuthology.misc.health.smithi110.stderr:2023-10-06T03:29:20.686+0000 7fe193ee7700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe18c0c8060 0x7fe18c0ad9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:20.696 INFO:teuthology.misc.health.smithi110.stderr:2023-10-06T03:29:20.686+0000 7fe193ee7700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe18c0c8430 0x7fe18c0adf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:20.696 INFO:teuthology.misc.health.smithi110.stderr:2023-10-06T03:29:20.686+0000 7fe193ee7700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe18c0c6600 con 0x7fe18c0c8060 2023-10-06T03:29:20.696 INFO:teuthology.misc.health.smithi110.stderr:2023-10-06T03:29:20.686+0000 7fe193ee7700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe18c155e10 con 0x7fe18c0c8430 2023-10-06T03:29:20.696 INFO:teuthology.misc.health.smithi110.stderr:2023-10-06T03:29:20.686+0000 7fe193ee7700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe18c154f20 con 0x7fe18c0c7950 2023-10-06T03:29:20.696 INFO:teuthology.misc.health.smithi110.stderr:2023-10-06T03:29:20.686+0000 7fe191482700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe18c0c8060 0x7fe18c0ad9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:20.696 INFO:teuthology.misc.health.smithi110.stderr:2023-10-06T03:29:20.686+0000 7fe192484700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe18c0c8430 0x7fe18c0adf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:20.696 INFO:teuthology.misc.health.smithi110.stderr:2023-10-06T03:29:20.690+0000 7fe191482700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe18c0c8060 0x7fe18c0ad9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:53786/0 (socket says 172.21.15.110:53786) 2023-10-06T03:29:20.696 INFO:teuthology.misc.health.smithi110.stderr:2023-10-06T03:29:20.690+0000 7fe192484700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe18c0c8430 0x7fe18c0adf00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.110:37608/0 (socket says 172.21.15.110:37608) 2023-10-06T03:29:20.696 INFO:teuthology.misc.health.smithi110.stderr:2023-10-06T03:29:20.690+0000 7fe191482700 1 -- 172.21.15.110:0/2529837798 learned_addr learned my addr 172.21.15.110:0/2529837798 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:29:20.696 INFO:teuthology.misc.health.smithi110.stderr:2023-10-06T03:29:20.690+0000 7fe191c83700 1 --2- 172.21.15.110:0/2529837798 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe18c0c7950 0x7fe18c142670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:20.696 INFO:teuthology.misc.health.smithi110.stderr:2023-10-06T03:29:20.690+0000 7fe192484700 1 -- 172.21.15.110:0/2529837798 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe18c0c7950 msgr2=0x7fe18c142670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:20.697 INFO:teuthology.misc.health.smithi110.stderr:2023-10-06T03:29:20.690+0000 7fe192484700 1 --2- 172.21.15.110:0/2529837798 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe18c0c7950 0x7fe18c142670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:20.697 INFO:teuthology.misc.health.smithi110.stderr:2023-10-06T03:29:20.690+0000 7fe192484700 1 -- 172.21.15.110:0/2529837798 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe18c0c8060 msgr2=0x7fe18c0ad9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:20.697 INFO:teuthology.misc.health.smithi110.stderr:2023-10-06T03:29:20.690+0000 7fe192484700 1 --2- 172.21.15.110:0/2529837798 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe18c0c8060 0x7fe18c0ad9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:20.697 INFO:teuthology.misc.health.smithi110.stderr:2023-10-06T03:29:20.690+0000 7fe192484700 1 -- 172.21.15.110:0/2529837798 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe178009420 con 0x7fe18c0c8430 2023-10-06T03:29:20.698 INFO:teuthology.misc.health.smithi110.stderr:2023-10-06T03:29:20.690+0000 7fe192484700 1 --2- 172.21.15.110:0/2529837798 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe18c0c8430 0x7fe18c0adf00 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fe18400c4a0 tx=0x7fe184007cd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:29:20.698 INFO:teuthology.misc.health.smithi110.stderr:2023-10-06T03:29:20.690+0000 7fe189ffb700 1 -- 172.21.15.110:0/2529837798 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1840070d0 con 0x7fe18c0c8430 2023-10-06T03:29:20.698 INFO:teuthology.misc.health.smithi110.stderr:2023-10-06T03:29:20.690+0000 7fe189ffb700 1 -- 172.21.15.110:0/2529837798 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe184007230 con 0x7fe18c0c8430 2023-10-06T03:29:20.698 INFO:teuthology.misc.health.smithi110.stderr:2023-10-06T03:29:20.690+0000 7fe193ee7700 1 -- 172.21.15.110:0/2529837798 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe18c155200 con 0x7fe18c0c8430 2023-10-06T03:29:20.698 INFO:teuthology.misc.health.smithi110.stderr:2023-10-06T03:29:20.690+0000 7fe189ffb700 1 -- 172.21.15.110:0/2529837798 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1840075b0 con 0x7fe18c0c8430 2023-10-06T03:29:20.698 INFO:teuthology.misc.health.smithi110.stderr:2023-10-06T03:29:20.690+0000 7fe193ee7700 1 -- 172.21.15.110:0/2529837798 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe18c1557c0 con 0x7fe18c0c8430 2023-10-06T03:29:20.699 INFO:teuthology.misc.health.smithi110.stderr:2023-10-06T03:29:20.690+0000 7fe189ffb700 1 -- 172.21.15.110:0/2529837798 <== mon.1 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 4) v1 ==== 79903+0+0 (secure 0 0 0) 0x7fe184004a40 con 0x7fe18c0c8430 2023-10-06T03:29:20.699 INFO:teuthology.misc.health.smithi110.stderr:2023-10-06T03:29:20.690+0000 7fe193ee7700 1 -- 172.21.15.110:0/2529837798 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe18c155fc0 con 0x7fe18c0c8430 2023-10-06T03:29:20.699 INFO:teuthology.misc.health.smithi110.stderr:2023-10-06T03:29:20.690+0000 7fe189ffb700 1 --2- 172.21.15.110:0/2529837798 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fe1700604c0 0x7fe170062970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:20.699 INFO:teuthology.misc.health.smithi110.stderr:2023-10-06T03:29:20.690+0000 7fe189ffb700 1 -- 172.21.15.110:0/2529837798 <== mon.1 v2:172.21.15.192:3300/0 5 ==== osd_map(14..14 src has 1..14) v4 ==== 3067+0+0 (secure 0 0 0) 0x7fe18407b3c0 con 0x7fe18c0c8430 2023-10-06T03:29:20.699 INFO:teuthology.misc.health.smithi110.stderr:2023-10-06T03:29:20.690+0000 7fe191c83700 1 --2- 172.21.15.110:0/2529837798 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fe1700604c0 0x7fe170062970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:20.699 INFO:teuthology.misc.health.smithi110.stderr:2023-10-06T03:29:20.694+0000 7fe191c83700 1 --2- 172.21.15.110:0/2529837798 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fe1700604c0 0x7fe170062970 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fe18000f570 tx=0x7fe18000b040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:29:20.705 INFO:teuthology.misc.health.smithi110.stderr:2023-10-06T03:29:20.698+0000 7fe189ffb700 1 -- 172.21.15.110:0/2529837798 <== mon.1 v2:172.21.15.192:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fe18c155fc0 con 0x7fe18c0c8430 2023-10-06T03:29:20.792 INFO:tasks.ceph.osd.7.smithi192.stderr:2023-10-06T03:29:20.786+0000 7f6b59db8700 -1 osd.7 0 waiting for initial osdmap 2023-10-06T03:29:20.802 INFO:tasks.ceph.osd.7.smithi192.stderr:2023-10-06T03:29:20.798+0000 7f6b52b5c700 -1 osd.7 14 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-06T03:29:20.867 INFO:teuthology.misc.health.smithi110.stderr:2023-10-06T03:29:20.862+0000 7fe193ee7700 1 -- 172.21.15.110:0/2529837798 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe18c06ffe0 con 0x7fe18c0c8430 2023-10-06T03:29:20.869 INFO:teuthology.misc.health.smithi110.stderr:2023-10-06T03:29:20.862+0000 7fe189ffb700 1 -- 172.21.15.110:0/2529837798 <== mon.1 v2:172.21.15.192:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v14) v1 ==== 74+0+8314 (secure 0 0 0) 0x7fe18404c450 con 0x7fe18c0c8430 2023-10-06T03:29:20.869 INFO:teuthology.misc.health.smithi110.stdout: 2023-10-06T03:29:20.869 INFO:teuthology.misc.health.smithi110.stdout:{"epoch":14,"fsid":"93b8f6ac-4c92-4b45-b8e4-b59617024aa2","created":"2023-10-06T03:29:10.519583+0000","modified":"2023-10-06T03:29:19.687077+0000","last_up_change":"2023-10-06T03:29:19.687077+0000","last_in_change":"2023-10-06T03:29:12.560436+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"5866eb6c-2adc-46e4-be7a-e5bcda0f4065","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"a38de118-062b-4a05-9782-2ceddf6ff4df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6800","nonce":33396},{"type":"v1","addr":"172.21.15.110:6801","nonce":33396}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6802","nonce":33396},{"type":"v1","addr":"172.21.15.110:6803","nonce":33396}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6806","nonce":33396},{"type":"v1","addr":"172.21.15.110:6807","nonce":33396}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6804","nonce":33396},{"type":"v1","addr":"172.21.15.110:6805","nonce":33396}]},"public_addr":"172.21.15.110:6801/33396","cluster_addr":"172.21.15.110:6803/33396","heartbeat_back_addr":"172.21.15.110:6807/33396","heartbeat_front_addr":"172.21.15.110:6805/33396","state":["exists","up"]},{"osd":2,"uuid":"56e39be2-3407-4b69-877e-cf16617b06d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6816","nonce":33397},{"type":"v1","addr":"172.21.15.110:6817","nonce":33397}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6818","nonce":33397},{"type":"v1","addr":"172.21.15.110:6819","nonce":33397}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6822","nonce":33397},{"type":"v1","addr":"172.21.15.110:6823","nonce":33397}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6820","nonce":33397},{"type":"v1","addr":"172.21.15.110:6821","nonce":33397}]},"public_addr":"172.21.15.110:6817/33397","cluster_addr":"172.21.15.110:6819/33397","heartbeat_back_addr":"172.21.15.110:6823/33397","heartbeat_front_addr":"172.21.15.110:6821/33397","state":["exists","up"]},{"osd":3,"uuid":"01c82d13-c1d4-4bde-b55e-3e0089b40115","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6808","nonce":33395},{"type":"v1","addr":"172.21.15.110:6809","nonce":33395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6810","nonce":33395},{"type":"v1","addr":"172.21.15.110:6811","nonce":33395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6814","nonce":33395},{"type":"v1","addr":"172.21.15.110:6815","nonce":33395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6812","nonce":33395},{"type":"v1","addr":"172.21.15.110:6813","nonce":33395}]},"public_addr":"172.21.15.110:6809/33395","cluster_addr":"172.21.15.110:6811/33395","heartbeat_back_addr":"172.21.15.110:6815/33395","heartbeat_front_addr":"172.21.15.110:6813/33395","state":["exists","up"]},{"osd":4,"uuid":"86983430-5fa3-4d17-998f-03e2cd843237","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"6945257f-821d-4619-a63f-ebe857eba6f4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7854d1ae-7060-4780-8f97-73cea5e7a12f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f1ebdfd5-b859-482b-9f97-c18c4a26f445","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-06T03:29:20.871 INFO:teuthology.misc.health.smithi110.stderr:2023-10-06T03:29:20.866+0000 7fe193ee7700 1 -- 172.21.15.110:0/2529837798 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fe1700604c0 msgr2=0x7fe170062970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:20.871 INFO:teuthology.misc.health.smithi110.stderr:2023-10-06T03:29:20.866+0000 7fe193ee7700 1 --2- 172.21.15.110:0/2529837798 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fe1700604c0 0x7fe170062970 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fe18000f570 tx=0x7fe18000b040 comp rx=0 tx=0).stop 2023-10-06T03:29:20.871 INFO:teuthology.misc.health.smithi110.stderr:2023-10-06T03:29:20.866+0000 7fe193ee7700 1 -- 172.21.15.110:0/2529837798 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe18c0c8430 msgr2=0x7fe18c0adf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:20.872 INFO:teuthology.misc.health.smithi110.stderr:2023-10-06T03:29:20.866+0000 7fe193ee7700 1 --2- 172.21.15.110:0/2529837798 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe18c0c8430 0x7fe18c0adf00 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fe18400c4a0 tx=0x7fe184007cd0 comp rx=0 tx=0).stop 2023-10-06T03:29:20.872 INFO:teuthology.misc.health.smithi110.stderr:2023-10-06T03:29:20.866+0000 7fe193ee7700 1 -- 172.21.15.110:0/2529837798 shutdown_connections 2023-10-06T03:29:20.872 INFO:teuthology.misc.health.smithi110.stderr:2023-10-06T03:29:20.866+0000 7fe193ee7700 1 --2- 172.21.15.110:0/2529837798 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fe1700604c0 0x7fe170062970 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:20.872 INFO:teuthology.misc.health.smithi110.stderr:2023-10-06T03:29:20.866+0000 7fe193ee7700 1 --2- 172.21.15.110:0/2529837798 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe18c0c8430 0x7fe18c0adf00 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:20.872 INFO:teuthology.misc.health.smithi110.stderr:2023-10-06T03:29:20.866+0000 7fe193ee7700 1 --2- 172.21.15.110:0/2529837798 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe18c0c8060 0x7fe18c0ad9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:20.872 INFO:teuthology.misc.health.smithi110.stderr:2023-10-06T03:29:20.866+0000 7fe193ee7700 1 --2- 172.21.15.110:0/2529837798 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe18c0c7950 0x7fe18c142670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:20.872 INFO:teuthology.misc.health.smithi110.stderr:2023-10-06T03:29:20.866+0000 7fe193ee7700 1 -- 172.21.15.110:0/2529837798 >> 172.21.15.110:0/2529837798 conn(0x7fe18c06c0e0 msgr2=0x7fe18c0cbe30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:29:20.872 INFO:teuthology.misc.health.smithi110.stderr:2023-10-06T03:29:20.866+0000 7fe193ee7700 1 -- 172.21.15.110:0/2529837798 shutdown_connections 2023-10-06T03:29:20.872 INFO:teuthology.misc.health.smithi110.stderr:2023-10-06T03:29:20.866+0000 7fe193ee7700 1 -- 172.21.15.110:0/2529837798 wait complete. 2023-10-06T03:29:20.885 DEBUG:teuthology.misc:3 of 8 OSDs are up 2023-10-06T03:29:20.950 INFO:tasks.ceph.osd.4.smithi192.stderr:2023-10-06T03:29:20.942+0000 7fda0d224700 -1 osd.4 0 waiting for initial osdmap 2023-10-06T03:29:20.959 INFO:tasks.ceph.osd.4.smithi192.stderr:2023-10-06T03:29:20.954+0000 7fda05fc8700 -1 osd.4 14 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-06T03:29:20.978 INFO:tasks.ceph.osd.6.smithi192.stderr:2023-10-06T03:29:20.974+0000 7fa55c2c6700 -1 osd.6 0 waiting for initial osdmap 2023-10-06T03:29:20.984 INFO:tasks.ceph.osd.6.smithi192.stderr:2023-10-06T03:29:20.978+0000 7fa55506a700 -1 osd.6 15 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-06T03:29:21.001 INFO:tasks.ceph.osd.5.smithi192.stderr:2023-10-06T03:29:20.994+0000 7f4ade20d700 -1 osd.5 0 waiting for initial osdmap 2023-10-06T03:29:21.009 INFO:tasks.ceph.osd.5.smithi192.stderr:2023-10-06T03:29:21.002+0000 7f4ad6fb1700 -1 osd.5 15 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-06T03:29:27.692 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-10-06T03:29:27.795 INFO:teuthology.misc.health.smithi110.stderr:2023-10-06T03:29:27.786+0000 7f29f9791700 1 Processor -- start 2023-10-06T03:29:27.795 INFO:teuthology.misc.health.smithi110.stderr:2023-10-06T03:29:27.786+0000 7f29f9791700 1 -- start start 2023-10-06T03:29:27.799 INFO:teuthology.misc.health.smithi110.stderr:2023-10-06T03:29:27.790+0000 7f29f9791700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f29f40caa60 0x7f29f40b0c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:27.799 INFO:teuthology.misc.health.smithi110.stderr:2023-10-06T03:29:27.790+0000 7f29f9791700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f29f40c65c0 0x7f29f4159620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:27.799 INFO:teuthology.misc.health.smithi110.stderr:2023-10-06T03:29:27.790+0000 7f29f9791700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29f406da70 con 0x7f29f4155b00 2023-10-06T03:29:27.799 INFO:teuthology.misc.health.smithi110.stderr:2023-10-06T03:29:27.790+0000 7f29f9791700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f29f40c2730 con 0x7f29f40caa60 2023-10-06T03:29:27.799 INFO:teuthology.misc.health.smithi110.stderr:2023-10-06T03:29:27.790+0000 7f29f9791700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29f406dfa0 con 0x7f29f40c65c0 2023-10-06T03:29:27.799 INFO:teuthology.misc.health.smithi110.stderr:2023-10-06T03:29:27.790+0000 7f29f3fff700 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f29f4155b00 0x7f29f40c5eb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:52512/0 (socket says 172.21.15.110:52512) 2023-10-06T03:29:27.799 INFO:teuthology.misc.health.smithi110.stderr:2023-10-06T03:29:27.790+0000 7f29f3fff700 1 -- 172.21.15.110:0/400177367 learned_addr learned my addr 172.21.15.110:0/400177367 (peer_addr_for_me v1:172.21.15.110:0/0) 2023-10-06T03:29:27.799 INFO:teuthology.misc.health.smithi110.stderr:2023-10-06T03:29:27.790+0000 7f29f8f90700 1 --2- 172.21.15.110:0/400177367 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f29f40c65c0 0x7f29f4159620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:27.799 INFO:teuthology.misc.health.smithi110.stderr:2023-10-06T03:29:27.790+0000 7f29eb7fe700 1 --2- 172.21.15.110:0/400177367 >> v2:172.21.15.192:3300/0 conn(0x7f29f40caa60 0x7f29f40b0c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:27.799 INFO:teuthology.misc.health.smithi110.stderr:2023-10-06T03:29:27.790+0000 7f29f37fe700 1 -- 172.21.15.110:0/400177367 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 710924149 0 0) 0x7f29f406da70 con 0x7f29f4155b00 2023-10-06T03:29:27.800 INFO:teuthology.misc.health.smithi110.stderr:2023-10-06T03:29:27.790+0000 7f29f37fe700 1 -- 172.21.15.110:0/400177367 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f29d4003570 con 0x7f29f4155b00 2023-10-06T03:29:27.800 INFO:teuthology.misc.health.smithi110.stderr:2023-10-06T03:29:27.790+0000 7f29eb7fe700 1 -- 172.21.15.110:0/400177367 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f29f40c65c0 msgr2=0x7f29f4159620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:29:27.800 INFO:teuthology.misc.health.smithi110.stderr:2023-10-06T03:29:27.790+0000 7f29eb7fe700 1 --2- 172.21.15.110:0/400177367 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f29f40c65c0 0x7f29f4159620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:27.800 INFO:teuthology.misc.health.smithi110.stderr:2023-10-06T03:29:27.790+0000 7f29eb7fe700 1 -- 172.21.15.110:0/400177367 >> v1:172.21.15.110:6789/0 conn(0x7f29f4155b00 legacy=0x7f29f40c5eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:27.800 INFO:teuthology.misc.health.smithi110.stderr:2023-10-06T03:29:27.790+0000 7f29eb7fe700 1 -- 172.21.15.110:0/400177367 --> v2:172.21.15.192:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29f40ca5d0 con 0x7f29f40caa60 2023-10-06T03:29:27.800 INFO:teuthology.misc.health.smithi110.stderr:2023-10-06T03:29:27.790+0000 7f29f8f90700 1 --2- 172.21.15.110:0/400177367 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f29f40c65c0 0x7f29f4159620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-06T03:29:27.800 INFO:teuthology.misc.health.smithi110.stderr:2023-10-06T03:29:27.790+0000 7f29eb7fe700 1 --2- 172.21.15.110:0/400177367 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f29f40caa60 0x7f29f40b0c30 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f29dc0098e0 tx=0x7f29dc0091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6608c8a47d37ddf5 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:29:27.800 INFO:teuthology.misc.health.smithi110.stderr:2023-10-06T03:29:27.790+0000 7f29f37fe700 1 -- 172.21.15.110:0/400177367 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29dc011630 con 0x7f29f40caa60 2023-10-06T03:29:27.800 INFO:teuthology.misc.health.smithi110.stderr:2023-10-06T03:29:27.790+0000 7f29f37fe700 1 -- 172.21.15.110:0/400177367 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29dc011790 con 0x7f29f40caa60 2023-10-06T03:29:27.800 INFO:teuthology.misc.health.smithi110.stderr:2023-10-06T03:29:27.790+0000 7f29f37fe700 1 -- 172.21.15.110:0/400177367 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29dc011b10 con 0x7f29f40caa60 2023-10-06T03:29:27.801 INFO:teuthology.misc.health.smithi110.stderr:2023-10-06T03:29:27.790+0000 7f29f9791700 1 -- 172.21.15.110:0/400177367 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f29f40caa60 msgr2=0x7f29f40b0c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:27.801 INFO:teuthology.misc.health.smithi110.stderr:2023-10-06T03:29:27.790+0000 7f29f9791700 1 --2- 172.21.15.110:0/400177367 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f29f40caa60 0x7f29f40b0c30 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f29dc0098e0 tx=0x7f29dc0091a0 comp rx=0 tx=0).stop 2023-10-06T03:29:27.801 INFO:teuthology.misc.health.smithi110.stderr:2023-10-06T03:29:27.790+0000 7f29f9791700 1 -- 172.21.15.110:0/400177367 shutdown_connections 2023-10-06T03:29:27.801 INFO:teuthology.misc.health.smithi110.stderr:2023-10-06T03:29:27.790+0000 7f29f9791700 1 --2- 172.21.15.110:0/400177367 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f29f40c65c0 0x7f29f4159620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:27.801 INFO:teuthology.misc.health.smithi110.stderr:2023-10-06T03:29:27.790+0000 7f29f9791700 1 --2- 172.21.15.110:0/400177367 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f29f40caa60 0x7f29f40b0c30 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:27.801 INFO:teuthology.misc.health.smithi110.stderr:2023-10-06T03:29:27.790+0000 7f29f9791700 1 -- 172.21.15.110:0/400177367 >> 172.21.15.110:0/400177367 conn(0x7f29f4067b20 msgr2=0x7f29f40cb910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:29:27.801 INFO:teuthology.misc.health.smithi110.stderr:2023-10-06T03:29:27.790+0000 7f29f9791700 1 -- 172.21.15.110:0/400177367 shutdown_connections 2023-10-06T03:29:27.801 INFO:teuthology.misc.health.smithi110.stderr:2023-10-06T03:29:27.790+0000 7f29f9791700 1 -- 172.21.15.110:0/400177367 wait complete. 2023-10-06T03:29:27.804 INFO:teuthology.misc.health.smithi110.stderr:2023-10-06T03:29:27.794+0000 7f29f9791700 1 Processor -- start 2023-10-06T03:29:27.804 INFO:teuthology.misc.health.smithi110.stderr:2023-10-06T03:29:27.794+0000 7f29f9791700 1 -- start start 2023-10-06T03:29:27.804 INFO:teuthology.misc.health.smithi110.stderr:2023-10-06T03:29:27.794+0000 7f29f9791700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f29f40c65c0 0x7f29f40c4560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:27.804 INFO:teuthology.misc.health.smithi110.stderr:2023-10-06T03:29:27.794+0000 7f29f9791700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f29f40caa60 0x7f29f4146ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:27.804 INFO:teuthology.misc.health.smithi110.stderr:2023-10-06T03:29:27.794+0000 7f29f9791700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f29f4155b00 0x7f29f40c3cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:27.804 INFO:teuthology.misc.health.smithi110.stderr:2023-10-06T03:29:27.794+0000 7f29f9791700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29f40c5750 con 0x7f29f4155b00 2023-10-06T03:29:27.804 INFO:teuthology.misc.health.smithi110.stderr:2023-10-06T03:29:27.794+0000 7f29f9791700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29f40c41f0 con 0x7f29f40caa60 2023-10-06T03:29:27.804 INFO:teuthology.misc.health.smithi110.stderr:2023-10-06T03:29:27.794+0000 7f29f9791700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29f40c4b40 con 0x7f29f40c65c0 2023-10-06T03:29:27.804 INFO:teuthology.misc.health.smithi110.stderr:2023-10-06T03:29:27.794+0000 7f29f8f90700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f29f4155b00 0x7f29f40c3cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:27.804 INFO:teuthology.misc.health.smithi110.stderr:2023-10-06T03:29:27.794+0000 7f29f8f90700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f29f4155b00 0x7f29f40c3cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:38784/0 (socket says 172.21.15.110:38784) 2023-10-06T03:29:27.805 INFO:teuthology.misc.health.smithi110.stderr:2023-10-06T03:29:27.794+0000 7f29f8f90700 1 -- 172.21.15.110:0/4217214221 learned_addr learned my addr 172.21.15.110:0/4217214221 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:29:27.805 INFO:teuthology.misc.health.smithi110.stderr:2023-10-06T03:29:27.794+0000 7f29eb7fe700 1 --2- 172.21.15.110:0/4217214221 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f29f40caa60 0x7f29f4146ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:27.805 INFO:teuthology.misc.health.smithi110.stderr:2023-10-06T03:29:27.794+0000 7f29f3fff700 1 --2- 172.21.15.110:0/4217214221 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f29f40c65c0 0x7f29f40c4560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:27.805 INFO:teuthology.misc.health.smithi110.stderr:2023-10-06T03:29:27.794+0000 7f29f8f90700 1 -- 172.21.15.110:0/4217214221 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f29f40c65c0 msgr2=0x7f29f40c4560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:27.805 INFO:teuthology.misc.health.smithi110.stderr:2023-10-06T03:29:27.794+0000 7f29f8f90700 1 --2- 172.21.15.110:0/4217214221 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f29f40c65c0 0x7f29f40c4560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:27.805 INFO:teuthology.misc.health.smithi110.stderr:2023-10-06T03:29:27.794+0000 7f29f8f90700 1 -- 172.21.15.110:0/4217214221 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f29f40caa60 msgr2=0x7f29f4146ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:27.805 INFO:teuthology.misc.health.smithi110.stderr:2023-10-06T03:29:27.794+0000 7f29f8f90700 1 --2- 172.21.15.110:0/4217214221 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f29f40caa60 0x7f29f4146ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:27.805 INFO:teuthology.misc.health.smithi110.stderr:2023-10-06T03:29:27.794+0000 7f29f8f90700 1 -- 172.21.15.110:0/4217214221 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29ec009a20 con 0x7f29f4155b00 2023-10-06T03:29:27.805 INFO:teuthology.misc.health.smithi110.stderr:2023-10-06T03:29:27.794+0000 7f29f8f90700 1 --2- 172.21.15.110:0/4217214221 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f29f4155b00 0x7f29f40c3cb0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f29ec009fc0 tx=0x7f29ec00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:29:27.805 INFO:teuthology.misc.health.smithi110.stderr:2023-10-06T03:29:27.794+0000 7f29f0ff9700 1 -- 172.21.15.110:0/4217214221 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29ec00cbd0 con 0x7f29f4155b00 2023-10-06T03:29:27.805 INFO:teuthology.misc.health.smithi110.stderr:2023-10-06T03:29:27.794+0000 7f29f0ff9700 1 -- 172.21.15.110:0/4217214221 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29ec00f040 con 0x7f29f4155b00 2023-10-06T03:29:27.805 INFO:teuthology.misc.health.smithi110.stderr:2023-10-06T03:29:27.794+0000 7f29f9791700 1 -- 172.21.15.110:0/4217214221 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29dc009420 con 0x7f29f4155b00 2023-10-06T03:29:27.806 INFO:teuthology.misc.health.smithi110.stderr:2023-10-06T03:29:27.794+0000 7f29f0ff9700 1 -- 172.21.15.110:0/4217214221 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29ec014480 con 0x7f29f4155b00 2023-10-06T03:29:27.806 INFO:teuthology.misc.health.smithi110.stderr:2023-10-06T03:29:27.798+0000 7f29f9791700 1 -- 172.21.15.110:0/4217214221 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29f40c5100 con 0x7f29f4155b00 2023-10-06T03:29:27.807 INFO:teuthology.misc.health.smithi110.stderr:2023-10-06T03:29:27.798+0000 7f29f9791700 1 -- 172.21.15.110:0/4217214221 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f29f40c58c0 con 0x7f29f4155b00 2023-10-06T03:29:27.807 INFO:teuthology.misc.health.smithi110.stderr:2023-10-06T03:29:27.798+0000 7f29f0ff9700 1 -- 172.21.15.110:0/4217214221 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7f29ec003020 con 0x7f29f4155b00 2023-10-06T03:29:27.807 INFO:teuthology.misc.health.smithi110.stderr:2023-10-06T03:29:27.798+0000 7f29f0ff9700 1 --2- 172.21.15.110:0/4217214221 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f29d8060550 0x7f29d8062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:27.807 INFO:teuthology.misc.health.smithi110.stderr:2023-10-06T03:29:27.798+0000 7f29f3fff700 1 --2- 172.21.15.110:0/4217214221 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f29d8060550 0x7f29d8062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:27.807 INFO:teuthology.misc.health.smithi110.stderr:2023-10-06T03:29:27.798+0000 7f29f0ff9700 1 -- 172.21.15.110:0/4217214221 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4853+0+0 (secure 0 0 0) 0x7f29ec07cd60 con 0x7f29f4155b00 2023-10-06T03:29:27.807 INFO:teuthology.misc.health.smithi110.stderr:2023-10-06T03:29:27.798+0000 7f29f3fff700 1 --2- 172.21.15.110:0/4217214221 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f29d8060550 0x7f29d8062a00 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f29e4000c00 tx=0x7f29e4009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:29:27.815 INFO:teuthology.misc.health.smithi110.stderr:2023-10-06T03:29:27.810+0000 7f29f0ff9700 1 -- 172.21.15.110:0/4217214221 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f29ec04d6a0 con 0x7f29f4155b00 2023-10-06T03:29:27.972 INFO:teuthology.misc.health.smithi110.stderr:2023-10-06T03:29:27.966+0000 7f29f9791700 1 -- 172.21.15.110:0/4217214221 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f29f404be40 con 0x7f29f4155b00 2023-10-06T03:29:27.973 INFO:teuthology.misc.health.smithi110.stderr:2023-10-06T03:29:27.966+0000 7f29f0ff9700 1 -- 172.21.15.110:0/4217214221 <== mon.0 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v17) v1 ==== 74+0+12175 (secure 0 0 0) 0x7f29ec018090 con 0x7f29f4155b00 2023-10-06T03:29:27.973 INFO:teuthology.misc.health.smithi110.stdout: 2023-10-06T03:29:27.973 INFO:teuthology.misc.health.smithi110.stdout:{"epoch":17,"fsid":"93b8f6ac-4c92-4b45-b8e4-b59617024aa2","created":"2023-10-06T03:29:10.519583+0000","modified":"2023-10-06T03:29:22.975299+0000","last_up_change":"2023-10-06T03:29:21.960511+0000","last_in_change":"2023-10-06T03:29:12.560436+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-06T03:29:20.703743+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"5866eb6c-2adc-46e4-be7a-e5bcda0f4065","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6824","nonce":33394},{"type":"v1","addr":"172.21.15.110:6825","nonce":33394}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6826","nonce":33394},{"type":"v1","addr":"172.21.15.110:6827","nonce":33394}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6830","nonce":33394},{"type":"v1","addr":"172.21.15.110:6831","nonce":33394}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6828","nonce":33394},{"type":"v1","addr":"172.21.15.110:6829","nonce":33394}]},"public_addr":"172.21.15.110:6825/33394","cluster_addr":"172.21.15.110:6827/33394","heartbeat_back_addr":"172.21.15.110:6831/33394","heartbeat_front_addr":"172.21.15.110:6829/33394","state":["exists","up"]},{"osd":1,"uuid":"a38de118-062b-4a05-9782-2ceddf6ff4df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6800","nonce":33396},{"type":"v1","addr":"172.21.15.110:6801","nonce":33396}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6802","nonce":33396},{"type":"v1","addr":"172.21.15.110:6803","nonce":33396}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6806","nonce":33396},{"type":"v1","addr":"172.21.15.110:6807","nonce":33396}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6804","nonce":33396},{"type":"v1","addr":"172.21.15.110:6805","nonce":33396}]},"public_addr":"172.21.15.110:6801/33396","cluster_addr":"172.21.15.110:6803/33396","heartbeat_back_addr":"172.21.15.110:6807/33396","heartbeat_front_addr":"172.21.15.110:6805/33396","state":["exists","up"]},{"osd":2,"uuid":"56e39be2-3407-4b69-877e-cf16617b06d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6816","nonce":33397},{"type":"v1","addr":"172.21.15.110:6817","nonce":33397}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6818","nonce":33397},{"type":"v1","addr":"172.21.15.110:6819","nonce":33397}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6822","nonce":33397},{"type":"v1","addr":"172.21.15.110:6823","nonce":33397}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6820","nonce":33397},{"type":"v1","addr":"172.21.15.110:6821","nonce":33397}]},"public_addr":"172.21.15.110:6817/33397","cluster_addr":"172.21.15.110:6819/33397","heartbeat_back_addr":"172.21.15.110:6823/33397","heartbeat_front_addr":"172.21.15.110:6821/33397","state":["exists","up"]},{"osd":3,"uuid":"01c82d13-c1d4-4bde-b55e-3e0089b40115","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6808","nonce":33395},{"type":"v1","addr":"172.21.15.110:6809","nonce":33395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6810","nonce":33395},{"type":"v1","addr":"172.21.15.110:6811","nonce":33395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6814","nonce":33395},{"type":"v1","addr":"172.21.15.110:6815","nonce":33395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6812","nonce":33395},{"type":"v1","addr":"172.21.15.110:6813","nonce":33395}]},"public_addr":"172.21.15.110:6809/33395","cluster_addr":"172.21.15.110:6811/33395","heartbeat_back_addr":"172.21.15.110:6815/33395","heartbeat_front_addr":"172.21.15.110:6813/33395","state":["exists","up"]},{"osd":4,"uuid":"86983430-5fa3-4d17-998f-03e2cd843237","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":33683},{"type":"v1","addr":"172.21.15.192:6817","nonce":33683}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":33683},{"type":"v1","addr":"172.21.15.192:6819","nonce":33683}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":33683},{"type":"v1","addr":"172.21.15.192:6823","nonce":33683}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6820","nonce":33683},{"type":"v1","addr":"172.21.15.192:6821","nonce":33683}]},"public_addr":"172.21.15.192:6817/33683","cluster_addr":"172.21.15.192:6819/33683","heartbeat_back_addr":"172.21.15.192:6823/33683","heartbeat_front_addr":"172.21.15.192:6821/33683","state":["exists","up"]},{"osd":5,"uuid":"6945257f-821d-4619-a63f-ebe857eba6f4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":33684},{"type":"v1","addr":"172.21.15.192:6801","nonce":33684}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":33684},{"type":"v1","addr":"172.21.15.192:6803","nonce":33684}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":33684},{"type":"v1","addr":"172.21.15.192:6807","nonce":33684}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6804","nonce":33684},{"type":"v1","addr":"172.21.15.192:6805","nonce":33684}]},"public_addr":"172.21.15.192:6801/33684","cluster_addr":"172.21.15.192:6803/33684","heartbeat_back_addr":"172.21.15.192:6807/33684","heartbeat_front_addr":"172.21.15.192:6805/33684","state":["exists","up"]},{"osd":6,"uuid":"7854d1ae-7060-4780-8f97-73cea5e7a12f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6824","nonce":33686},{"type":"v1","addr":"172.21.15.192:6825","nonce":33686}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":33686},{"type":"v1","addr":"172.21.15.192:6827","nonce":33686}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":33686},{"type":"v1","addr":"172.21.15.192:6831","nonce":33686}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6828","nonce":33686},{"type":"v1","addr":"172.21.15.192:6829","nonce":33686}]},"public_addr":"172.21.15.192:6825/33686","cluster_addr":"172.21.15.192:6827/33686","heartbeat_back_addr":"172.21.15.192:6831/33686","heartbeat_front_addr":"172.21.15.192:6829/33686","state":["exists","up"]},{"osd":7,"uuid":"f1ebdfd5-b859-482b-9f97-c18c4a26f445","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6808","nonce":33685},{"type":"v1","addr":"172.21.15.192:6809","nonce":33685}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":33685},{"type":"v1","addr":"172.21.15.192:6811","nonce":33685}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":33685},{"type":"v1","addr":"172.21.15.192:6815","nonce":33685}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6812","nonce":33685},{"type":"v1","addr":"172.21.15.192:6813","nonce":33685}]},"public_addr":"172.21.15.192:6809/33685","cluster_addr":"172.21.15.192:6811/33685","heartbeat_back_addr":"172.21.15.192:6815/33685","heartbeat_front_addr":"172.21.15.192:6813/33685","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T03:29:18.002048+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T03:29:17.478576+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T03:29:18.034944+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T03:29:17.628513+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T03:29:18.047338+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T03:29:17.772349+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T03:29:17.883958+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T03:29:18.151838+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-06T03:29:27.975 INFO:teuthology.misc.health.smithi110.stderr:2023-10-06T03:29:27.970+0000 7f29f9791700 1 -- 172.21.15.110:0/4217214221 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f29d8060550 msgr2=0x7f29d8062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:27.976 INFO:teuthology.misc.health.smithi110.stderr:2023-10-06T03:29:27.970+0000 7f29f9791700 1 --2- 172.21.15.110:0/4217214221 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f29d8060550 0x7f29d8062a00 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f29e4000c00 tx=0x7f29e4009040 comp rx=0 tx=0).stop 2023-10-06T03:29:27.976 INFO:teuthology.misc.health.smithi110.stderr:2023-10-06T03:29:27.970+0000 7f29f9791700 1 -- 172.21.15.110:0/4217214221 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f29f4155b00 msgr2=0x7f29f40c3cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:27.976 INFO:teuthology.misc.health.smithi110.stderr:2023-10-06T03:29:27.970+0000 7f29f9791700 1 --2- 172.21.15.110:0/4217214221 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f29f4155b00 0x7f29f40c3cb0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f29ec009fc0 tx=0x7f29ec00b040 comp rx=0 tx=0).stop 2023-10-06T03:29:27.976 INFO:teuthology.misc.health.smithi110.stderr:2023-10-06T03:29:27.970+0000 7f29f9791700 1 -- 172.21.15.110:0/4217214221 shutdown_connections 2023-10-06T03:29:27.976 INFO:teuthology.misc.health.smithi110.stderr:2023-10-06T03:29:27.970+0000 7f29f9791700 1 --2- 172.21.15.110:0/4217214221 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f29d8060550 0x7f29d8062a00 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:27.976 INFO:teuthology.misc.health.smithi110.stderr:2023-10-06T03:29:27.970+0000 7f29f9791700 1 --2- 172.21.15.110:0/4217214221 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f29f4155b00 0x7f29f40c3cb0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:27.976 INFO:teuthology.misc.health.smithi110.stderr:2023-10-06T03:29:27.970+0000 7f29f9791700 1 --2- 172.21.15.110:0/4217214221 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f29f40caa60 0x7f29f4146ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:27.976 INFO:teuthology.misc.health.smithi110.stderr:2023-10-06T03:29:27.970+0000 7f29f9791700 1 --2- 172.21.15.110:0/4217214221 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f29f40c65c0 0x7f29f40c4560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:27.976 INFO:teuthology.misc.health.smithi110.stderr:2023-10-06T03:29:27.970+0000 7f29f9791700 1 -- 172.21.15.110:0/4217214221 >> 172.21.15.110:0/4217214221 conn(0x7f29f4067b20 msgr2=0x7f29f4154d60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:29:27.976 INFO:teuthology.misc.health.smithi110.stderr:2023-10-06T03:29:27.970+0000 7f29f9791700 1 -- 172.21.15.110:0/4217214221 shutdown_connections 2023-10-06T03:29:27.977 INFO:teuthology.misc.health.smithi110.stderr:2023-10-06T03:29:27.970+0000 7f29f9791700 1 -- 172.21.15.110:0/4217214221 wait complete. 2023-10-06T03:29:27.986 DEBUG:teuthology.misc:8 of 8 OSDs are up 2023-10-06T03:29:27.986 INFO:tasks.ceph:Creating RBD pool 2023-10-06T03:29:27.986 DEBUG:teuthology.orchestra.run.smithi110:> sudo ceph --cluster ceph osd pool create rbd 8 2023-10-06T03:29:28.090 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:28.082+0000 7ff349e58700 1 Processor -- start 2023-10-06T03:29:28.090 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:28.082+0000 7ff349e58700 1 -- start start 2023-10-06T03:29:28.092 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:28.086+0000 7ff349e58700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7ff344156b50 0x7ff3440c74e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:28.092 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:28.086+0000 7ff349e58700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7ff3440c7a20 0x7ff344067440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:28.092 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:28.086+0000 7ff349e58700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff34406b9f0 con 0x7ff344067980 2023-10-06T03:29:28.093 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:28.086+0000 7ff349e58700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff3440c7e50 con 0x7ff3440c7a20 2023-10-06T03:29:28.093 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:28.086+0000 7ff349e58700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff344068100 con 0x7ff344156b50 2023-10-06T03:29:28.093 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:28.086+0000 7ff33affd700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7ff3440c7a20 0x7ff344067440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:28.093 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:28.086+0000 7ff343fff700 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7ff344067980 0x7ff34406be80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:52514/0 (socket says 172.21.15.110:52514) 2023-10-06T03:29:28.093 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:28.086+0000 7ff343fff700 1 -- 172.21.15.110:0/2020198498 learned_addr learned my addr 172.21.15.110:0/2020198498 (peer_addr_for_me v1:172.21.15.110:0/0) 2023-10-06T03:29:28.093 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:28.086+0000 7ff3437fe700 1 --2- 172.21.15.110:0/2020198498 >> v2:172.21.15.192:3300/0 conn(0x7ff344156b50 0x7ff3440c74e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:28.093 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:28.086+0000 7ff33affd700 1 -- 172.21.15.110:0/2020198498 >> v2:172.21.15.192:3300/0 conn(0x7ff344156b50 msgr2=0x7ff3440c74e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:29:28.093 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:28.086+0000 7ff33affd700 1 --2- 172.21.15.110:0/2020198498 >> v2:172.21.15.192:3300/0 conn(0x7ff344156b50 0x7ff3440c74e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:28.093 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:28.086+0000 7ff33affd700 1 -- 172.21.15.110:0/2020198498 >> v1:172.21.15.110:6789/0 conn(0x7ff344067980 legacy=0x7ff34406be80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:28.093 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:28.086+0000 7ff33affd700 1 -- 172.21.15.110:0/2020198498 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3440683e0 con 0x7ff3440c7a20 2023-10-06T03:29:28.093 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:28.086+0000 7ff3437fe700 1 --2- 172.21.15.110:0/2020198498 >> v2:172.21.15.192:3300/0 conn(0x7ff344156b50 0x7ff3440c74e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-06T03:29:28.093 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:28.086+0000 7ff342ffd700 1 -- 172.21.15.110:0/2020198498 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2200068406 0 0) 0x7ff34406b9f0 con 0x7ff344067980 2023-10-06T03:29:28.094 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:28.086+0000 7ff33affd700 1 --2- 172.21.15.110:0/2020198498 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff3440c7a20 0x7ff344067440 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7ff344067100 tx=0x7ff32c0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d719eb607115f3d server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:29:28.094 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:28.086+0000 7ff342ffd700 1 -- 172.21.15.110:0/2020198498 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff32c00b6b0 con 0x7ff3440c7a20 2023-10-06T03:29:28.094 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:28.086+0000 7ff342ffd700 1 -- 172.21.15.110:0/2020198498 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff32c00b810 con 0x7ff3440c7a20 2023-10-06T03:29:28.094 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:28.086+0000 7ff342ffd700 1 -- 172.21.15.110:0/2020198498 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff32c00bbb0 con 0x7ff3440c7a20 2023-10-06T03:29:28.095 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:28.086+0000 7ff349e58700 1 -- 172.21.15.110:0/2020198498 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff3440c7a20 msgr2=0x7ff344067440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:28.095 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:28.086+0000 7ff349e58700 1 --2- 172.21.15.110:0/2020198498 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff3440c7a20 0x7ff344067440 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7ff344067100 tx=0x7ff32c0091a0 comp rx=0 tx=0).stop 2023-10-06T03:29:28.095 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:28.086+0000 7ff349e58700 1 -- 172.21.15.110:0/2020198498 shutdown_connections 2023-10-06T03:29:28.095 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:28.086+0000 7ff349e58700 1 --2- 172.21.15.110:0/2020198498 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff3440c7a20 0x7ff344067440 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:28.095 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:28.086+0000 7ff349e58700 1 --2- 172.21.15.110:0/2020198498 >> v2:172.21.15.192:3300/0 conn(0x7ff344156b50 0x7ff3440c74e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:28.095 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:28.086+0000 7ff349e58700 1 -- 172.21.15.110:0/2020198498 >> 172.21.15.110:0/2020198498 conn(0x7ff3440c8f00 msgr2=0x7ff3440c9300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:29:28.095 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:28.086+0000 7ff349e58700 1 -- 172.21.15.110:0/2020198498 shutdown_connections 2023-10-06T03:29:28.095 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:28.086+0000 7ff349e58700 1 -- 172.21.15.110:0/2020198498 wait complete. 2023-10-06T03:29:28.096 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:28.086+0000 7ff349e58700 1 Processor -- start 2023-10-06T03:29:28.096 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:28.086+0000 7ff349e58700 1 -- start start 2023-10-06T03:29:28.097 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:28.090+0000 7ff349e58700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff344067980 0x7ff34414f490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:28.097 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:28.090+0000 7ff349e58700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff3440c7a20 0x7ff3440c5cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:28.097 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:28.090+0000 7ff349e58700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff344156b50 0x7ff3440c1230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:28.098 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:28.090+0000 7ff349e58700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff34414f9d0 con 0x7ff3440c7a20 2023-10-06T03:29:28.098 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:28.090+0000 7ff349e58700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3440ca270 con 0x7ff344156b50 2023-10-06T03:29:28.098 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:28.090+0000 7ff349e58700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff344169820 con 0x7ff344067980 2023-10-06T03:29:28.098 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:28.090+0000 7ff33affd700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff3440c7a20 0x7ff3440c5cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:28.098 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:28.090+0000 7ff33affd700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff3440c7a20 0x7ff3440c5cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:38806/0 (socket says 172.21.15.110:38806) 2023-10-06T03:29:28.098 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:28.090+0000 7ff33affd700 1 -- 172.21.15.110:0/432042753 learned_addr learned my addr 172.21.15.110:0/432042753 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:29:28.098 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:28.090+0000 7ff3437fe700 1 --2- 172.21.15.110:0/432042753 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff344067980 0x7ff34414f490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:28.098 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:28.090+0000 7ff343fff700 1 --2- 172.21.15.110:0/432042753 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff344156b50 0x7ff3440c1230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:28.098 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:28.090+0000 7ff33affd700 1 -- 172.21.15.110:0/432042753 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff344067980 msgr2=0x7ff34414f490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:28.098 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:28.090+0000 7ff33affd700 1 --2- 172.21.15.110:0/432042753 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff344067980 0x7ff34414f490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:28.098 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:28.090+0000 7ff33affd700 1 -- 172.21.15.110:0/432042753 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff344156b50 msgr2=0x7ff3440c1230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:28.099 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:28.090+0000 7ff33affd700 1 --2- 172.21.15.110:0/432042753 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff344156b50 0x7ff3440c1230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:28.099 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:28.090+0000 7ff33affd700 1 -- 172.21.15.110:0/432042753 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff32c009420 con 0x7ff3440c7a20 2023-10-06T03:29:28.099 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:28.090+0000 7ff343fff700 1 --2- 172.21.15.110:0/432042753 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff344156b50 0x7ff3440c1230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-06T03:29:28.099 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:28.090+0000 7ff33affd700 1 --2- 172.21.15.110:0/432042753 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff3440c7a20 0x7ff3440c5cd0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7ff34406f8c0 tx=0x7ff32c017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:29:28.099 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:28.090+0000 7ff33bfff700 1 -- 172.21.15.110:0/432042753 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff32c011420 con 0x7ff3440c7a20 2023-10-06T03:29:28.099 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:28.090+0000 7ff33bfff700 1 -- 172.21.15.110:0/432042753 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff32c011580 con 0x7ff3440c7a20 2023-10-06T03:29:28.099 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:28.090+0000 7ff349e58700 1 -- 172.21.15.110:0/432042753 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff344169b00 con 0x7ff3440c7a20 2023-10-06T03:29:28.099 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:28.090+0000 7ff33bfff700 1 -- 172.21.15.110:0/432042753 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff32c011920 con 0x7ff3440c7a20 2023-10-06T03:29:28.099 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:28.090+0000 7ff349e58700 1 -- 172.21.15.110:0/432042753 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff34416a0c0 con 0x7ff3440c7a20 2023-10-06T03:29:28.100 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:28.090+0000 7ff349e58700 1 -- 172.21.15.110:0/432042753 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff34406ff60 con 0x7ff3440c7a20 2023-10-06T03:29:28.100 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:28.090+0000 7ff33bfff700 1 -- 172.21.15.110:0/432042753 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7ff32c011cd0 con 0x7ff3440c7a20 2023-10-06T03:29:28.100 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:28.094+0000 7ff33bfff700 1 --2- 172.21.15.110:0/432042753 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7ff324060550 0x7ff324062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:28.100 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:28.094+0000 7ff3437fe700 1 --2- 172.21.15.110:0/432042753 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7ff324060550 0x7ff324062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:28.100 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:28.094+0000 7ff33bfff700 1 -- 172.21.15.110:0/432042753 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4853+0+0 (secure 0 0 0) 0x7ff32c07e0a0 con 0x7ff3440c7a20 2023-10-06T03:29:28.100 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:28.094+0000 7ff3437fe700 1 --2- 172.21.15.110:0/432042753 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7ff324060550 0x7ff324062a00 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7ff334006fd0 tx=0x7ff334008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:29:28.108 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:28.102+0000 7ff33bfff700 1 -- 172.21.15.110:0/432042753 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7ff32c04e9c0 con 0x7ff3440c7a20 2023-10-06T03:29:28.268 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:28.262+0000 7ff349e58700 1 -- 172.21.15.110:0/432042753 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7ff3440686c0 con 0x7ff3440c7a20 2023-10-06T03:29:28.367 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:28.358+0000 7ff33bfff700 1 -- 172.21.15.110:0/432042753 <== mon.0 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}]=0 pool 'rbd' created v18) v1 ==== 109+0+0 (secure 0 0 0) 0x7ff32c051fe0 con 0x7ff3440c7a20 2023-10-06T03:29:28.371 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:28.366+0000 7ff349e58700 1 -- 172.21.15.110:0/432042753 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7ff324060550 msgr2=0x7ff324062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:28.372 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:28.366+0000 7ff349e58700 1 --2- 172.21.15.110:0/432042753 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7ff324060550 0x7ff324062a00 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7ff334006fd0 tx=0x7ff334008040 comp rx=0 tx=0).stop 2023-10-06T03:29:28.372 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:28.366+0000 7ff349e58700 1 -- 172.21.15.110:0/432042753 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff3440c7a20 msgr2=0x7ff3440c5cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:28.372 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:28.366+0000 7ff349e58700 1 --2- 172.21.15.110:0/432042753 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff3440c7a20 0x7ff3440c5cd0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7ff34406f8c0 tx=0x7ff32c017000 comp rx=0 tx=0).stop 2023-10-06T03:29:28.372 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:28.366+0000 7ff349e58700 1 -- 172.21.15.110:0/432042753 shutdown_connections 2023-10-06T03:29:28.372 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:28.366+0000 7ff349e58700 1 --2- 172.21.15.110:0/432042753 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7ff324060550 0x7ff324062a00 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:28.372 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:28.366+0000 7ff349e58700 1 --2- 172.21.15.110:0/432042753 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff344156b50 0x7ff3440c1230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:28.372 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:28.366+0000 7ff349e58700 1 --2- 172.21.15.110:0/432042753 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff3440c7a20 0x7ff3440c5cd0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:28.372 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:28.366+0000 7ff349e58700 1 --2- 172.21.15.110:0/432042753 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff344067980 0x7ff34414f490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:28.372 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:28.366+0000 7ff349e58700 1 -- 172.21.15.110:0/432042753 >> 172.21.15.110:0/432042753 conn(0x7ff3440c8f00 msgr2=0x7ff34415a5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:29:28.372 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:28.366+0000 7ff349e58700 1 -- 172.21.15.110:0/432042753 shutdown_connections 2023-10-06T03:29:28.373 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:28.366+0000 7ff349e58700 1 -- 172.21.15.110:0/432042753 wait complete. 2023-10-06T03:29:28.373 INFO:teuthology.orchestra.run.smithi110.stderr:pool 'rbd' created 2023-10-06T03:29:28.385 DEBUG:teuthology.orchestra.run.smithi110:> sudo ceph --cluster ceph osd pool application enable rbd rbd --yes-i-really-mean-it 2023-10-06T03:29:28.496 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:28.490+0000 7f5aaf59e700 1 Processor -- start 2023-10-06T03:29:28.496 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:28.490+0000 7f5aaf59e700 1 -- start start 2023-10-06T03:29:28.499 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:28.490+0000 7f5aaf59e700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f5ab00aeb60 0x7f5ab00aef30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:28.499 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:28.490+0000 7f5aaf59e700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5ab00cf4b0 0x7f5ab00cf900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:28.499 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:28.490+0000 7f5aaf59e700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f5ab01589b0 0x7f5ab0158e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:28.499 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:28.490+0000 7f5aaf59e700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5ab015d930 con 0x7f5ab00aeb60 2023-10-06T03:29:28.499 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:28.490+0000 7f5aaf59e700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5ab00ca2d0 con 0x7f5ab01589b0 2023-10-06T03:29:28.499 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:28.490+0000 7f5aaf59e700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ab015cd20 con 0x7f5ab00cf4b0 2023-10-06T03:29:28.499 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:28.490+0000 7f5aae59c700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f5ab00aeb60 0x7f5ab00aef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:28.499 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:28.490+0000 7f5aae59c700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f5ab00aeb60 0x7f5ab00aef30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:38814/0 (socket says 172.21.15.110:38814) 2023-10-06T03:29:28.499 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:28.490+0000 7f5aae59c700 1 -- 172.21.15.110:0/2585763137 learned_addr learned my addr 172.21.15.110:0/2585763137 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:29:28.499 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:28.490+0000 7f5aaed9d700 1 --2- 172.21.15.110:0/2585763137 >> v2:172.21.15.192:3300/0 conn(0x7f5ab01589b0 0x7f5ab0158e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:28.500 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:28.490+0000 7f5aadd9b700 1 --2- 172.21.15.110:0/2585763137 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5ab00cf4b0 0x7f5ab00cf900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:28.500 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:28.490+0000 7f5aae59c700 1 -- 172.21.15.110:0/2585763137 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5ab00cf4b0 msgr2=0x7f5ab00cf900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:29:28.500 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:28.490+0000 7f5aae59c700 1 --2- 172.21.15.110:0/2585763137 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5ab00cf4b0 0x7f5ab00cf900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:28.500 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:28.490+0000 7f5aae59c700 1 -- 172.21.15.110:0/2585763137 >> v2:172.21.15.192:3300/0 conn(0x7f5ab01589b0 msgr2=0x7f5ab0158e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:29:28.500 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:28.490+0000 7f5aae59c700 1 --2- 172.21.15.110:0/2585763137 >> v2:172.21.15.192:3300/0 conn(0x7f5ab01589b0 0x7f5ab0158e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:28.500 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:28.490+0000 7f5aae59c700 1 -- 172.21.15.110:0/2585763137 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ab015d000 con 0x7f5ab00aeb60 2023-10-06T03:29:28.500 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:28.490+0000 7f5aadd9b700 1 --2- 172.21.15.110:0/2585763137 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5ab00cf4b0 0x7f5ab00cf900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-06T03:29:28.500 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:28.490+0000 7f5aae59c700 1 --2- 172.21.15.110:0/2585763137 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5ab00aeb60 0x7f5ab00aef30 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f5ab00cd950 tx=0x7f5a98009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=86ef85c112b0c2be server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:29:28.500 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:28.490+0000 7f5aad59a700 1 -- 172.21.15.110:0/2585763137 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a9800b860 con 0x7f5ab00aeb60 2023-10-06T03:29:28.500 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:28.490+0000 7f5aad59a700 1 -- 172.21.15.110:0/2585763137 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a9800b9c0 con 0x7f5ab00aeb60 2023-10-06T03:29:28.500 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:28.490+0000 7f5aad59a700 1 -- 172.21.15.110:0/2585763137 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a9800bd80 con 0x7f5ab00aeb60 2023-10-06T03:29:28.501 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:28.494+0000 7f5aaf59e700 1 -- 172.21.15.110:0/2585763137 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5ab00aeb60 msgr2=0x7f5ab00aef30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:28.501 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:28.494+0000 7f5aaf59e700 1 --2- 172.21.15.110:0/2585763137 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5ab00aeb60 0x7f5ab00aef30 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f5ab00cd950 tx=0x7f5a98009250 comp rx=0 tx=0).stop 2023-10-06T03:29:28.501 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:28.494+0000 7f5aaf59e700 1 -- 172.21.15.110:0/2585763137 shutdown_connections 2023-10-06T03:29:28.502 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:28.494+0000 7f5aaf59e700 1 --2- 172.21.15.110:0/2585763137 >> v2:172.21.15.192:3300/0 conn(0x7f5ab01589b0 0x7f5ab0158e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:28.502 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:28.494+0000 7f5aaf59e700 1 --2- 172.21.15.110:0/2585763137 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5ab00cf4b0 0x7f5ab00cf900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:28.502 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:28.494+0000 7f5aaf59e700 1 --2- 172.21.15.110:0/2585763137 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5ab00aeb60 0x7f5ab00aef30 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:28.502 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:28.494+0000 7f5aaf59e700 1 -- 172.21.15.110:0/2585763137 >> 172.21.15.110:0/2585763137 conn(0x7f5ab00cc920 msgr2=0x7f5ab00ccd20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:29:28.502 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:28.494+0000 7f5aaf59e700 1 -- 172.21.15.110:0/2585763137 shutdown_connections 2023-10-06T03:29:28.502 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:28.494+0000 7f5aaf59e700 1 -- 172.21.15.110:0/2585763137 wait complete. 2023-10-06T03:29:28.504 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:28.494+0000 7f5aaf59e700 1 Processor -- start 2023-10-06T03:29:28.504 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:28.494+0000 7f5aaf59e700 1 -- start start 2023-10-06T03:29:28.504 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:28.498+0000 7f5aaf59e700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5ab00aeb60 0x7f5ab01557f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:28.505 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:28.498+0000 7f5aaf59e700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5ab00cf4b0 0x7f5ab00b41e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:28.505 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:28.498+0000 7f5aaf59e700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5ab01589b0 0x7f5ab00c9600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:28.505 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:28.498+0000 7f5aaf59e700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ab00b2ef0 con 0x7f5ab00cf4b0 2023-10-06T03:29:28.505 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:28.498+0000 7f5aaf59e700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ab0158fc0 con 0x7f5ab00aeb60 2023-10-06T03:29:28.505 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:28.498+0000 7f5aaf59e700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ab00b3140 con 0x7f5ab01589b0 2023-10-06T03:29:28.505 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:28.498+0000 7f5aadd9b700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5ab00cf4b0 0x7f5ab00b41e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:28.505 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:28.498+0000 7f5aae59c700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5ab00aeb60 0x7f5ab01557f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:28.505 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:28.498+0000 7f5aadd9b700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5ab00cf4b0 0x7f5ab00b41e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:38828/0 (socket says 172.21.15.110:38828) 2023-10-06T03:29:28.505 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:28.498+0000 7f5aaed9d700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5ab01589b0 0x7f5ab00c9600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:28.505 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:28.498+0000 7f5aadd9b700 1 -- 172.21.15.110:0/4265136337 learned_addr learned my addr 172.21.15.110:0/4265136337 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:29:28.505 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:28.498+0000 7f5aadd9b700 1 -- 172.21.15.110:0/4265136337 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5ab01589b0 msgr2=0x7f5ab00c9600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:28.505 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:28.498+0000 7f5aadd9b700 1 --2- 172.21.15.110:0/4265136337 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5ab01589b0 0x7f5ab00c9600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:28.506 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:28.498+0000 7f5aadd9b700 1 -- 172.21.15.110:0/4265136337 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5ab00aeb60 msgr2=0x7f5ab01557f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:28.506 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:28.498+0000 7f5aadd9b700 1 --2- 172.21.15.110:0/4265136337 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5ab00aeb60 0x7f5ab01557f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:28.506 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:28.498+0000 7f5aadd9b700 1 -- 172.21.15.110:0/4265136337 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a980094f0 con 0x7f5ab00cf4b0 2023-10-06T03:29:28.506 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:28.498+0000 7f5aadd9b700 1 --2- 172.21.15.110:0/4265136337 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5ab00cf4b0 0x7f5ab00b41e0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f5aa000c850 tx=0x7f5aa0008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:29:28.507 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:28.498+0000 7f5aa67fc700 1 -- 172.21.15.110:0/4265136337 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5aa0011600 con 0x7f5ab00cf4b0 2023-10-06T03:29:28.507 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:28.498+0000 7f5aa67fc700 1 -- 172.21.15.110:0/4265136337 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5aa0011760 con 0x7f5ab00cf4b0 2023-10-06T03:29:28.507 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:28.498+0000 7f5aaf59e700 1 -- 172.21.15.110:0/4265136337 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ab00b3420 con 0x7f5ab00cf4b0 2023-10-06T03:29:28.507 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:28.498+0000 7f5aa67fc700 1 -- 172.21.15.110:0/4265136337 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5aa0011b00 con 0x7f5ab00cf4b0 2023-10-06T03:29:28.507 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:28.498+0000 7f5aaf59e700 1 -- 172.21.15.110:0/4265136337 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ab00b39e0 con 0x7f5ab00cf4b0 2023-10-06T03:29:28.507 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:28.498+0000 7f5aaf59e700 1 -- 172.21.15.110:0/4265136337 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ab0070ff0 con 0x7f5ab00cf4b0 2023-10-06T03:29:28.508 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:28.502+0000 7f5aa67fc700 1 -- 172.21.15.110:0/4265136337 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7f5aa0018350 con 0x7f5ab00cf4b0 2023-10-06T03:29:28.509 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:28.502+0000 7f5aa67fc700 1 --2- 172.21.15.110:0/4265136337 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f5a90060580 0x7f5a90062a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:28.509 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:28.502+0000 7f5aa67fc700 1 -- 172.21.15.110:0/4265136337 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 5195+0+0 (secure 0 0 0) 0x7f5aa007c420 con 0x7f5ab00cf4b0 2023-10-06T03:29:28.509 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:28.502+0000 7f5aae59c700 1 --2- 172.21.15.110:0/4265136337 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f5a90060580 0x7f5a90062a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:28.509 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:28.502+0000 7f5aae59c700 1 --2- 172.21.15.110:0/4265136337 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f5a90060580 0x7f5a90062a30 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f5ab00cd950 tx=0x7f5a98017000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:29:28.514 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:28.506+0000 7f5aa67fc700 1 -- 172.21.15.110:0/4265136337 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f5aa0050600 con 0x7f5ab00cf4b0 2023-10-06T03:29:28.675 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:28.666+0000 7f5aaf59e700 1 -- 172.21.15.110:0/4265136337 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f5ab015d2e0 con 0x7f5ab00cf4b0 2023-10-06T03:29:29.376 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:29.370+0000 7f5aa67fc700 1 -- 172.21.15.110:0/4265136337 <== mon.0 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true}]=0 enabled application 'rbd' on pool 'rbd' v19) v1 ==== 173+0+0 (secure 0 0 0) 0x7f5aa0050190 con 0x7f5ab00cf4b0 2023-10-06T03:29:29.380 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:29.374+0000 7f5aaf59e700 1 -- 172.21.15.110:0/4265136337 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f5a90060580 msgr2=0x7f5a90062a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:29.381 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:29.374+0000 7f5aaf59e700 1 --2- 172.21.15.110:0/4265136337 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f5a90060580 0x7f5a90062a30 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f5ab00cd950 tx=0x7f5a98017000 comp rx=0 tx=0).stop 2023-10-06T03:29:29.381 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:29.374+0000 7f5aaf59e700 1 -- 172.21.15.110:0/4265136337 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5ab00cf4b0 msgr2=0x7f5ab00b41e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:29.381 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:29.374+0000 7f5aaf59e700 1 --2- 172.21.15.110:0/4265136337 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5ab00cf4b0 0x7f5ab00b41e0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f5aa000c850 tx=0x7f5aa0008040 comp rx=0 tx=0).stop 2023-10-06T03:29:29.381 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:29.374+0000 7f5aaf59e700 1 -- 172.21.15.110:0/4265136337 shutdown_connections 2023-10-06T03:29:29.381 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:29.374+0000 7f5aaf59e700 1 --2- 172.21.15.110:0/4265136337 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f5a90060580 0x7f5a90062a30 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:29.382 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:29.374+0000 7f5aaf59e700 1 --2- 172.21.15.110:0/4265136337 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5ab01589b0 0x7f5ab00c9600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:29.382 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:29.374+0000 7f5aaf59e700 1 --2- 172.21.15.110:0/4265136337 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5ab00cf4b0 0x7f5ab00b41e0 secure :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f5aa000c850 tx=0x7f5aa0008040 comp rx=0 tx=0).stop 2023-10-06T03:29:29.382 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:29.374+0000 7f5aaf59e700 1 --2- 172.21.15.110:0/4265136337 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5ab00aeb60 0x7f5ab01557f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:29.382 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:29.374+0000 7f5aaf59e700 1 -- 172.21.15.110:0/4265136337 >> 172.21.15.110:0/4265136337 conn(0x7f5ab00cc920 msgr2=0x7f5ab006b7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:29:29.382 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:29.374+0000 7f5aaf59e700 1 -- 172.21.15.110:0/4265136337 shutdown_connections 2023-10-06T03:29:29.383 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:29.374+0000 7f5aaf59e700 1 -- 172.21.15.110:0/4265136337 wait complete. 2023-10-06T03:29:29.385 INFO:teuthology.orchestra.run.smithi110.stderr:enabled application 'rbd' on pool 'rbd' 2023-10-06T03:29:29.396 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2023-10-06T03:29:29.397 INFO:tasks.ceph.mds.a:Restarting daemon 2023-10-06T03:29:29.397 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-10-06T03:29:29.399 INFO:tasks.ceph.mds.a:Started 2023-10-06T03:29:29.399 INFO:tasks.ceph.mds.c:Restarting daemon 2023-10-06T03:29:29.399 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-10-06T03:29:29.401 INFO:tasks.ceph.mds.c:Started 2023-10-06T03:29:29.401 INFO:tasks.ceph.mds.b:Restarting daemon 2023-10-06T03:29:29.401 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2023-10-06T03:29:29.404 INFO:tasks.ceph.mds.b:Started 2023-10-06T03:29:29.404 INFO:tasks.ceph.mds.d:Restarting daemon 2023-10-06T03:29:29.404 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-10-06T03:29:29.406 INFO:tasks.ceph.mds.d:Started 2023-10-06T03:29:29.407 INFO:tasks.ceph:Setting up CephFS filesystem(s)... 2023-10-06T03:29:29.407 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-06T03:29:29.470 INFO:tasks.ceph.mds.c.smithi110.stdout:starting mds.c at 2023-10-06T03:29:29.476 INFO:tasks.ceph.mds.b.smithi192.stdout:starting mds.b at 2023-10-06T03:29:29.484 INFO:tasks.ceph.mds.a.smithi110.stdout:starting mds.a at 2023-10-06T03:29:29.492 INFO:tasks.ceph.mds.d.smithi192.stdout:starting mds.d at 2023-10-06T03:29:29.519 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:29.510+0000 7f1a99384700 1 Processor -- start 2023-10-06T03:29:29.519 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:29.510+0000 7f1a99384700 1 -- start start 2023-10-06T03:29:29.522 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:29.514+0000 7f1a99384700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f1a941550c0 0x7f1a94155490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:29.522 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:29.514+0000 7f1a99384700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f1a940caf80 0x7f1a940cb420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:29.522 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:29.514+0000 7f1a99384700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a94156220 con 0x7f1a941566b0 2023-10-06T03:29:29.522 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:29.514+0000 7f1a99384700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1a941559d0 con 0x7f1a940caf80 2023-10-06T03:29:29.523 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:29.514+0000 7f1a99384700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1a940cb960 con 0x7f1a941550c0 2023-10-06T03:29:29.523 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:29.514+0000 7f1a98b83700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f1a940caf80 0x7f1a940cb420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:29.523 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:29.514+0000 7f1a98b83700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f1a940caf80 0x7f1a940cb420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:38866/0 (socket says 172.21.15.110:38866) 2023-10-06T03:29:29.523 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:29.514+0000 7f1a98b83700 1 -- 172.21.15.110:0/97499060 learned_addr learned my addr 172.21.15.110:0/97499060 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:29:29.523 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:29.514+0000 7f1a93fff700 1 --2- 172.21.15.110:0/97499060 >> v2:172.21.15.192:3300/0 conn(0x7f1a941550c0 0x7f1a94155490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:29.523 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:29.514+0000 7f1a98b83700 1 -- 172.21.15.110:0/97499060 >> v2:172.21.15.192:3300/0 conn(0x7f1a941550c0 msgr2=0x7f1a94155490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:29:29.523 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:29.514+0000 7f1a98b83700 1 --2- 172.21.15.110:0/97499060 >> v2:172.21.15.192:3300/0 conn(0x7f1a941550c0 0x7f1a94155490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:29.523 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:29.514+0000 7f1a98b83700 1 -- 172.21.15.110:0/97499060 >> v1:172.21.15.192:6789/0 conn(0x7f1a941566b0 legacy=0x7f1a940670a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:29.523 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:29.514+0000 7f1a98b83700 1 -- 172.21.15.110:0/97499060 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a94067b60 con 0x7f1a940caf80 2023-10-06T03:29:29.523 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:29.514+0000 7f1a93fff700 1 --2- 172.21.15.110:0/97499060 >> v2:172.21.15.192:3300/0 conn(0x7f1a941550c0 0x7f1a94155490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-06T03:29:29.524 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:29.514+0000 7f1a98b83700 1 --2- 172.21.15.110:0/97499060 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1a940caf80 0x7f1a940cb420 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f1a8c00c0d0 tx=0x7f1a8c00a950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=605a5440933b0739 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:29:29.524 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:29.514+0000 7f1a937fe700 1 -- 172.21.15.110:0/97499060 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a8c00f070 con 0x7f1a940caf80 2023-10-06T03:29:29.524 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:29.514+0000 7f1a937fe700 1 -- 172.21.15.110:0/97499060 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a8c00a2e0 con 0x7f1a940caf80 2023-10-06T03:29:29.524 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:29.514+0000 7f1a937fe700 1 -- 172.21.15.110:0/97499060 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a8c00cdf0 con 0x7f1a940caf80 2023-10-06T03:29:29.525 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:29.514+0000 7f1a99384700 1 -- 172.21.15.110:0/97499060 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1a940caf80 msgr2=0x7f1a940cb420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:29.525 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:29.514+0000 7f1a99384700 1 --2- 172.21.15.110:0/97499060 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1a940caf80 0x7f1a940cb420 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f1a8c00c0d0 tx=0x7f1a8c00a950 comp rx=0 tx=0).stop 2023-10-06T03:29:29.525 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:29.514+0000 7f1a99384700 1 -- 172.21.15.110:0/97499060 shutdown_connections 2023-10-06T03:29:29.525 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:29.514+0000 7f1a99384700 1 --2- 172.21.15.110:0/97499060 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1a940caf80 0x7f1a940cb420 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:29.525 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:29.514+0000 7f1a99384700 1 --2- 172.21.15.110:0/97499060 >> v2:172.21.15.192:3300/0 conn(0x7f1a941550c0 0x7f1a94155490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:29.525 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:29.514+0000 7f1a99384700 1 -- 172.21.15.110:0/97499060 >> 172.21.15.110:0/97499060 conn(0x7f1a940c8d00 msgr2=0x7f1a940c9100 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:29:29.525 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:29.518+0000 7f1a99384700 1 -- 172.21.15.110:0/97499060 shutdown_connections 2023-10-06T03:29:29.525 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:29.518+0000 7f1a99384700 1 -- 172.21.15.110:0/97499060 wait complete. 2023-10-06T03:29:29.526 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:29.518+0000 7f1a99384700 1 Processor -- start 2023-10-06T03:29:29.526 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:29.518+0000 7f1a99384700 1 -- start start 2023-10-06T03:29:29.527 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:29.518+0000 7f1a99384700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1a940caf80 0x7f1a94147e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:29.528 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:29.518+0000 7f1a99384700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1a941550c0 0x7f1a9414c680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:29.528 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:29.518+0000 7f1a99384700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1a941566b0 0x7f1a9414fb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:29.528 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:29.518+0000 7f1a99384700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a9414e270 con 0x7f1a940caf80 2023-10-06T03:29:29.528 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:29.518+0000 7f1a99384700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a94148390 con 0x7f1a941566b0 2023-10-06T03:29:29.528 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:29.518+0000 7f1a99384700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a9414e480 con 0x7f1a941550c0 2023-10-06T03:29:29.528 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:29.518+0000 7f1a93fff700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1a940caf80 0x7f1a94147e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:29.528 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:29.518+0000 7f1a98b83700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1a941566b0 0x7f1a9414fb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:29.528 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:29.518+0000 7f1a93fff700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1a940caf80 0x7f1a94147e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:38876/0 (socket says 172.21.15.110:38876) 2023-10-06T03:29:29.528 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:29.518+0000 7f1a98b83700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1a941566b0 0x7f1a9414fb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.110:46816/0 (socket says 172.21.15.110:46816) 2023-10-06T03:29:29.528 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:29.518+0000 7f1a93fff700 1 -- 172.21.15.110:0/191039404 learned_addr learned my addr 172.21.15.110:0/191039404 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:29:29.528 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:29.518+0000 7f1a8bfff700 1 --2- 172.21.15.110:0/191039404 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1a941550c0 0x7f1a9414c680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:29.529 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:29.518+0000 7f1a93fff700 1 -- 172.21.15.110:0/191039404 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1a941550c0 msgr2=0x7f1a9414c680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:29.529 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:29.518+0000 7f1a93fff700 1 --2- 172.21.15.110:0/191039404 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1a941550c0 0x7f1a9414c680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:29.529 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:29.518+0000 7f1a93fff700 1 -- 172.21.15.110:0/191039404 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1a941566b0 msgr2=0x7f1a9414fb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:29.529 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:29.518+0000 7f1a93fff700 1 --2- 172.21.15.110:0/191039404 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1a941566b0 0x7f1a9414fb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:29.529 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:29.518+0000 7f1a93fff700 1 -- 172.21.15.110:0/191039404 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a8c00abf0 con 0x7f1a940caf80 2023-10-06T03:29:29.529 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:29.518+0000 7f1a93fff700 1 --2- 172.21.15.110:0/191039404 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1a940caf80 0x7f1a94147e50 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f1a8400c830 tx=0x7f1a84008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:29:29.529 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:29.522+0000 7f1a90ff9700 1 -- 172.21.15.110:0/191039404 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a84005a00 con 0x7f1a940caf80 2023-10-06T03:29:29.529 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:29.522+0000 7f1a90ff9700 1 -- 172.21.15.110:0/191039404 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a84005b60 con 0x7f1a940caf80 2023-10-06T03:29:29.529 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:29.522+0000 7f1a99384700 1 -- 172.21.15.110:0/191039404 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a9414e760 con 0x7f1a940caf80 2023-10-06T03:29:29.529 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:29.522+0000 7f1a90ff9700 1 -- 172.21.15.110:0/191039404 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a84011600 con 0x7f1a940caf80 2023-10-06T03:29:29.529 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:29.522+0000 7f1a99384700 1 -- 172.21.15.110:0/191039404 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a9414eda0 con 0x7f1a940caf80 2023-10-06T03:29:29.530 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:29.522+0000 7f1a99384700 1 -- 172.21.15.110:0/191039404 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a94070ff0 con 0x7f1a940caf80 2023-10-06T03:29:29.531 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:29.522+0000 7f1a90ff9700 1 -- 172.21.15.110:0/191039404 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7f1a84011970 con 0x7f1a940caf80 2023-10-06T03:29:29.531 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:29.522+0000 7f1a90ff9700 1 --2- 172.21.15.110:0/191039404 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f1a74060550 0x7f1a74062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:29.531 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:29.522+0000 7f1a90ff9700 1 -- 172.21.15.110:0/191039404 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 5206+0+0 (secure 0 0 0) 0x7f1a8407c390 con 0x7f1a940caf80 2023-10-06T03:29:29.531 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:29.522+0000 7f1a8bfff700 1 --2- 172.21.15.110:0/191039404 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f1a74060550 0x7f1a74062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:29.531 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:29.522+0000 7f1a8bfff700 1 --2- 172.21.15.110:0/191039404 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f1a74060550 0x7f1a74062a00 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f1a941497c0 tx=0x7f1a7c007ec0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:29:29.536 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:29.530+0000 7f1a90ff9700 1 -- 172.21.15.110:0/191039404 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f1a84050600 con 0x7f1a940caf80 2023-10-06T03:29:29.703 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:29.698+0000 7f1a99384700 1 -- 172.21.15.110:0/191039404 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1a94155bd0 con 0x7f1a940caf80 2023-10-06T03:29:29.705 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:29.698+0000 7f1a90ff9700 1 -- 172.21.15.110:0/191039404 <== mon.0 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v19) v1 ==== 74+0+13708 (secure 0 0 0) 0x7f1a84050190 con 0x7f1a940caf80 2023-10-06T03:29:29.705 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-10-06T03:29:29.705 INFO:teuthology.orchestra.run.smithi110.stdout:{"epoch":19,"fsid":"93b8f6ac-4c92-4b45-b8e4-b59617024aa2","created":"2023-10-06T03:29:10.519583+0000","modified":"2023-10-06T03:29:29.360907+0000","last_up_change":"2023-10-06T03:29:21.960511+0000","last_in_change":"2023-10-06T03:29:12.560436+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-06T03:29:20.703743+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2023-10-06T03:29:28.307281+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"5866eb6c-2adc-46e4-be7a-e5bcda0f4065","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6824","nonce":33394},{"type":"v1","addr":"172.21.15.110:6825","nonce":33394}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6826","nonce":33394},{"type":"v1","addr":"172.21.15.110:6827","nonce":33394}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6830","nonce":33394},{"type":"v1","addr":"172.21.15.110:6831","nonce":33394}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6828","nonce":33394},{"type":"v1","addr":"172.21.15.110:6829","nonce":33394}]},"public_addr":"172.21.15.110:6825/33394","cluster_addr":"172.21.15.110:6827/33394","heartbeat_back_addr":"172.21.15.110:6831/33394","heartbeat_front_addr":"172.21.15.110:6829/33394","state":["exists","up"]},{"osd":1,"uuid":"a38de118-062b-4a05-9782-2ceddf6ff4df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6800","nonce":33396},{"type":"v1","addr":"172.21.15.110:6801","nonce":33396}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6802","nonce":33396},{"type":"v1","addr":"172.21.15.110:6803","nonce":33396}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6806","nonce":33396},{"type":"v1","addr":"172.21.15.110:6807","nonce":33396}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6804","nonce":33396},{"type":"v1","addr":"172.21.15.110:6805","nonce":33396}]},"public_addr":"172.21.15.110:6801/33396","cluster_addr":"172.21.15.110:6803/33396","heartbeat_back_addr":"172.21.15.110:6807/33396","heartbeat_front_addr":"172.21.15.110:6805/33396","state":["exists","up"]},{"osd":2,"uuid":"56e39be2-3407-4b69-877e-cf16617b06d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6816","nonce":33397},{"type":"v1","addr":"172.21.15.110:6817","nonce":33397}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6818","nonce":33397},{"type":"v1","addr":"172.21.15.110:6819","nonce":33397}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6822","nonce":33397},{"type":"v1","addr":"172.21.15.110:6823","nonce":33397}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6820","nonce":33397},{"type":"v1","addr":"172.21.15.110:6821","nonce":33397}]},"public_addr":"172.21.15.110:6817/33397","cluster_addr":"172.21.15.110:6819/33397","heartbeat_back_addr":"172.21.15.110:6823/33397","heartbeat_front_addr":"172.21.15.110:6821/33397","state":["exists","up"]},{"osd":3,"uuid":"01c82d13-c1d4-4bde-b55e-3e0089b40115","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6808","nonce":33395},{"type":"v1","addr":"172.21.15.110:6809","nonce":33395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6810","nonce":33395},{"type":"v1","addr":"172.21.15.110:6811","nonce":33395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6814","nonce":33395},{"type":"v1","addr":"172.21.15.110:6815","nonce":33395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6812","nonce":33395},{"type":"v1","addr":"172.21.15.110:6813","nonce":33395}]},"public_addr":"172.21.15.110:6809/33395","cluster_addr":"172.21.15.110:6811/33395","heartbeat_back_addr":"172.21.15.110:6815/33395","heartbeat_front_addr":"172.21.15.110:6813/33395","state":["exists","up"]},{"osd":4,"uuid":"86983430-5fa3-4d17-998f-03e2cd843237","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":33683},{"type":"v1","addr":"172.21.15.192:6817","nonce":33683}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":33683},{"type":"v1","addr":"172.21.15.192:6819","nonce":33683}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":33683},{"type":"v1","addr":"172.21.15.192:6823","nonce":33683}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6820","nonce":33683},{"type":"v1","addr":"172.21.15.192:6821","nonce":33683}]},"public_addr":"172.21.15.192:6817/33683","cluster_addr":"172.21.15.192:6819/33683","heartbeat_back_addr":"172.21.15.192:6823/33683","heartbeat_front_addr":"172.21.15.192:6821/33683","state":["exists","up"]},{"osd":5,"uuid":"6945257f-821d-4619-a63f-ebe857eba6f4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":33684},{"type":"v1","addr":"172.21.15.192:6801","nonce":33684}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":33684},{"type":"v1","addr":"172.21.15.192:6803","nonce":33684}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":33684},{"type":"v1","addr":"172.21.15.192:6807","nonce":33684}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6804","nonce":33684},{"type":"v1","addr":"172.21.15.192:6805","nonce":33684}]},"public_addr":"172.21.15.192:6801/33684","cluster_addr":"172.21.15.192:6803/33684","heartbeat_back_addr":"172.21.15.192:6807/33684","heartbeat_front_addr":"172.21.15.192:6805/33684","state":["exists","up"]},{"osd":6,"uuid":"7854d1ae-7060-4780-8f97-73cea5e7a12f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6824","nonce":33686},{"type":"v1","addr":"172.21.15.192:6825","nonce":33686}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":33686},{"type":"v1","addr":"172.21.15.192:6827","nonce":33686}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":33686},{"type":"v1","addr":"172.21.15.192:6831","nonce":33686}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6828","nonce":33686},{"type":"v1","addr":"172.21.15.192:6829","nonce":33686}]},"public_addr":"172.21.15.192:6825/33686","cluster_addr":"172.21.15.192:6827/33686","heartbeat_back_addr":"172.21.15.192:6831/33686","heartbeat_front_addr":"172.21.15.192:6829/33686","state":["exists","up"]},{"osd":7,"uuid":"f1ebdfd5-b859-482b-9f97-c18c4a26f445","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6808","nonce":33685},{"type":"v1","addr":"172.21.15.192:6809","nonce":33685}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":33685},{"type":"v1","addr":"172.21.15.192:6811","nonce":33685}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":33685},{"type":"v1","addr":"172.21.15.192:6815","nonce":33685}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6812","nonce":33685},{"type":"v1","addr":"172.21.15.192:6813","nonce":33685}]},"public_addr":"172.21.15.192:6809/33685","cluster_addr":"172.21.15.192:6811/33685","heartbeat_back_addr":"172.21.15.192:6815/33685","heartbeat_front_addr":"172.21.15.192:6813/33685","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T03:29:18.002048+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T03:29:17.478576+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T03:29:18.034944+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T03:29:17.628513+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T03:29:18.047338+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T03:29:17.772349+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T03:29:17.883958+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T03:29:18.151838+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-06T03:29:29.707 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:29.702+0000 7f1a99384700 1 -- 172.21.15.110:0/191039404 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f1a74060550 msgr2=0x7f1a74062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:29.707 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:29.702+0000 7f1a99384700 1 --2- 172.21.15.110:0/191039404 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f1a74060550 0x7f1a74062a00 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f1a941497c0 tx=0x7f1a7c007ec0 comp rx=0 tx=0).stop 2023-10-06T03:29:29.708 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:29.702+0000 7f1a99384700 1 -- 172.21.15.110:0/191039404 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1a940caf80 msgr2=0x7f1a94147e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:29.708 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:29.702+0000 7f1a99384700 1 --2- 172.21.15.110:0/191039404 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1a940caf80 0x7f1a94147e50 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f1a8400c830 tx=0x7f1a84008040 comp rx=0 tx=0).stop 2023-10-06T03:29:29.708 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:29.702+0000 7f1a99384700 1 -- 172.21.15.110:0/191039404 shutdown_connections 2023-10-06T03:29:29.708 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:29.702+0000 7f1a99384700 1 --2- 172.21.15.110:0/191039404 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f1a74060550 0x7f1a74062a00 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:29.708 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:29.702+0000 7f1a99384700 1 --2- 172.21.15.110:0/191039404 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1a941566b0 0x7f1a9414fb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:29.708 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:29.702+0000 7f1a99384700 1 --2- 172.21.15.110:0/191039404 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1a941550c0 0x7f1a9414c680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:29.708 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:29.702+0000 7f1a99384700 1 --2- 172.21.15.110:0/191039404 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1a940caf80 0x7f1a94147e50 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:29.708 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:29.702+0000 7f1a99384700 1 -- 172.21.15.110:0/191039404 >> 172.21.15.110:0/191039404 conn(0x7f1a940c8d00 msgr2=0x7f1a940c1160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:29:29.708 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:29.702+0000 7f1a99384700 1 -- 172.21.15.110:0/191039404 shutdown_connections 2023-10-06T03:29:29.708 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:29.702+0000 7f1a99384700 1 -- 172.21.15.110:0/191039404 wait complete. 2023-10-06T03:29:29.718 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-06T03:29:20.703743+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-10-06T03:29:28.307281+0000', 'flags': 32769, 'flags_names': 'hashpspool,creating', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}] 2023-10-06T03:29:29.719 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-06T03:29:29.830 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:29.822+0000 7ff97e2e1700 1 Processor -- start 2023-10-06T03:29:29.830 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:29.822+0000 7ff97e2e1700 1 -- start start 2023-10-06T03:29:29.831 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:29.826+0000 7ff97e2e1700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff97806b7c0 0x7ff9780ce170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:29.831 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:29.826+0000 7ff97e2e1700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9780cfa00 con 0x7ff97806c2a0 2023-10-06T03:29:29.831 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:29.826+0000 7ff97e2e1700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9780cedf0 con 0x7ff9780ce6b0 2023-10-06T03:29:29.831 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:29.826+0000 7ff97e2e1700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9780cb1c0 con 0x7ff97806b7c0 2023-10-06T03:29:29.832 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:29.826+0000 7ff97dae0700 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7ff97806c2a0 0x7ff978162ac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:52522/0 (socket says 172.21.15.110:52522) 2023-10-06T03:29:29.832 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:29.826+0000 7ff97dae0700 1 -- 172.21.15.110:0/3512383838 learned_addr learned my addr 172.21.15.110:0/3512383838 (peer_addr_for_me v1:172.21.15.110:0/0) 2023-10-06T03:29:29.835 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:29.826+0000 7ff97d2df700 1 --2- 172.21.15.110:0/3512383838 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff97806b7c0 0x7ff9780ce170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:29.835 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:29.826+0000 7ff96ffff700 1 -- 172.21.15.110:0/3512383838 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1878425872 0 0) 0x7ff9780cfa00 con 0x7ff97806c2a0 2023-10-06T03:29:29.835 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:29.826+0000 7ff96ffff700 1 -- 172.21.15.110:0/3512383838 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff964003570 con 0x7ff97806c2a0 2023-10-06T03:29:29.835 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:29.826+0000 7ff96ffff700 1 -- 172.21.15.110:0/3512383838 <== mon.0 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1937569597 0 0) 0x7ff964003570 con 0x7ff97806c2a0 2023-10-06T03:29:29.835 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:29.826+0000 7ff96ffff700 1 -- 172.21.15.110:0/3512383838 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff97806b7c0 msgr2=0x7ff9780ce170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:29:29.835 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:29.826+0000 7ff96ffff700 1 --2- 172.21.15.110:0/3512383838 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff97806b7c0 0x7ff9780ce170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:29.835 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:29.826+0000 7ff96ffff700 1 -- 172.21.15.110:0/3512383838 >> v1:172.21.15.192:6789/0 conn(0x7ff9780ce6b0 legacy=0x7ff97806bb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:29.835 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:29.826+0000 7ff96ffff700 1 -- 172.21.15.110:0/3512383838 --> v1:172.21.15.110:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9780cf0d0 con 0x7ff97806c2a0 2023-10-06T03:29:29.836 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:29.826+0000 7ff96ffff700 1 -- 172.21.15.110:0/3512383838 <== mon.0 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 118603703 0 0) 0x7ff9700017f0 con 0x7ff97806c2a0 2023-10-06T03:29:29.836 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:29.826+0000 7ff96ffff700 1 -- 172.21.15.110:0/3512383838 >> v1:172.21.15.110:6789/0 conn(0x7ff97806c2a0 legacy=0x7ff978162ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:29.836 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:29.826+0000 7ff96ffff700 1 --2- 172.21.15.110:0/3512383838 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff964003a90 0x7ff964003e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:29.836 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:29.826+0000 7ff96ffff700 1 --2- 172.21.15.110:0/3512383838 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff964004810 0x7ff964004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:29.836 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:29.826+0000 7ff96ffff700 1 --2- 172.21.15.110:0/3512383838 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff97806b7c0 0x7ff9640051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:29.836 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:29.826+0000 7ff96ffff700 1 -- 172.21.15.110:0/3512383838 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9780cb1c0 con 0x7ff964003a90 2023-10-06T03:29:29.836 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:29.826+0000 7ff96ffff700 1 -- 172.21.15.110:0/3512383838 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff96400bb10 con 0x7ff964004810 2023-10-06T03:29:29.836 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:29.826+0000 7ff96ffff700 1 -- 172.21.15.110:0/3512383838 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff96400bcd0 con 0x7ff97806b7c0 2023-10-06T03:29:29.836 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:29.826+0000 7ff97d2df700 1 --2- 172.21.15.110:0/3512383838 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff964003a90 0x7ff964003e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:29.836 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:29.826+0000 7ff97cade700 1 --2- 172.21.15.110:0/3512383838 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff964004810 0x7ff964004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:29.836 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:29.826+0000 7ff97dae0700 1 --2- 172.21.15.110:0/3512383838 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff97806b7c0 0x7ff9640051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:29.837 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:29.826+0000 7ff97d2df700 1 -- 172.21.15.110:0/3512383838 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff97806b7c0 msgr2=0x7ff9640051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:29:29.837 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:29.826+0000 7ff97d2df700 1 --2- 172.21.15.110:0/3512383838 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff97806b7c0 0x7ff9640051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:29.837 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:29.826+0000 7ff97d2df700 1 -- 172.21.15.110:0/3512383838 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff964004810 msgr2=0x7ff964004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:29:29.837 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:29.826+0000 7ff97d2df700 1 --2- 172.21.15.110:0/3512383838 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff964004810 0x7ff964004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:29.837 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:29.826+0000 7ff97d2df700 1 -- 172.21.15.110:0/3512383838 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff964003570 con 0x7ff964003a90 2023-10-06T03:29:29.837 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:29.826+0000 7ff97cade700 1 --2- 172.21.15.110:0/3512383838 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff964004810 0x7ff964004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-06T03:29:29.837 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:29.826+0000 7ff97dae0700 1 --2- 172.21.15.110:0/3512383838 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff97806b7c0 0x7ff9640051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-06T03:29:29.837 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:29.826+0000 7ff97d2df700 1 --2- 172.21.15.110:0/3512383838 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff964003a90 0x7ff964003e60 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7ff968009c00 tx=0x7ff9680091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df31bead78f75c15 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:29:29.837 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:29.826+0000 7ff96ffff700 1 -- 172.21.15.110:0/3512383838 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff96800e710 con 0x7ff964003a90 2023-10-06T03:29:29.837 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:29.826+0000 7ff96ffff700 1 -- 172.21.15.110:0/3512383838 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff96800e870 con 0x7ff964003a90 2023-10-06T03:29:29.837 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:29.826+0000 7ff96ffff700 1 -- 172.21.15.110:0/3512383838 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff96800ec30 con 0x7ff964003a90 2023-10-06T03:29:29.838 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:29.826+0000 7ff97e2e1700 1 -- 172.21.15.110:0/3512383838 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff964003a90 msgr2=0x7ff964003e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:29.838 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:29.826+0000 7ff97e2e1700 1 --2- 172.21.15.110:0/3512383838 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff964003a90 0x7ff964003e60 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7ff968009c00 tx=0x7ff9680091a0 comp rx=0 tx=0).stop 2023-10-06T03:29:29.838 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:29.826+0000 7ff97dae0700 1 -- 172.21.15.110:0/3512383838 reap_dead start 2023-10-06T03:29:29.838 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:29.826+0000 7ff97e2e1700 1 -- 172.21.15.110:0/3512383838 shutdown_connections 2023-10-06T03:29:29.838 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:29.826+0000 7ff97e2e1700 1 -- 172.21.15.110:0/3512383838 >> 172.21.15.110:0/3512383838 conn(0x7ff9780cfea0 msgr2=0x7ff9780d02a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:29:29.838 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:29.826+0000 7ff97e2e1700 1 -- 172.21.15.110:0/3512383838 shutdown_connections 2023-10-06T03:29:29.838 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:29.826+0000 7ff97e2e1700 1 -- 172.21.15.110:0/3512383838 wait complete. 2023-10-06T03:29:29.840 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:29.830+0000 7ff97e2e1700 1 Processor -- start 2023-10-06T03:29:29.840 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:29.830+0000 7ff97e2e1700 1 -- start start 2023-10-06T03:29:29.840 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:29.830+0000 7ff97e2e1700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff97814ebc0 0x7ff9780b4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:29.840 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:29.830+0000 7ff97e2e1700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff97814ef90 0x7ff9780b52d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:29.840 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:29.830+0000 7ff97e2e1700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff978155fd0 0x7ff978156420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:29.840 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:29.830+0000 7ff97e2e1700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff978157910 con 0x7ff97814ebc0 2023-10-06T03:29:29.841 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:29.830+0000 7ff97e2e1700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff978156d00 con 0x7ff978155fd0 2023-10-06T03:29:29.841 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:29.830+0000 7ff97e2e1700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff978156fe0 con 0x7ff97814ef90 2023-10-06T03:29:29.841 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:29.830+0000 7ff97d2df700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff97814ebc0 0x7ff9780b4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:29.841 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:29.830+0000 7ff97d2df700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff97814ebc0 0x7ff9780b4d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:38898/0 (socket says 172.21.15.110:38898) 2023-10-06T03:29:29.841 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:29.830+0000 7ff97d2df700 1 -- 172.21.15.110:0/2781807665 learned_addr learned my addr 172.21.15.110:0/2781807665 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:29:29.841 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:29.830+0000 7ff97dae0700 1 --2- 172.21.15.110:0/2781807665 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff978155fd0 0x7ff978156420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:29.841 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:29.830+0000 7ff97cade700 1 --2- 172.21.15.110:0/2781807665 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff97814ef90 0x7ff9780b52d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:29.841 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:29.830+0000 7ff97d2df700 1 -- 172.21.15.110:0/2781807665 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff97814ef90 msgr2=0x7ff9780b52d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:29.841 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:29.830+0000 7ff97d2df700 1 --2- 172.21.15.110:0/2781807665 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff97814ef90 0x7ff9780b52d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:29.841 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:29.830+0000 7ff97d2df700 1 -- 172.21.15.110:0/2781807665 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff978155fd0 msgr2=0x7ff978156420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:29.842 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:29.830+0000 7ff97d2df700 1 --2- 172.21.15.110:0/2781807665 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff978155fd0 0x7ff978156420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:29.842 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:29.830+0000 7ff97d2df700 1 -- 172.21.15.110:0/2781807665 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9781572c0 con 0x7ff97814ebc0 2023-10-06T03:29:29.842 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:29.834+0000 7ff97d2df700 1 --2- 172.21.15.110:0/2781807665 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff97814ebc0 0x7ff9780b4d90 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7ff964004770 tx=0x7ff96801a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:29:29.842 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:29.834+0000 7ff96d7fa700 1 -- 172.21.15.110:0/2781807665 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff968022070 con 0x7ff97814ebc0 2023-10-06T03:29:29.842 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:29.834+0000 7ff96d7fa700 1 -- 172.21.15.110:0/2781807665 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff96800e9e0 con 0x7ff97814ebc0 2023-10-06T03:29:29.842 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:29.834+0000 7ff97e2e1700 1 -- 172.21.15.110:0/2781807665 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9781575a0 con 0x7ff97814ebc0 2023-10-06T03:29:29.842 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:29.834+0000 7ff96d7fa700 1 -- 172.21.15.110:0/2781807665 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff96800e710 con 0x7ff97814ebc0 2023-10-06T03:29:29.842 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:29.834+0000 7ff97e2e1700 1 -- 172.21.15.110:0/2781807665 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9781593d0 con 0x7ff97814ebc0 2023-10-06T03:29:29.843 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:29.834+0000 7ff97e2e1700 1 -- 172.21.15.110:0/2781807665 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff978070ff0 con 0x7ff97814ebc0 2023-10-06T03:29:29.843 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:29.834+0000 7ff96d7fa700 1 -- 172.21.15.110:0/2781807665 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7ff96801c660 con 0x7ff97814ebc0 2023-10-06T03:29:29.843 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:29.834+0000 7ff96d7fa700 1 --2- 172.21.15.110:0/2781807665 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7ff95c060580 0x7ff95c062a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:29.843 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:29.834+0000 7ff96d7fa700 1 -- 172.21.15.110:0/2781807665 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 5206+0+0 (secure 0 0 0) 0x7ff96807db20 con 0x7ff97814ebc0 2023-10-06T03:29:29.843 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:29.838+0000 7ff97cade700 1 --2- 172.21.15.110:0/2781807665 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7ff95c060580 0x7ff95c062a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:29.844 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:29.838+0000 7ff97cade700 1 --2- 172.21.15.110:0/2781807665 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7ff95c060580 0x7ff95c062a30 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7ff974007a60 tx=0x7ff974009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:29:29.849 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:29.842+0000 7ff96d7fa700 1 -- 172.21.15.110:0/2781807665 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7ff968051e50 con 0x7ff97814ebc0 2023-10-06T03:29:30.011 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:30.006+0000 7ff97e2e1700 1 -- 172.21.15.110:0/2781807665 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7ff9780cf3b0 con 0x7ff97814ebc0 2023-10-06T03:29:30.012 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:30.006+0000 7ff96d7fa700 1 -- 172.21.15.110:0/2781807665 <== mon.0 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v19) v1 ==== 93+0+10 (secure 0 0 0) 0x7ff9680170a0 con 0x7ff97814ebc0 2023-10-06T03:29:30.012 INFO:teuthology.orchestra.run.smithi110.stdout:pg_num: 1 2023-10-06T03:29:30.015 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:30.006+0000 7ff97e2e1700 1 -- 172.21.15.110:0/2781807665 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7ff95c060580 msgr2=0x7ff95c062a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:30.015 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:30.006+0000 7ff97e2e1700 1 --2- 172.21.15.110:0/2781807665 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7ff95c060580 0x7ff95c062a30 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7ff974007a60 tx=0x7ff974009040 comp rx=0 tx=0).stop 2023-10-06T03:29:30.015 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:30.006+0000 7ff97e2e1700 1 -- 172.21.15.110:0/2781807665 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff97814ebc0 msgr2=0x7ff9780b4d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:30.015 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:30.006+0000 7ff97e2e1700 1 --2- 172.21.15.110:0/2781807665 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff97814ebc0 0x7ff9780b4d90 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7ff964004770 tx=0x7ff96801a000 comp rx=0 tx=0).stop 2023-10-06T03:29:30.015 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:30.006+0000 7ff97e2e1700 1 -- 172.21.15.110:0/2781807665 shutdown_connections 2023-10-06T03:29:30.015 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:30.006+0000 7ff97e2e1700 1 --2- 172.21.15.110:0/2781807665 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7ff95c060580 0x7ff95c062a30 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:30.015 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:30.006+0000 7ff97e2e1700 1 --2- 172.21.15.110:0/2781807665 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff978155fd0 0x7ff978156420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:30.015 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:30.006+0000 7ff97e2e1700 1 --2- 172.21.15.110:0/2781807665 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff97814ef90 0x7ff9780b52d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:30.016 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:30.006+0000 7ff97e2e1700 1 --2- 172.21.15.110:0/2781807665 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff97814ebc0 0x7ff9780b4d90 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:30.016 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:30.006+0000 7ff97e2e1700 1 -- 172.21.15.110:0/2781807665 >> 172.21.15.110:0/2781807665 conn(0x7ff9780cfea0 msgr2=0x7ff9780b0db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:29:30.016 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:30.006+0000 7ff97e2e1700 1 -- 172.21.15.110:0/2781807665 shutdown_connections 2023-10-06T03:29:30.016 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:30.006+0000 7ff97e2e1700 1 -- 172.21.15.110:0/2781807665 wait complete. 2023-10-06T03:29:30.025 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-10-06T03:29:30.136 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:30.130+0000 7f558759e700 1 Processor -- start 2023-10-06T03:29:30.136 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:30.130+0000 7f558759e700 1 -- start start 2023-10-06T03:29:30.138 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:30.130+0000 7f558759e700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f558806a560 0x7f558815cd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:30.138 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:30.130+0000 7f558759e700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f558815d2d0 0x7f55880cad40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:30.138 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:30.130+0000 7f558759e700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f558806a9a0 con 0x7f558806a190 2023-10-06T03:29:30.138 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:30.130+0000 7f558759e700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f558815dc80 con 0x7f558806a560 2023-10-06T03:29:30.138 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:30.130+0000 7f558759e700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55880ca8b0 con 0x7f558815d2d0 2023-10-06T03:29:30.138 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:30.130+0000 7f558659c700 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f558806a190 0x7f55880c73d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:52538/0 (socket says 172.21.15.110:52538) 2023-10-06T03:29:30.138 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:30.130+0000 7f558659c700 1 -- 172.21.15.110:0/186218775 learned_addr learned my addr 172.21.15.110:0/186218775 (peer_addr_for_me v1:172.21.15.110:0/0) 2023-10-06T03:29:30.139 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:30.130+0000 7f558559a700 1 -- 172.21.15.110:0/186218775 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1095917580 0 0) 0x7f558806a9a0 con 0x7f558806a190 2023-10-06T03:29:30.139 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:30.130+0000 7f5585d9b700 1 --2- 172.21.15.110:0/186218775 >> v2:172.21.15.192:3300/0 conn(0x7f558806a560 0x7f558815cd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:30.139 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:30.130+0000 7f5586d9d700 1 --2- 172.21.15.110:0/186218775 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f558815d2d0 0x7f55880cad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:30.139 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:30.130+0000 7f558559a700 1 -- 172.21.15.110:0/186218775 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5570003570 con 0x7f558806a190 2023-10-06T03:29:30.141 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:30.130+0000 7f558559a700 1 -- 172.21.15.110:0/186218775 <== mon.0 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1299578319 0 0) 0x7f5570003570 con 0x7f558806a190 2023-10-06T03:29:30.141 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:30.130+0000 7f558559a700 1 -- 172.21.15.110:0/186218775 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f558815d2d0 msgr2=0x7f55880cad40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:29:30.141 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:30.130+0000 7f558559a700 1 --2- 172.21.15.110:0/186218775 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f558815d2d0 0x7f55880cad40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:30.141 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:30.130+0000 7f558559a700 1 -- 172.21.15.110:0/186218775 >> v2:172.21.15.192:3300/0 conn(0x7f558806a560 msgr2=0x7f558815cd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:29:30.141 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:30.130+0000 7f558559a700 1 --2- 172.21.15.110:0/186218775 >> v2:172.21.15.192:3300/0 conn(0x7f558806a560 0x7f558815cd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:30.142 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:30.130+0000 7f558559a700 1 -- 172.21.15.110:0/186218775 --> v1:172.21.15.110:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f558815d710 con 0x7f558806a190 2023-10-06T03:29:30.142 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:30.130+0000 7f558559a700 1 -- 172.21.15.110:0/186218775 <== mon.0 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 118603703 0 0) 0x7f55780017f0 con 0x7f558806a190 2023-10-06T03:29:30.142 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:30.130+0000 7f558559a700 1 -- 172.21.15.110:0/186218775 >> v1:172.21.15.110:6789/0 conn(0x7f558806a190 legacy=0x7f55880c73d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:30.142 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:30.134+0000 7f558559a700 1 --2- 172.21.15.110:0/186218775 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5570003a60 0x7f5570003e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:30.142 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:30.134+0000 7f558559a700 1 --2- 172.21.15.110:0/186218775 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f558815d2d0 0x7f5570004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:30.142 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:30.134+0000 7f558559a700 1 --2- 172.21.15.110:0/186218775 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5570004d50 0x7f55700051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:30.142 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:30.134+0000 7f558559a700 1 -- 172.21.15.110:0/186218775 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f558815dc80 con 0x7f5570003a60 2023-10-06T03:29:30.142 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:30.134+0000 7f558559a700 1 -- 172.21.15.110:0/186218775 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55880ca8b0 con 0x7f5570004d50 2023-10-06T03:29:30.142 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:30.134+0000 7f558559a700 1 -- 172.21.15.110:0/186218775 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f557000ba60 con 0x7f558815d2d0 2023-10-06T03:29:30.143 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:30.134+0000 7f558659c700 1 --2- 172.21.15.110:0/186218775 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5570003a60 0x7f5570003e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:30.143 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:30.134+0000 7f5586d9d700 1 --2- 172.21.15.110:0/186218775 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5570004d50 0x7f55700051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:30.143 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:30.134+0000 7f5585d9b700 1 --2- 172.21.15.110:0/186218775 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f558815d2d0 0x7f5570004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:30.143 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:30.134+0000 7f558659c700 1 -- 172.21.15.110:0/186218775 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f558815d2d0 msgr2=0x7f5570004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:29:30.143 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:30.134+0000 7f558659c700 1 --2- 172.21.15.110:0/186218775 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f558815d2d0 0x7f5570004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:30.143 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:30.134+0000 7f558659c700 1 -- 172.21.15.110:0/186218775 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5570004d50 msgr2=0x7f55700051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:29:30.143 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:30.134+0000 7f558659c700 1 --2- 172.21.15.110:0/186218775 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5570004d50 0x7f55700051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:30.143 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:30.134+0000 7f558659c700 1 -- 172.21.15.110:0/186218775 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5570003570 con 0x7f5570003a60 2023-10-06T03:29:30.143 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:30.134+0000 7f5585d9b700 1 --2- 172.21.15.110:0/186218775 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f558815d2d0 0x7f5570004810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-06T03:29:30.143 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:30.134+0000 7f558659c700 1 --2- 172.21.15.110:0/186218775 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5570003a60 0x7f5570003e30 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f5578005bb0 tx=0x7f55780051c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55abd51331b5f09d server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:29:30.143 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:30.134+0000 7f558559a700 1 -- 172.21.15.110:0/186218775 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55780054f0 con 0x7f5570003a60 2023-10-06T03:29:30.144 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:30.134+0000 7f558559a700 1 -- 172.21.15.110:0/186218775 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f557800d980 con 0x7f5570003a60 2023-10-06T03:29:30.144 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:30.134+0000 7f558559a700 1 -- 172.21.15.110:0/186218775 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f557800dd20 con 0x7f5570003a60 2023-10-06T03:29:30.145 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:30.134+0000 7f558759e700 1 -- 172.21.15.110:0/186218775 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5570003a60 msgr2=0x7f5570003e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:30.145 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:30.134+0000 7f558759e700 1 --2- 172.21.15.110:0/186218775 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5570003a60 0x7f5570003e30 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f5578005bb0 tx=0x7f55780051c0 comp rx=0 tx=0).stop 2023-10-06T03:29:30.145 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:30.134+0000 7f5586d9d700 1 -- 172.21.15.110:0/186218775 reap_dead start 2023-10-06T03:29:30.145 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:30.134+0000 7f558759e700 1 -- 172.21.15.110:0/186218775 shutdown_connections 2023-10-06T03:29:30.145 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:30.134+0000 7f558759e700 1 -- 172.21.15.110:0/186218775 >> 172.21.15.110:0/186218775 conn(0x7f5588067b70 msgr2=0x7f55880cc560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:29:30.145 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:30.134+0000 7f558759e700 1 -- 172.21.15.110:0/186218775 shutdown_connections 2023-10-06T03:29:30.145 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:30.134+0000 7f558759e700 1 -- 172.21.15.110:0/186218775 wait complete. 2023-10-06T03:29:30.145 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:30.138+0000 7f558759e700 1 Processor -- start 2023-10-06T03:29:30.145 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:30.138+0000 7f558759e700 1 -- start start 2023-10-06T03:29:30.148 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:30.138+0000 7f558759e700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5570003a60 0x7f5588151df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:30.148 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:30.138+0000 7f558759e700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f558814ad20 0x7f55880c2540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:30.148 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:30.138+0000 7f558759e700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f558814b190 0x7f55880acf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:30.148 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:30.138+0000 7f558759e700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55880c20b0 con 0x7f558814b190 2023-10-06T03:29:30.148 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:30.138+0000 7f558759e700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55880c14a0 con 0x7f558814ad20 2023-10-06T03:29:30.149 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:30.138+0000 7f558759e700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55880c1780 con 0x7f5570003a60 2023-10-06T03:29:30.149 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:30.138+0000 7f5586d9d700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f558814b190 0x7f55880acf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:30.149 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:30.138+0000 7f5586d9d700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f558814b190 0x7f55880acf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:38914/0 (socket says 172.21.15.110:38914) 2023-10-06T03:29:30.149 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:30.138+0000 7f5586d9d700 1 -- 172.21.15.110:0/1416845739 learned_addr learned my addr 172.21.15.110:0/1416845739 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:29:30.149 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:30.138+0000 7f558659c700 1 --2- 172.21.15.110:0/1416845739 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5570003a60 0x7f5588151df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:30.149 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:30.138+0000 7f5585d9b700 1 --2- 172.21.15.110:0/1416845739 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f558814ad20 0x7f55880c2540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:30.149 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:30.138+0000 7f5586d9d700 1 -- 172.21.15.110:0/1416845739 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5570003a60 msgr2=0x7f5588151df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:30.149 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:30.138+0000 7f5586d9d700 1 --2- 172.21.15.110:0/1416845739 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5570003a60 0x7f5588151df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:30.149 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:30.138+0000 7f5586d9d700 1 -- 172.21.15.110:0/1416845739 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f558814ad20 msgr2=0x7f55880c2540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:30.149 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:30.138+0000 7f5586d9d700 1 --2- 172.21.15.110:0/1416845739 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f558814ad20 0x7f55880c2540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:30.150 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:30.138+0000 7f5586d9d700 1 -- 172.21.15.110:0/1416845739 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55880c1a60 con 0x7f558814b190 2023-10-06T03:29:30.150 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:30.138+0000 7f5586d9d700 1 --2- 172.21.15.110:0/1416845739 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f558814b190 0x7f55880acf90 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f5588069ce0 tx=0x7f55800091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:29:30.150 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:30.138+0000 7f55767fc700 1 -- 172.21.15.110:0/1416845739 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5580011ed0 con 0x7f558814b190 2023-10-06T03:29:30.150 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:30.138+0000 7f55767fc700 1 -- 172.21.15.110:0/1416845739 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f558000f8b0 con 0x7f558814b190 2023-10-06T03:29:30.150 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:30.138+0000 7f558759e700 1 -- 172.21.15.110:0/1416845739 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f55880c1d40 con 0x7f558814b190 2023-10-06T03:29:30.150 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:30.138+0000 7f55767fc700 1 -- 172.21.15.110:0/1416845739 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f558000fcb0 con 0x7f558814b190 2023-10-06T03:29:30.150 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:30.138+0000 7f558759e700 1 -- 172.21.15.110:0/1416845739 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f55880c2a80 con 0x7f558814b190 2023-10-06T03:29:30.150 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:30.142+0000 7f558759e700 1 -- 172.21.15.110:0/1416845739 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f55880c1f00 con 0x7f558814b190 2023-10-06T03:29:30.151 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:30.142+0000 7f55767fc700 1 -- 172.21.15.110:0/1416845739 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7f5580014650 con 0x7f558814b190 2023-10-06T03:29:30.151 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:30.142+0000 7f55767fc700 1 --2- 172.21.15.110:0/1416845739 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f5568060580 0x7f5568062a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:30.151 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:30.142+0000 7f55767fc700 1 -- 172.21.15.110:0/1416845739 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 5206+0+0 (secure 0 0 0) 0x7f558007cc40 con 0x7f558814b190 2023-10-06T03:29:30.151 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:30.142+0000 7f558659c700 1 --2- 172.21.15.110:0/1416845739 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f5568060580 0x7f5568062a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:30.151 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:30.142+0000 7f558659c700 1 --2- 172.21.15.110:0/1416845739 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f5568060580 0x7f5568062a30 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f5578005200 tx=0x7f5578016000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:29:30.156 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:30.150+0000 7f55767fc700 1 -- 172.21.15.110:0/1416845739 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f5580050f30 con 0x7f558814b190 2023-10-06T03:29:30.316 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:30.310+0000 7f558759e700 1 -- 172.21.15.110:0/1416845739 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f5588070ff0 con 0x7f558814b190 2023-10-06T03:29:30.321 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:30.314+0000 7f55767fc700 1 -- 172.21.15.110:0/1416845739 <== mon.0 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v19) v1 ==== 92+0+10 (secure 0 0 0) 0x7f55800220a0 con 0x7f558814b190 2023-10-06T03:29:30.321 INFO:teuthology.orchestra.run.smithi110.stdout:pg_num: 8 2023-10-06T03:29:30.324 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:30.318+0000 7f558759e700 1 -- 172.21.15.110:0/1416845739 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f5568060580 msgr2=0x7f5568062a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:30.324 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:30.318+0000 7f558759e700 1 --2- 172.21.15.110:0/1416845739 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f5568060580 0x7f5568062a30 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f5578005200 tx=0x7f5578016000 comp rx=0 tx=0).stop 2023-10-06T03:29:30.324 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:30.318+0000 7f558759e700 1 -- 172.21.15.110:0/1416845739 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f558814b190 msgr2=0x7f55880acf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:30.324 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:30.318+0000 7f558759e700 1 --2- 172.21.15.110:0/1416845739 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f558814b190 0x7f55880acf90 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f5588069ce0 tx=0x7f55800091a0 comp rx=0 tx=0).stop 2023-10-06T03:29:30.324 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:30.318+0000 7f558759e700 1 -- 172.21.15.110:0/1416845739 shutdown_connections 2023-10-06T03:29:30.324 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:30.318+0000 7f558759e700 1 --2- 172.21.15.110:0/1416845739 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f5568060580 0x7f5568062a30 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:30.324 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:30.318+0000 7f558759e700 1 --2- 172.21.15.110:0/1416845739 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f558814b190 0x7f55880acf90 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:30.324 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:30.318+0000 7f558759e700 1 --2- 172.21.15.110:0/1416845739 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f558814ad20 0x7f55880c2540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:30.324 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:30.318+0000 7f558759e700 1 --2- 172.21.15.110:0/1416845739 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5570003a60 0x7f5588151df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:30.325 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:30.318+0000 7f558759e700 1 -- 172.21.15.110:0/1416845739 >> 172.21.15.110:0/1416845739 conn(0x7f5588067b70 msgr2=0x7f5588153810 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:29:30.325 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:30.318+0000 7f558759e700 1 -- 172.21.15.110:0/1416845739 shutdown_connections 2023-10-06T03:29:30.325 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:30.318+0000 7f558759e700 1 -- 172.21.15.110:0/1416845739 wait complete. 2023-10-06T03:29:30.337 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-06T03:29:30.448 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:30.442+0000 7f16c309a700 1 Processor -- start 2023-10-06T03:29:30.448 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:30.442+0000 7f16c309a700 1 -- start start 2023-10-06T03:29:30.451 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:30.442+0000 7f16c309a700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f16bc155750 0x7f16bc0afc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:30.451 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:30.442+0000 7f16c309a700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f16bc0c6ac0 0x7f16bc0b13d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:30.451 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:30.442+0000 7f16c309a700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16bc0c9d40 con 0x7f16bc0b1910 2023-10-06T03:29:30.451 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:30.442+0000 7f16c309a700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f16bc063b40 con 0x7f16bc0c6ac0 2023-10-06T03:29:30.451 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:30.442+0000 7f16c309a700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f16bc0b1d20 con 0x7f16bc155750 2023-10-06T03:29:30.451 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:30.442+0000 7f16bbfff700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f16bc0c6ac0 0x7f16bc0b13d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:30.452 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:30.442+0000 7f16c1637700 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f16bc0b1910 0x7f16bc063430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:52548/0 (socket says 172.21.15.110:52548) 2023-10-06T03:29:30.452 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:30.442+0000 7f16c1637700 1 -- 172.21.15.110:0/2700345910 learned_addr learned my addr 172.21.15.110:0/2700345910 (peer_addr_for_me v1:172.21.15.110:0/0) 2023-10-06T03:29:30.452 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:30.442+0000 7f16c0e36700 1 --2- 172.21.15.110:0/2700345910 >> v2:172.21.15.192:3300/0 conn(0x7f16bc155750 0x7f16bc0afc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:30.452 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:30.442+0000 7f16c0e36700 1 -- 172.21.15.110:0/2700345910 >> v2:172.21.15.110:3300/0 conn(0x7f16bc0c6ac0 msgr2=0x7f16bc0b13d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:29:30.452 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:30.442+0000 7f16c0e36700 1 --2- 172.21.15.110:0/2700345910 >> v2:172.21.15.110:3300/0 conn(0x7f16bc0c6ac0 0x7f16bc0b13d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:30.452 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:30.442+0000 7f16c0e36700 1 -- 172.21.15.110:0/2700345910 >> v1:172.21.15.110:6789/0 conn(0x7f16bc0b1910 legacy=0x7f16bc063430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:30.452 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:30.442+0000 7f16bb7fe700 1 -- 172.21.15.110:0/2700345910 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1686024994 0 0) 0x7f16bc0c9d40 con 0x7f16bc0b1910 2023-10-06T03:29:30.452 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:30.442+0000 7f16c0e36700 1 -- 172.21.15.110:0/2700345910 --> v2:172.21.15.192:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16bc067ae0 con 0x7f16bc155750 2023-10-06T03:29:30.452 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:30.442+0000 7f16bbfff700 1 --2- 172.21.15.110:0/2700345910 >> v2:172.21.15.110:3300/0 conn(0x7f16bc0c6ac0 0x7f16bc0b13d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-06T03:29:30.453 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:30.442+0000 7f16c0e36700 1 --2- 172.21.15.110:0/2700345910 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f16bc155750 0x7f16bc0afc00 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f16bc069a00 tx=0x7f16b4009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e436ae6573101611 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:29:30.454 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:30.442+0000 7f16bb7fe700 1 -- 172.21.15.110:0/2700345910 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16b400e070 con 0x7f16bc155750 2023-10-06T03:29:30.454 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:30.442+0000 7f16bb7fe700 1 -- 172.21.15.110:0/2700345910 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16b400bb60 con 0x7f16bc155750 2023-10-06T03:29:30.455 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:30.446+0000 7f16bb7fe700 1 -- 172.21.15.110:0/2700345910 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16b4012430 con 0x7f16bc155750 2023-10-06T03:29:30.455 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:30.446+0000 7f16c309a700 1 -- 172.21.15.110:0/2700345910 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f16bc155750 msgr2=0x7f16bc0afc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:30.455 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:30.446+0000 7f16c309a700 1 --2- 172.21.15.110:0/2700345910 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f16bc155750 0x7f16bc0afc00 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f16bc069a00 tx=0x7f16b4009780 comp rx=0 tx=0).stop 2023-10-06T03:29:30.455 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:30.446+0000 7f16c309a700 1 -- 172.21.15.110:0/2700345910 shutdown_connections 2023-10-06T03:29:30.455 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:30.446+0000 7f16c309a700 1 --2- 172.21.15.110:0/2700345910 >> v2:172.21.15.110:3300/0 conn(0x7f16bc0c6ac0 0x7f16bc0b13d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:30.455 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:30.446+0000 7f16c309a700 1 --2- 172.21.15.110:0/2700345910 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f16bc155750 0x7f16bc0afc00 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:30.455 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:30.446+0000 7f16c309a700 1 -- 172.21.15.110:0/2700345910 >> 172.21.15.110:0/2700345910 conn(0x7f16bc095a60 msgr2=0x7f16bc0670a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:29:30.456 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:30.446+0000 7f16c309a700 1 -- 172.21.15.110:0/2700345910 shutdown_connections 2023-10-06T03:29:30.456 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:30.446+0000 7f16c309a700 1 -- 172.21.15.110:0/2700345910 wait complete. 2023-10-06T03:29:30.456 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:30.450+0000 7f16c309a700 1 Processor -- start 2023-10-06T03:29:30.456 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:30.450+0000 7f16c309a700 1 -- start start 2023-10-06T03:29:30.460 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:30.450+0000 7f16c309a700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f16bc0b1910 0x7f16bc1498a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:30.460 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:30.450+0000 7f16c309a700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f16bc0c6ac0 0x7f16bc168bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:30.460 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:30.450+0000 7f16c309a700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f16bc155750 0x7f16bc099350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:30.460 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:30.450+0000 7f16c309a700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16bc097070 con 0x7f16bc155750 2023-10-06T03:29:30.461 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:30.450+0000 7f16c309a700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16bc097bf0 con 0x7f16bc0b1910 2023-10-06T03:29:30.461 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:30.450+0000 7f16c309a700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16bc0972c0 con 0x7f16bc0c6ac0 2023-10-06T03:29:30.461 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:30.450+0000 7f16c1637700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f16bc155750 0x7f16bc099350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:30.461 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:30.450+0000 7f16c0e36700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f16bc0b1910 0x7f16bc1498a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:30.461 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:30.450+0000 7f16c1637700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f16bc155750 0x7f16bc099350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:38938/0 (socket says 172.21.15.110:38938) 2023-10-06T03:29:30.461 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:30.450+0000 7f16c1637700 1 -- 172.21.15.110:0/111522379 learned_addr learned my addr 172.21.15.110:0/111522379 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:29:30.461 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:30.450+0000 7f16bbfff700 1 --2- 172.21.15.110:0/111522379 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f16bc0c6ac0 0x7f16bc168bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:30.461 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:30.450+0000 7f16c1637700 1 -- 172.21.15.110:0/111522379 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f16bc0c6ac0 msgr2=0x7f16bc168bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:30.462 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:30.450+0000 7f16c1637700 1 --2- 172.21.15.110:0/111522379 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f16bc0c6ac0 0x7f16bc168bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:30.462 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:30.450+0000 7f16c1637700 1 -- 172.21.15.110:0/111522379 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f16bc0b1910 msgr2=0x7f16bc1498a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:30.462 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:30.450+0000 7f16c1637700 1 --2- 172.21.15.110:0/111522379 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f16bc0b1910 0x7f16bc1498a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:30.462 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:30.450+0000 7f16c1637700 1 -- 172.21.15.110:0/111522379 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16ac009420 con 0x7f16bc155750 2023-10-06T03:29:30.462 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:30.450+0000 7f16c1637700 1 --2- 172.21.15.110:0/111522379 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f16bc155750 0x7f16bc099350 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f16a400c560 tx=0x7f16a4007f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:29:30.462 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:30.454+0000 7f16b8ff9700 1 -- 172.21.15.110:0/111522379 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16a4005b30 con 0x7f16bc155750 2023-10-06T03:29:30.462 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:30.454+0000 7f16b8ff9700 1 -- 172.21.15.110:0/111522379 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16a4005c90 con 0x7f16bc155750 2023-10-06T03:29:30.462 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:30.454+0000 7f16c309a700 1 -- 172.21.15.110:0/111522379 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16b4009a20 con 0x7f16bc155750 2023-10-06T03:29:30.463 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:30.454+0000 7f16b8ff9700 1 -- 172.21.15.110:0/111522379 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16a4007320 con 0x7f16bc155750 2023-10-06T03:29:30.463 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:30.454+0000 7f16c309a700 1 -- 172.21.15.110:0/111522379 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16bc097880 con 0x7f16bc155750 2023-10-06T03:29:30.464 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:30.454+0000 7f16c309a700 1 -- 172.21.15.110:0/111522379 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f16bc063f50 con 0x7f16bc155750 2023-10-06T03:29:30.464 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:30.454+0000 7f16b8ff9700 1 -- 172.21.15.110:0/111522379 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7f16a401b3b0 con 0x7f16bc155750 2023-10-06T03:29:30.464 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:30.454+0000 7f16b8ff9700 1 --2- 172.21.15.110:0/111522379 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f16a0060550 0x7f16a0062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:30.464 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:30.454+0000 7f16b8ff9700 1 -- 172.21.15.110:0/111522379 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5206+0+0 (secure 0 0 0) 0x7f16a407b9b0 con 0x7f16bc155750 2023-10-06T03:29:30.464 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:30.454+0000 7f16c0e36700 1 --2- 172.21.15.110:0/111522379 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f16a0060550 0x7f16a0062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:30.464 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:30.458+0000 7f16c0e36700 1 --2- 172.21.15.110:0/111522379 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f16a0060550 0x7f16a0062a00 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f16b40097c0 tx=0x7f16b4019000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:29:30.471 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:30.466+0000 7f16b8ff9700 1 -- 172.21.15.110:0/111522379 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f16a404c190 con 0x7f16bc155750 2023-10-06T03:29:30.672 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:30.666+0000 7f16c309a700 1 -- 172.21.15.110:0/111522379 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f16bc06ff60 con 0x7f16bc155750 2023-10-06T03:29:30.672 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:30.666+0000 7f16b8ff9700 1 -- 172.21.15.110:0/111522379 <== mon.0 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) v1 ==== 93+0+3533 (secure 0 0 0) 0x7f16a404f7b0 con 0x7f16bc155750 2023-10-06T03:29:30.673 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-10-06T03:29:30.673 INFO:teuthology.orchestra.run.smithi110.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4279,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6833/1434505700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6832","nonce":1434505700},{"type":"v1","addr":"172.21.15.192:6833","nonce":1434505700}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4281,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/1168370789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6834","nonce":1168370789},{"type":"v1","addr":"172.21.15.110:6835","nonce":1168370789}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4288,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6835/1271810086","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":1271810086},{"type":"v1","addr":"172.21.15.192:6835","nonce":1271810086}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4293,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6837/540320464","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6836","nonce":540320464},{"type":"v1","addr":"172.21.15.110:6837","nonce":540320464}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[]} 2023-10-06T03:29:30.677 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:30.670+0000 7f16c309a700 1 -- 172.21.15.110:0/111522379 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f16a0060550 msgr2=0x7f16a0062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:30.677 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:30.670+0000 7f16c309a700 1 --2- 172.21.15.110:0/111522379 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f16a0060550 0x7f16a0062a00 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f16b40097c0 tx=0x7f16b4019000 comp rx=0 tx=0).stop 2023-10-06T03:29:30.677 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:30.670+0000 7f16c309a700 1 -- 172.21.15.110:0/111522379 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f16bc155750 msgr2=0x7f16bc099350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:30.677 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:30.670+0000 7f16c309a700 1 --2- 172.21.15.110:0/111522379 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f16bc155750 0x7f16bc099350 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f16a400c560 tx=0x7f16a4007f90 comp rx=0 tx=0).stop 2023-10-06T03:29:30.677 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:30.670+0000 7f16c309a700 1 -- 172.21.15.110:0/111522379 shutdown_connections 2023-10-06T03:29:30.678 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:30.670+0000 7f16c309a700 1 --2- 172.21.15.110:0/111522379 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f16a0060550 0x7f16a0062a00 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:30.678 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:30.670+0000 7f16c309a700 1 --2- 172.21.15.110:0/111522379 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f16bc155750 0x7f16bc099350 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:30.678 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:30.670+0000 7f16c309a700 1 --2- 172.21.15.110:0/111522379 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f16bc0c6ac0 0x7f16bc168bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:30.678 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:30.670+0000 7f16c309a700 1 --2- 172.21.15.110:0/111522379 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f16bc0b1910 0x7f16bc1498a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:30.678 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:30.670+0000 7f16c309a700 1 -- 172.21.15.110:0/111522379 >> 172.21.15.110:0/111522379 conn(0x7f16bc095a60 msgr2=0x7f16bc0cc580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:29:30.679 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:30.670+0000 7f16c309a700 1 -- 172.21.15.110:0/111522379 shutdown_connections 2023-10-06T03:29:30.679 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:30.670+0000 7f16c309a700 1 -- 172.21.15.110:0/111522379 wait complete. 2023-10-06T03:29:30.679 INFO:teuthology.orchestra.run.smithi110.stderr:dumped fsmap epoch 3 2023-10-06T03:29:30.687 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-06T03:29:30.793 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:30.786+0000 7eff18f0e700 1 Processor -- start 2023-10-06T03:29:30.794 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:30.786+0000 7eff18f0e700 1 -- start start 2023-10-06T03:29:30.796 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:30.786+0000 7eff18f0e700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7eff1406caf0 0x7eff140af910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:30.797 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:30.786+0000 7eff18f0e700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff1415e660 con 0x7eff1415cec0 2023-10-06T03:29:30.797 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:30.786+0000 7eff18f0e700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff1415da50 con 0x7eff140afe50 2023-10-06T03:29:30.797 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:30.786+0000 7eff18f0e700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7eff140b2a50 con 0x7eff1406caf0 2023-10-06T03:29:30.797 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:30.786+0000 7eff137fe700 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7eff1415cec0 0x7eff1415d290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:52550/0 (socket says 172.21.15.110:52550) 2023-10-06T03:29:30.797 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:30.786+0000 7eff12ffd700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7eff1406caf0 0x7eff140af910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:30.797 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:30.786+0000 7eff137fe700 1 -- 172.21.15.110:0/1537197428 learned_addr learned my addr 172.21.15.110:0/1537197428 (peer_addr_for_me v1:172.21.15.110:0/0) 2023-10-06T03:29:30.797 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:30.786+0000 7eff127fc700 1 -- 172.21.15.110:0/1537197428 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2013940266 0 0) 0x7eff1415e660 con 0x7eff1415cec0 2023-10-06T03:29:30.797 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:30.786+0000 7eff127fc700 1 -- 172.21.15.110:0/1537197428 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7eff00003570 con 0x7eff1415cec0 2023-10-06T03:29:30.797 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:30.786+0000 7eff12ffd700 1 -- 172.21.15.110:0/1537197428 >> v1:172.21.15.192:6789/0 conn(0x7eff140afe50 legacy=0x7eff140b0570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:30.797 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:30.786+0000 7eff12ffd700 1 -- 172.21.15.110:0/1537197428 >> v1:172.21.15.110:6789/0 conn(0x7eff1415cec0 legacy=0x7eff1415d290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:30.797 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:30.786+0000 7eff12ffd700 1 -- 172.21.15.110:0/1537197428 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff1415dd30 con 0x7eff1406caf0 2023-10-06T03:29:30.798 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:30.786+0000 7eff12ffd700 1 --2- 172.21.15.110:0/1537197428 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7eff1406caf0 0x7eff140af910 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7efefc009e60 tx=0x7efefc009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=58746f8da6bf4a61 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:29:30.798 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:30.786+0000 7eff127fc700 1 -- 172.21.15.110:0/1537197428 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efefc00e070 con 0x7eff1406caf0 2023-10-06T03:29:30.798 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:30.786+0000 7eff127fc700 1 -- 172.21.15.110:0/1537197428 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efefc00bb60 con 0x7eff1406caf0 2023-10-06T03:29:30.798 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:30.786+0000 7eff127fc700 1 -- 172.21.15.110:0/1537197428 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efefc012430 con 0x7eff1406caf0 2023-10-06T03:29:30.798 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:30.790+0000 7eff18f0e700 1 -- 172.21.15.110:0/1537197428 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7eff1406caf0 msgr2=0x7eff140af910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:30.798 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:30.790+0000 7eff18f0e700 1 --2- 172.21.15.110:0/1537197428 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7eff1406caf0 0x7eff140af910 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7efefc009e60 tx=0x7efefc009780 comp rx=0 tx=0).stop 2023-10-06T03:29:30.798 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:30.790+0000 7eff18f0e700 1 -- 172.21.15.110:0/1537197428 shutdown_connections 2023-10-06T03:29:30.798 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:30.790+0000 7eff18f0e700 1 --2- 172.21.15.110:0/1537197428 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7eff1406caf0 0x7eff140af910 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:30.798 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:30.790+0000 7eff18f0e700 1 -- 172.21.15.110:0/1537197428 >> 172.21.15.110:0/1537197428 conn(0x7eff140670a0 msgr2=0x7eff140674a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:29:30.798 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:30.790+0000 7eff18f0e700 1 -- 172.21.15.110:0/1537197428 shutdown_connections 2023-10-06T03:29:30.798 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:30.790+0000 7eff18f0e700 1 -- 172.21.15.110:0/1537197428 wait complete. 2023-10-06T03:29:30.800 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:30.790+0000 7eff18f0e700 1 Processor -- start 2023-10-06T03:29:30.800 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:30.790+0000 7eff18f0e700 1 -- start start 2023-10-06T03:29:30.800 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:30.790+0000 7eff18f0e700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7eff1406caf0 0x7eff140addf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:30.801 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:30.790+0000 7eff18f0e700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7eff140afe50 0x7eff1415a3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:30.801 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:30.790+0000 7eff18f0e700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7eff1415cec0 0x7eff140a44c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:30.801 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:30.790+0000 7eff18f0e700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff1415a8e0 con 0x7eff1406caf0 2023-10-06T03:29:30.801 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:30.790+0000 7eff18f0e700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff1415b330 con 0x7eff140afe50 2023-10-06T03:29:30.801 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:30.790+0000 7eff18f0e700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff140aed90 con 0x7eff1415cec0 2023-10-06T03:29:30.801 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:30.790+0000 7eff137fe700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7eff1406caf0 0x7eff140addf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:30.801 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:30.790+0000 7eff137fe700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7eff1406caf0 0x7eff140addf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:38948/0 (socket says 172.21.15.110:38948) 2023-10-06T03:29:30.801 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:30.790+0000 7eff137fe700 1 -- 172.21.15.110:0/130714100 learned_addr learned my addr 172.21.15.110:0/130714100 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:29:30.801 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:30.790+0000 7eff12ffd700 1 --2- 172.21.15.110:0/130714100 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7eff140afe50 0x7eff1415a3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:30.801 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:30.790+0000 7eff13fff700 1 --2- 172.21.15.110:0/130714100 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7eff1415cec0 0x7eff140a44c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:30.801 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:30.790+0000 7eff137fe700 1 -- 172.21.15.110:0/130714100 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7eff1415cec0 msgr2=0x7eff140a44c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:30.802 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:30.790+0000 7eff137fe700 1 --2- 172.21.15.110:0/130714100 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7eff1415cec0 0x7eff140a44c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:30.802 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:30.790+0000 7eff137fe700 1 -- 172.21.15.110:0/130714100 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7eff140afe50 msgr2=0x7eff1415a3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:30.802 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:30.790+0000 7eff137fe700 1 --2- 172.21.15.110:0/130714100 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7eff140afe50 0x7eff1415a3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:30.802 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:30.790+0000 7eff137fe700 1 -- 172.21.15.110:0/130714100 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efefc009a20 con 0x7eff1406caf0 2023-10-06T03:29:30.802 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:30.790+0000 7eff13fff700 1 --2- 172.21.15.110:0/130714100 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7eff1415cec0 0x7eff140a44c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-06T03:29:30.802 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:30.794+0000 7eff137fe700 1 --2- 172.21.15.110:0/130714100 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7eff1406caf0 0x7eff140addf0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7eff0400c7f0 tx=0x7eff04007e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:29:30.802 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:30.794+0000 7eff0b7fe700 1 -- 172.21.15.110:0/130714100 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff04005b30 con 0x7eff1406caf0 2023-10-06T03:29:30.802 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:30.794+0000 7eff0b7fe700 1 -- 172.21.15.110:0/130714100 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff04005cb0 con 0x7eff1406caf0 2023-10-06T03:29:30.802 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:30.794+0000 7eff18f0e700 1 -- 172.21.15.110:0/130714100 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff140c9690 con 0x7eff1406caf0 2023-10-06T03:29:30.802 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:30.794+0000 7eff0b7fe700 1 -- 172.21.15.110:0/130714100 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff04007320 con 0x7eff1406caf0 2023-10-06T03:29:30.802 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:30.794+0000 7eff18f0e700 1 -- 172.21.15.110:0/130714100 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff140ae7d0 con 0x7eff1406caf0 2023-10-06T03:29:30.803 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:30.794+0000 7eff18f0e700 1 -- 172.21.15.110:0/130714100 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff14070ff0 con 0x7eff1406caf0 2023-10-06T03:29:30.803 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:30.794+0000 7eff0b7fe700 1 -- 172.21.15.110:0/130714100 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7eff0401b3b0 con 0x7eff1406caf0 2023-10-06T03:29:30.803 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:30.794+0000 7eff0b7fe700 1 --2- 172.21.15.110:0/130714100 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7efef4060550 0x7efef4062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:30.804 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:30.794+0000 7eff0b7fe700 1 -- 172.21.15.110:0/130714100 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5206+0+0 (secure 0 0 0) 0x7eff0407ba60 con 0x7eff1406caf0 2023-10-06T03:29:30.804 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:30.794+0000 7eff12ffd700 1 --2- 172.21.15.110:0/130714100 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7efef4060550 0x7efef4062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:30.804 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:30.794+0000 7eff12ffd700 1 --2- 172.21.15.110:0/130714100 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7efef4060550 0x7efef4062a00 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7efefc009e60 tx=0x7efefc019000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:29:30.811 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:30.806+0000 7eff0b7fe700 1 -- 172.21.15.110:0/130714100 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7eff0404c240 con 0x7eff1406caf0 2023-10-06T03:29:30.967 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:30.962+0000 7eff18f0e700 1 -- 172.21.15.110:0/130714100 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7eff1406c8d0 con 0x7eff1406caf0 2023-10-06T03:29:30.968 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:30.962+0000 7eff0b7fe700 1 -- 172.21.15.110:0/130714100 <== mon.0 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+13695 (secure 0 0 0) 0x7eff04017090 con 0x7eff1406caf0 2023-10-06T03:29:30.968 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-10-06T03:29:30.968 INFO:teuthology.orchestra.run.smithi110.stdout:{"epoch":20,"fsid":"93b8f6ac-4c92-4b45-b8e4-b59617024aa2","created":"2023-10-06T03:29:10.519583+0000","modified":"2023-10-06T03:29:30.369050+0000","last_up_change":"2023-10-06T03:29:21.960511+0000","last_in_change":"2023-10-06T03:29:12.560436+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-06T03:29:20.703743+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2023-10-06T03:29:28.307281+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"5866eb6c-2adc-46e4-be7a-e5bcda0f4065","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6824","nonce":33394},{"type":"v1","addr":"172.21.15.110:6825","nonce":33394}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6826","nonce":33394},{"type":"v1","addr":"172.21.15.110:6827","nonce":33394}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6830","nonce":33394},{"type":"v1","addr":"172.21.15.110:6831","nonce":33394}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6828","nonce":33394},{"type":"v1","addr":"172.21.15.110:6829","nonce":33394}]},"public_addr":"172.21.15.110:6825/33394","cluster_addr":"172.21.15.110:6827/33394","heartbeat_back_addr":"172.21.15.110:6831/33394","heartbeat_front_addr":"172.21.15.110:6829/33394","state":["exists","up"]},{"osd":1,"uuid":"a38de118-062b-4a05-9782-2ceddf6ff4df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6800","nonce":33396},{"type":"v1","addr":"172.21.15.110:6801","nonce":33396}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6802","nonce":33396},{"type":"v1","addr":"172.21.15.110:6803","nonce":33396}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6806","nonce":33396},{"type":"v1","addr":"172.21.15.110:6807","nonce":33396}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6804","nonce":33396},{"type":"v1","addr":"172.21.15.110:6805","nonce":33396}]},"public_addr":"172.21.15.110:6801/33396","cluster_addr":"172.21.15.110:6803/33396","heartbeat_back_addr":"172.21.15.110:6807/33396","heartbeat_front_addr":"172.21.15.110:6805/33396","state":["exists","up"]},{"osd":2,"uuid":"56e39be2-3407-4b69-877e-cf16617b06d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6816","nonce":33397},{"type":"v1","addr":"172.21.15.110:6817","nonce":33397}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6818","nonce":33397},{"type":"v1","addr":"172.21.15.110:6819","nonce":33397}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6822","nonce":33397},{"type":"v1","addr":"172.21.15.110:6823","nonce":33397}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6820","nonce":33397},{"type":"v1","addr":"172.21.15.110:6821","nonce":33397}]},"public_addr":"172.21.15.110:6817/33397","cluster_addr":"172.21.15.110:6819/33397","heartbeat_back_addr":"172.21.15.110:6823/33397","heartbeat_front_addr":"172.21.15.110:6821/33397","state":["exists","up"]},{"osd":3,"uuid":"01c82d13-c1d4-4bde-b55e-3e0089b40115","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6808","nonce":33395},{"type":"v1","addr":"172.21.15.110:6809","nonce":33395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6810","nonce":33395},{"type":"v1","addr":"172.21.15.110:6811","nonce":33395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6814","nonce":33395},{"type":"v1","addr":"172.21.15.110:6815","nonce":33395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6812","nonce":33395},{"type":"v1","addr":"172.21.15.110:6813","nonce":33395}]},"public_addr":"172.21.15.110:6809/33395","cluster_addr":"172.21.15.110:6811/33395","heartbeat_back_addr":"172.21.15.110:6815/33395","heartbeat_front_addr":"172.21.15.110:6813/33395","state":["exists","up"]},{"osd":4,"uuid":"86983430-5fa3-4d17-998f-03e2cd843237","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":33683},{"type":"v1","addr":"172.21.15.192:6817","nonce":33683}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":33683},{"type":"v1","addr":"172.21.15.192:6819","nonce":33683}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":33683},{"type":"v1","addr":"172.21.15.192:6823","nonce":33683}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6820","nonce":33683},{"type":"v1","addr":"172.21.15.192:6821","nonce":33683}]},"public_addr":"172.21.15.192:6817/33683","cluster_addr":"172.21.15.192:6819/33683","heartbeat_back_addr":"172.21.15.192:6823/33683","heartbeat_front_addr":"172.21.15.192:6821/33683","state":["exists","up"]},{"osd":5,"uuid":"6945257f-821d-4619-a63f-ebe857eba6f4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":33684},{"type":"v1","addr":"172.21.15.192:6801","nonce":33684}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":33684},{"type":"v1","addr":"172.21.15.192:6803","nonce":33684}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":33684},{"type":"v1","addr":"172.21.15.192:6807","nonce":33684}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6804","nonce":33684},{"type":"v1","addr":"172.21.15.192:6805","nonce":33684}]},"public_addr":"172.21.15.192:6801/33684","cluster_addr":"172.21.15.192:6803/33684","heartbeat_back_addr":"172.21.15.192:6807/33684","heartbeat_front_addr":"172.21.15.192:6805/33684","state":["exists","up"]},{"osd":6,"uuid":"7854d1ae-7060-4780-8f97-73cea5e7a12f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6824","nonce":33686},{"type":"v1","addr":"172.21.15.192:6825","nonce":33686}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":33686},{"type":"v1","addr":"172.21.15.192:6827","nonce":33686}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":33686},{"type":"v1","addr":"172.21.15.192:6831","nonce":33686}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6828","nonce":33686},{"type":"v1","addr":"172.21.15.192:6829","nonce":33686}]},"public_addr":"172.21.15.192:6825/33686","cluster_addr":"172.21.15.192:6827/33686","heartbeat_back_addr":"172.21.15.192:6831/33686","heartbeat_front_addr":"172.21.15.192:6829/33686","state":["exists","up"]},{"osd":7,"uuid":"f1ebdfd5-b859-482b-9f97-c18c4a26f445","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6808","nonce":33685},{"type":"v1","addr":"172.21.15.192:6809","nonce":33685}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":33685},{"type":"v1","addr":"172.21.15.192:6811","nonce":33685}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":33685},{"type":"v1","addr":"172.21.15.192:6815","nonce":33685}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6812","nonce":33685},{"type":"v1","addr":"172.21.15.192:6813","nonce":33685}]},"public_addr":"172.21.15.192:6809/33685","cluster_addr":"172.21.15.192:6811/33685","heartbeat_back_addr":"172.21.15.192:6815/33685","heartbeat_front_addr":"172.21.15.192:6813/33685","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T03:29:18.002048+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T03:29:17.478576+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T03:29:18.034944+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T03:29:17.628513+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T03:29:18.047338+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T03:29:17.772349+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T03:29:17.883958+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T03:29:18.151838+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-06T03:29:30.971 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:30.962+0000 7eff18f0e700 1 -- 172.21.15.110:0/130714100 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7efef4060550 msgr2=0x7efef4062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:30.971 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:30.962+0000 7eff18f0e700 1 --2- 172.21.15.110:0/130714100 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7efef4060550 0x7efef4062a00 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7efefc009e60 tx=0x7efefc019000 comp rx=0 tx=0).stop 2023-10-06T03:29:30.971 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:30.962+0000 7eff18f0e700 1 -- 172.21.15.110:0/130714100 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7eff1406caf0 msgr2=0x7eff140addf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:30.971 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:30.962+0000 7eff18f0e700 1 --2- 172.21.15.110:0/130714100 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7eff1406caf0 0x7eff140addf0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7eff0400c7f0 tx=0x7eff04007e00 comp rx=0 tx=0).stop 2023-10-06T03:29:30.971 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:30.962+0000 7eff18f0e700 1 -- 172.21.15.110:0/130714100 shutdown_connections 2023-10-06T03:29:30.971 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:30.962+0000 7eff18f0e700 1 --2- 172.21.15.110:0/130714100 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7efef4060550 0x7efef4062a00 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:30.971 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:30.966+0000 7eff18f0e700 1 --2- 172.21.15.110:0/130714100 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7eff1415cec0 0x7eff140a44c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:30.972 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:30.966+0000 7eff18f0e700 1 --2- 172.21.15.110:0/130714100 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7eff140afe50 0x7eff1415a3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:30.972 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:30.966+0000 7eff18f0e700 1 --2- 172.21.15.110:0/130714100 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7eff1406caf0 0x7eff140addf0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:30.972 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:30.966+0000 7eff18f0e700 1 -- 172.21.15.110:0/130714100 >> 172.21.15.110:0/130714100 conn(0x7eff140670a0 msgr2=0x7eff1406cf10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:29:30.972 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:30.966+0000 7eff18f0e700 1 -- 172.21.15.110:0/130714100 shutdown_connections 2023-10-06T03:29:30.972 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:30.966+0000 7eff18f0e700 1 -- 172.21.15.110:0/130714100 wait complete. 2023-10-06T03:29:30.982 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-06T03:29:20.703743+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-10-06T03:29:28.307281+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}] 2023-10-06T03:29:30.982 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-06T03:29:31.098 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:31.090+0000 7f76d6f45700 1 Processor -- start 2023-10-06T03:29:31.099 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:31.090+0000 7f76d6f45700 1 -- start start 2023-10-06T03:29:31.101 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:31.094+0000 7f76d6f45700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f76d006cee0 0x7f76d0157090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:31.101 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:31.094+0000 7f76d6f45700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76d00672b0 con 0x7f76d0156cc0 2023-10-06T03:29:31.101 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:31.094+0000 7f76d6f45700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76d015b2b0 con 0x7f76d00c8a10 2023-10-06T03:29:31.101 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:31.094+0000 7f76d6f45700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f76d015b590 con 0x7f76d006cee0 2023-10-06T03:29:31.101 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:31.094+0000 7f76d6744700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f76d006cee0 0x7f76d0157090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:31.101 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:31.094+0000 7f76d6744700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f76d006cee0 0x7f76d0157090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:36516/0 (socket says 172.21.15.110:36516) 2023-10-06T03:29:31.101 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:31.094+0000 7f76d5f43700 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f76d0156cc0 0x7f76d00caee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:55480/0 (socket says 172.21.15.110:55480) 2023-10-06T03:29:31.101 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:31.094+0000 7f76d6744700 1 -- 172.21.15.110:0/1899836887 learned_addr learned my addr 172.21.15.110:0/1899836887 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:29:31.101 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:31.094+0000 7f76d6744700 1 -- 172.21.15.110:0/1899836887 >> v1:172.21.15.192:6789/0 conn(0x7f76d00c8a10 legacy=0x7f76d006c7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:31.101 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:31.094+0000 7f76d6744700 1 -- 172.21.15.110:0/1899836887 >> v1:172.21.15.110:6789/0 conn(0x7f76d0156cc0 legacy=0x7f76d00caee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:31.102 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:31.094+0000 7f76d6744700 1 -- 172.21.15.110:0/1899836887 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76d015b870 con 0x7f76d006cee0 2023-10-06T03:29:31.102 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:31.094+0000 7f76d6744700 1 --2- 172.21.15.110:0/1899836887 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f76d006cee0 0x7f76d0157090 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f76c8009f20 tx=0x7f76c8009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e63d3dea646661f1 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:29:31.102 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:31.094+0000 7f76d4f41700 1 -- 172.21.15.110:0/1899836887 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76c800e070 con 0x7f76d006cee0 2023-10-06T03:29:31.102 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:31.094+0000 7f76d4f41700 1 -- 172.21.15.110:0/1899836887 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f76c800ba80 con 0x7f76d006cee0 2023-10-06T03:29:31.102 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:31.094+0000 7f76d4f41700 1 -- 172.21.15.110:0/1899836887 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76c80092e0 con 0x7f76d006cee0 2023-10-06T03:29:31.103 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:31.094+0000 7f76d6f45700 1 -- 172.21.15.110:0/1899836887 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f76d006cee0 msgr2=0x7f76d0157090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:31.103 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:31.094+0000 7f76d6f45700 1 --2- 172.21.15.110:0/1899836887 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f76d006cee0 0x7f76d0157090 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f76c8009f20 tx=0x7f76c8009840 comp rx=0 tx=0).stop 2023-10-06T03:29:31.103 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:31.094+0000 7f76d6f45700 1 -- 172.21.15.110:0/1899836887 shutdown_connections 2023-10-06T03:29:31.103 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:31.094+0000 7f76d6f45700 1 --2- 172.21.15.110:0/1899836887 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f76d006cee0 0x7f76d0157090 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:31.104 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:31.094+0000 7f76d6f45700 1 -- 172.21.15.110:0/1899836887 >> 172.21.15.110:0/1899836887 conn(0x7f76d006ab30 msgr2=0x7f76d00adff0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:29:31.104 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:31.094+0000 7f76d6f45700 1 -- 172.21.15.110:0/1899836887 shutdown_connections 2023-10-06T03:29:31.104 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:31.094+0000 7f76d6f45700 1 -- 172.21.15.110:0/1899836887 wait complete. 2023-10-06T03:29:31.105 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:31.098+0000 7f76d6f45700 1 Processor -- start 2023-10-06T03:29:31.105 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:31.098+0000 7f76d6f45700 1 -- start start 2023-10-06T03:29:31.108 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:31.098+0000 7f76d6f45700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f76d006cee0 0x7f76d01519f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:31.108 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:31.102+0000 7f76d6f45700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f76d00c8a10 0x7f76d01501f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:31.109 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:31.102+0000 7f76d6f45700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f76d0156cc0 0x7f76d014d630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:31.109 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:31.102+0000 7f76d6f45700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76d01515a0 con 0x7f76d006cee0 2023-10-06T03:29:31.109 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:31.102+0000 7f76d6f45700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76d0150730 con 0x7f76d00c8a10 2023-10-06T03:29:31.109 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:31.102+0000 7f76d6f45700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76d0150990 con 0x7f76d0156cc0 2023-10-06T03:29:31.109 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:31.102+0000 7f76d5f43700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f76d006cee0 0x7f76d01519f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:31.109 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:31.102+0000 7f76d5f43700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f76d006cee0 0x7f76d01519f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:36526/0 (socket says 172.21.15.110:36526) 2023-10-06T03:29:31.109 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:31.102+0000 7f76d5f43700 1 -- 172.21.15.110:0/268327687 learned_addr learned my addr 172.21.15.110:0/268327687 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:29:31.109 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:31.102+0000 7f76d5742700 1 --2- 172.21.15.110:0/268327687 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f76d00c8a10 0x7f76d01501f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:31.109 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:31.102+0000 7f76d6744700 1 --2- 172.21.15.110:0/268327687 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f76d0156cc0 0x7f76d014d630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:31.109 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:31.102+0000 7f76d5f43700 1 -- 172.21.15.110:0/268327687 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f76d0156cc0 msgr2=0x7f76d014d630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:31.109 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:31.102+0000 7f76d5f43700 1 --2- 172.21.15.110:0/268327687 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f76d0156cc0 0x7f76d014d630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:31.110 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:31.102+0000 7f76d5f43700 1 -- 172.21.15.110:0/268327687 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f76d00c8a10 msgr2=0x7f76d01501f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:31.110 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:31.102+0000 7f76d5f43700 1 --2- 172.21.15.110:0/268327687 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f76d00c8a10 0x7f76d01501f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:31.110 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:31.102+0000 7f76d5f43700 1 -- 172.21.15.110:0/268327687 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76c8009ae0 con 0x7f76d006cee0 2023-10-06T03:29:31.110 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:31.102+0000 7f76d5f43700 1 --2- 172.21.15.110:0/268327687 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f76d006cee0 0x7f76d01519f0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f76cc00c560 tx=0x7f76cc007f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:29:31.110 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:31.102+0000 7f76c1ffb700 1 -- 172.21.15.110:0/268327687 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76cc005a70 con 0x7f76d006cee0 2023-10-06T03:29:31.110 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:31.102+0000 7f76c1ffb700 1 -- 172.21.15.110:0/268327687 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f76cc005bd0 con 0x7f76d006cee0 2023-10-06T03:29:31.110 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:31.102+0000 7f76d6f45700 1 -- 172.21.15.110:0/268327687 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76d0150c70 con 0x7f76d006cee0 2023-10-06T03:29:31.111 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:31.102+0000 7f76c1ffb700 1 -- 172.21.15.110:0/268327687 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76cc0070d0 con 0x7f76d006cee0 2023-10-06T03:29:31.111 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:31.102+0000 7f76d6f45700 1 -- 172.21.15.110:0/268327687 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76d0151230 con 0x7f76d006cee0 2023-10-06T03:29:31.112 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:31.102+0000 7f76d6f45700 1 -- 172.21.15.110:0/268327687 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f76d0070ff0 con 0x7f76d006cee0 2023-10-06T03:29:31.112 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:31.102+0000 7f76c1ffb700 1 -- 172.21.15.110:0/268327687 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7f76cc007440 con 0x7f76d006cee0 2023-10-06T03:29:31.112 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:31.102+0000 7f76c1ffb700 1 --2- 172.21.15.110:0/268327687 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f76b4060580 0x7f76b4062a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:31.112 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:31.106+0000 7f76c1ffb700 1 -- 172.21.15.110:0/268327687 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5206+0+0 (secure 0 0 0) 0x7f76cc07b9f0 con 0x7f76d006cee0 2023-10-06T03:29:31.112 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:31.106+0000 7f76d5742700 1 --2- 172.21.15.110:0/268327687 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f76b4060580 0x7f76b4062a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:31.112 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:31.106+0000 7f76d5742700 1 --2- 172.21.15.110:0/268327687 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f76b4060580 0x7f76b4062a30 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f76d0151f30 tx=0x7f76c4000bc0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:29:31.116 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:31.110+0000 7f76c1ffb700 1 -- 172.21.15.110:0/268327687 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f76cc04c1d0 con 0x7f76d006cee0 2023-10-06T03:29:31.278 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:31.270+0000 7f76d6f45700 1 -- 172.21.15.110:0/268327687 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f76d015ab40 con 0x7f76d006cee0 2023-10-06T03:29:31.279 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:31.270+0000 7f76c1ffb700 1 -- 172.21.15.110:0/268327687 <== mon.0 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v20) v1 ==== 93+0+10 (secure 0 0 0) 0x7f76cc053e60 con 0x7f76d006cee0 2023-10-06T03:29:31.279 INFO:teuthology.orchestra.run.smithi110.stdout:pg_num: 1 2023-10-06T03:29:31.282 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:31.274+0000 7f76d6f45700 1 -- 172.21.15.110:0/268327687 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f76b4060580 msgr2=0x7f76b4062a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:31.282 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:31.274+0000 7f76d6f45700 1 --2- 172.21.15.110:0/268327687 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f76b4060580 0x7f76b4062a30 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f76d0151f30 tx=0x7f76c4000bc0 comp rx=0 tx=0).stop 2023-10-06T03:29:31.282 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:31.274+0000 7f76d6f45700 1 -- 172.21.15.110:0/268327687 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f76d006cee0 msgr2=0x7f76d01519f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:31.282 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:31.274+0000 7f76d6f45700 1 --2- 172.21.15.110:0/268327687 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f76d006cee0 0x7f76d01519f0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f76cc00c560 tx=0x7f76cc007f70 comp rx=0 tx=0).stop 2023-10-06T03:29:31.282 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:31.274+0000 7f76d6f45700 1 -- 172.21.15.110:0/268327687 shutdown_connections 2023-10-06T03:29:31.283 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:31.274+0000 7f76d6f45700 1 --2- 172.21.15.110:0/268327687 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f76b4060580 0x7f76b4062a30 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:31.283 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:31.274+0000 7f76d6f45700 1 --2- 172.21.15.110:0/268327687 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f76d0156cc0 0x7f76d014d630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:31.283 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:31.274+0000 7f76d6f45700 1 --2- 172.21.15.110:0/268327687 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f76d00c8a10 0x7f76d01501f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:31.283 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:31.274+0000 7f76d6f45700 1 --2- 172.21.15.110:0/268327687 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f76d006cee0 0x7f76d01519f0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:31.283 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:31.274+0000 7f76d6f45700 1 -- 172.21.15.110:0/268327687 >> 172.21.15.110:0/268327687 conn(0x7f76d006ab30 msgr2=0x7f76d0157630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:29:31.284 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:31.274+0000 7f76d6f45700 1 -- 172.21.15.110:0/268327687 shutdown_connections 2023-10-06T03:29:31.284 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:31.274+0000 7f76d6f45700 1 -- 172.21.15.110:0/268327687 wait complete. 2023-10-06T03:29:31.293 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-10-06T03:29:31.410 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:31.402+0000 7ff12ae8e700 1 Processor -- start 2023-10-06T03:29:31.411 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:31.402+0000 7ff12ae8e700 1 -- start start 2023-10-06T03:29:31.413 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:31.402+0000 7ff12ae8e700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7ff1240aef40 0x7ff1240ce200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:31.413 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:31.402+0000 7ff12ae8e700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff12415d210 0x7ff1240afa60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:31.413 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:31.402+0000 7ff12ae8e700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1240cf5f0 con 0x7ff1240cb1b0 2023-10-06T03:29:31.413 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:31.402+0000 7ff12ae8e700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff1240ce740 con 0x7ff1240aef40 2023-10-06T03:29:31.414 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:31.402+0000 7ff12ae8e700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1240ce9e0 con 0x7ff12415d210 2023-10-06T03:29:31.414 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:31.406+0000 7ff12a68d700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff12415d210 0x7ff1240afa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:31.415 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:31.406+0000 7ff12a68d700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff12415d210 0x7ff1240afa60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3301/0 says I am v2:172.21.15.110:33016/0 (socket says 172.21.15.110:33016) 2023-10-06T03:29:31.415 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:31.406+0000 7ff12a68d700 1 -- 172.21.15.110:0/3022506986 learned_addr learned my addr 172.21.15.110:0/3022506986 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:29:31.415 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:31.406+0000 7ff129e8c700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7ff1240aef40 0x7ff1240ce200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:31.415 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:31.406+0000 7ff12968b700 1 --1- 172.21.15.110:0/3022506986 >> v1:172.21.15.192:6789/0 conn(0x7ff1240cb1b0 0x7ff12415cb00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.110:54294/0 (socket says 172.21.15.110:54294) 2023-10-06T03:29:31.415 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:31.406+0000 7ff12a68d700 1 -- 172.21.15.110:0/3022506986 >> v2:172.21.15.192:3300/0 conn(0x7ff1240aef40 msgr2=0x7ff1240ce200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:29:31.415 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:31.406+0000 7ff12a68d700 1 --2- 172.21.15.110:0/3022506986 >> v2:172.21.15.192:3300/0 conn(0x7ff1240aef40 0x7ff1240ce200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:31.415 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:31.406+0000 7ff12a68d700 1 -- 172.21.15.110:0/3022506986 >> v1:172.21.15.192:6789/0 conn(0x7ff1240cb1b0 legacy=0x7ff12415cb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:31.415 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:31.406+0000 7ff12a68d700 1 -- 172.21.15.110:0/3022506986 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1240cecc0 con 0x7ff12415d210 2023-10-06T03:29:31.415 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:31.406+0000 7ff12a68d700 1 --2- 172.21.15.110:0/3022506986 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff12415d210 0x7ff1240afa60 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7ff11c009f20 tx=0x7ff11c009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c9ddb1b3cc0797f0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:29:31.415 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:31.406+0000 7ff128e8a700 1 -- 172.21.15.110:0/3022506986 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff11c00e070 con 0x7ff12415d210 2023-10-06T03:29:31.415 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:31.406+0000 7ff128e8a700 1 -- 172.21.15.110:0/3022506986 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff11c00b9d0 con 0x7ff12415d210 2023-10-06T03:29:31.416 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:31.406+0000 7ff128e8a700 1 -- 172.21.15.110:0/3022506986 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff11c0092e0 con 0x7ff12415d210 2023-10-06T03:29:31.419 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:31.406+0000 7ff12ae8e700 1 -- 172.21.15.110:0/3022506986 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff12415d210 msgr2=0x7ff1240afa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:31.419 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:31.406+0000 7ff12ae8e700 1 --2- 172.21.15.110:0/3022506986 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff12415d210 0x7ff1240afa60 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7ff11c009f20 tx=0x7ff11c009840 comp rx=0 tx=0).stop 2023-10-06T03:29:31.419 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:31.406+0000 7ff12ae8e700 1 -- 172.21.15.110:0/3022506986 shutdown_connections 2023-10-06T03:29:31.419 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:31.406+0000 7ff12ae8e700 1 --2- 172.21.15.110:0/3022506986 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff12415d210 0x7ff1240afa60 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:31.419 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:31.406+0000 7ff12ae8e700 1 --2- 172.21.15.110:0/3022506986 >> v2:172.21.15.192:3300/0 conn(0x7ff1240aef40 0x7ff1240ce200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:31.419 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:31.406+0000 7ff12ae8e700 1 -- 172.21.15.110:0/3022506986 >> 172.21.15.110:0/3022506986 conn(0x7ff1240cfec0 msgr2=0x7ff1240d02c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:29:31.419 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:31.406+0000 7ff12ae8e700 1 -- 172.21.15.110:0/3022506986 shutdown_connections 2023-10-06T03:29:31.419 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:31.406+0000 7ff12ae8e700 1 -- 172.21.15.110:0/3022506986 wait complete. 2023-10-06T03:29:31.419 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:31.410+0000 7ff12ae8e700 1 Processor -- start 2023-10-06T03:29:31.419 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:31.410+0000 7ff12ae8e700 1 -- start start 2023-10-06T03:29:31.420 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:31.410+0000 7ff12ae8e700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff1240aef40 0x7ff1240c7d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:31.420 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:31.410+0000 7ff12ae8e700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff1240cb1b0 0x7ff12414ca00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:31.420 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:31.410+0000 7ff12ae8e700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff12415d210 0x7ff1240c8c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:31.420 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:31.410+0000 7ff12ae8e700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff12414cf40 con 0x7ff12415d210 2023-10-06T03:29:31.420 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:31.410+0000 7ff12ae8e700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1240c78a0 con 0x7ff1240aef40 2023-10-06T03:29:31.420 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:31.410+0000 7ff12ae8e700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1240c96d0 con 0x7ff1240cb1b0 2023-10-06T03:29:31.420 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:31.410+0000 7ff12a68d700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff12415d210 0x7ff1240c8c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:31.420 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:31.410+0000 7ff12a68d700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff12415d210 0x7ff1240c8c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:36542/0 (socket says 172.21.15.110:36542) 2023-10-06T03:29:31.420 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:31.410+0000 7ff12a68d700 1 -- 172.21.15.110:0/3182695634 learned_addr learned my addr 172.21.15.110:0/3182695634 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:29:31.420 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:31.410+0000 7ff12968b700 1 --2- 172.21.15.110:0/3182695634 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff1240cb1b0 0x7ff12414ca00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:31.420 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:31.410+0000 7ff129e8c700 1 --2- 172.21.15.110:0/3182695634 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff1240aef40 0x7ff1240c7d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:31.420 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:31.410+0000 7ff12a68d700 1 -- 172.21.15.110:0/3182695634 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff1240cb1b0 msgr2=0x7ff12414ca00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:31.421 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:31.410+0000 7ff12a68d700 1 --2- 172.21.15.110:0/3182695634 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff1240cb1b0 0x7ff12414ca00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:31.421 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:31.410+0000 7ff12a68d700 1 -- 172.21.15.110:0/3182695634 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff1240aef40 msgr2=0x7ff1240c7d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:31.421 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:31.410+0000 7ff12a68d700 1 --2- 172.21.15.110:0/3182695634 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff1240aef40 0x7ff1240c7d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:31.421 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:31.410+0000 7ff12a68d700 1 -- 172.21.15.110:0/3182695634 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff11c009ae0 con 0x7ff12415d210 2023-10-06T03:29:31.421 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:31.410+0000 7ff12a68d700 1 --2- 172.21.15.110:0/3182695634 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff12415d210 0x7ff1240c8c80 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7ff1240ceb20 tx=0x7ff11c018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:29:31.421 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:31.410+0000 7ff119ffb700 1 -- 172.21.15.110:0/3182695634 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff11c0070a0 con 0x7ff12415d210 2023-10-06T03:29:31.421 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:31.410+0000 7ff119ffb700 1 -- 172.21.15.110:0/3182695634 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff11c007b20 con 0x7ff12415d210 2023-10-06T03:29:31.421 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:31.410+0000 7ff12ae8e700 1 -- 172.21.15.110:0/3182695634 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1240c6f70 con 0x7ff12415d210 2023-10-06T03:29:31.421 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:31.410+0000 7ff119ffb700 1 -- 172.21.15.110:0/3182695634 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff11c01c410 con 0x7ff12415d210 2023-10-06T03:29:31.421 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:31.410+0000 7ff12ae8e700 1 -- 172.21.15.110:0/3182695634 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1240c7530 con 0x7ff12415d210 2023-10-06T03:29:31.422 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:31.414+0000 7ff12ae8e700 1 -- 172.21.15.110:0/3182695634 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff124070ff0 con 0x7ff12415d210 2023-10-06T03:29:31.422 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:31.414+0000 7ff119ffb700 1 -- 172.21.15.110:0/3182695634 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7ff11c003050 con 0x7ff12415d210 2023-10-06T03:29:31.422 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:31.414+0000 7ff119ffb700 1 --2- 172.21.15.110:0/3182695634 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7ff108060550 0x7ff108062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:31.423 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:31.414+0000 7ff129e8c700 1 --2- 172.21.15.110:0/3182695634 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7ff108060550 0x7ff108062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:31.423 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:31.418+0000 7ff119ffb700 1 -- 172.21.15.110:0/3182695634 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5206+0+0 (secure 0 0 0) 0x7ff11c07de90 con 0x7ff12415d210 2023-10-06T03:29:31.423 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:31.418+0000 7ff129e8c700 1 --2- 172.21.15.110:0/3182695634 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7ff108060550 0x7ff108062a00 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7ff114009db0 tx=0x7ff1140091a0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:29:31.433 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:31.426+0000 7ff119ffb700 1 -- 172.21.15.110:0/3182695634 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7ff11c04e5d0 con 0x7ff12415d210 2023-10-06T03:29:31.595 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:31.590+0000 7ff12ae8e700 1 -- 172.21.15.110:0/3182695634 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7ff12406d8d0 con 0x7ff12415d210 2023-10-06T03:29:31.596 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:31.590+0000 7ff119ffb700 1 -- 172.21.15.110:0/3182695634 <== mon.0 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v20) v1 ==== 92+0+10 (secure 0 0 0) 0x7ff11c051bf0 con 0x7ff12415d210 2023-10-06T03:29:31.596 INFO:teuthology.orchestra.run.smithi110.stdout:pg_num: 8 2023-10-06T03:29:31.599 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:31.590+0000 7ff12ae8e700 1 -- 172.21.15.110:0/3182695634 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7ff108060550 msgr2=0x7ff108062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:31.599 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:31.590+0000 7ff12ae8e700 1 --2- 172.21.15.110:0/3182695634 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7ff108060550 0x7ff108062a00 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7ff114009db0 tx=0x7ff1140091a0 comp rx=0 tx=0).stop 2023-10-06T03:29:31.599 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:31.590+0000 7ff12ae8e700 1 -- 172.21.15.110:0/3182695634 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff12415d210 msgr2=0x7ff1240c8c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:31.599 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:31.590+0000 7ff12ae8e700 1 --2- 172.21.15.110:0/3182695634 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff12415d210 0x7ff1240c8c80 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7ff1240ceb20 tx=0x7ff11c018000 comp rx=0 tx=0).stop 2023-10-06T03:29:31.599 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:31.590+0000 7ff12ae8e700 1 -- 172.21.15.110:0/3182695634 shutdown_connections 2023-10-06T03:29:31.599 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:31.590+0000 7ff12ae8e700 1 --2- 172.21.15.110:0/3182695634 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7ff108060550 0x7ff108062a00 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:31.599 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:31.590+0000 7ff12ae8e700 1 --2- 172.21.15.110:0/3182695634 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff12415d210 0x7ff1240c8c80 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:31.599 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:31.590+0000 7ff12ae8e700 1 --2- 172.21.15.110:0/3182695634 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff1240cb1b0 0x7ff12414ca00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:31.600 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:31.590+0000 7ff12ae8e700 1 --2- 172.21.15.110:0/3182695634 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff1240aef40 0x7ff1240c7d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:31.600 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:31.590+0000 7ff12ae8e700 1 -- 172.21.15.110:0/3182695634 >> 172.21.15.110:0/3182695634 conn(0x7ff1240cfec0 msgr2=0x7ff124158a20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:29:31.600 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:31.590+0000 7ff12ae8e700 1 -- 172.21.15.110:0/3182695634 shutdown_connections 2023-10-06T03:29:31.600 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:31.590+0000 7ff12ae8e700 1 -- 172.21.15.110:0/3182695634 wait complete. 2023-10-06T03:29:31.611 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2023-10-06T03:29:31.715 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:31.706+0000 7fba6ce5e700 1 Processor -- start 2023-10-06T03:29:31.715 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:31.706+0000 7fba6ce5e700 1 -- start start 2023-10-06T03:29:31.716 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:31.710+0000 7fba6ce5e700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fba680ae120 0x7fba680af4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:31.716 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:31.710+0000 7fba6ce5e700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fba680b04d0 0x7fba680ca660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:31.716 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:31.710+0000 7fba6ce5e700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba68155780 con 0x7fba680add50 2023-10-06T03:29:31.716 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:31.710+0000 7fba6ce5e700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fba680cb4a0 con 0x7fba680ae120 2023-10-06T03:29:31.716 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:31.710+0000 7fba6ce5e700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba680b0950 con 0x7fba680b04d0 2023-10-06T03:29:31.716 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:31.710+0000 7fba66ffd700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fba680ae120 0x7fba680af4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:31.717 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:31.710+0000 7fba67fff700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fba680b04d0 0x7fba680ca660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:31.717 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:31.710+0000 7fba66ffd700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fba680ae120 0x7fba680af4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.110:43182/0 (socket says 172.21.15.110:43182) 2023-10-06T03:29:31.717 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:31.710+0000 7fba66ffd700 1 -- 172.21.15.110:0/2181503499 learned_addr learned my addr 172.21.15.110:0/2181503499 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:29:31.717 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:31.710+0000 7fba677fe700 1 --1- 172.21.15.110:0/2181503499 >> v1:172.21.15.192:6789/0 conn(0x7fba680add50 0x7fba680c5f40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.110:54310/0 (socket says 172.21.15.110:54310) 2023-10-06T03:29:31.718 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:31.710+0000 7fba66ffd700 1 -- 172.21.15.110:0/2181503499 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fba680b04d0 msgr2=0x7fba680ca660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:29:31.718 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:31.710+0000 7fba66ffd700 1 --2- 172.21.15.110:0/2181503499 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fba680b04d0 0x7fba680ca660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:31.718 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:31.710+0000 7fba66ffd700 1 -- 172.21.15.110:0/2181503499 >> v1:172.21.15.192:6789/0 conn(0x7fba680add50 legacy=0x7fba680c5f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:31.718 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:31.710+0000 7fba66ffd700 1 -- 172.21.15.110:0/2181503499 --> v2:172.21.15.192:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba680b0c10 con 0x7fba680ae120 2023-10-06T03:29:31.718 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:31.710+0000 7fba66ffd700 1 --2- 172.21.15.110:0/2181503499 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fba680ae120 0x7fba680af4f0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fba5800ba50 tx=0x7fba5800a200 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e764001c0efa1b94 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:29:31.719 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:31.710+0000 7fba667fc700 1 -- 172.21.15.110:0/2181503499 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba58009070 con 0x7fba680ae120 2023-10-06T03:29:31.719 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:31.710+0000 7fba667fc700 1 -- 172.21.15.110:0/2181503499 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba5800cb60 con 0x7fba680ae120 2023-10-06T03:29:31.719 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:31.710+0000 7fba667fc700 1 -- 172.21.15.110:0/2181503499 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba5800d450 con 0x7fba680ae120 2023-10-06T03:29:31.719 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:31.710+0000 7fba6ce5e700 1 -- 172.21.15.110:0/2181503499 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fba680ae120 msgr2=0x7fba680af4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:31.719 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:31.710+0000 7fba6ce5e700 1 --2- 172.21.15.110:0/2181503499 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fba680ae120 0x7fba680af4f0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fba5800ba50 tx=0x7fba5800a200 comp rx=0 tx=0).stop 2023-10-06T03:29:31.719 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:31.710+0000 7fba6ce5e700 1 -- 172.21.15.110:0/2181503499 shutdown_connections 2023-10-06T03:29:31.719 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:31.710+0000 7fba6ce5e700 1 --2- 172.21.15.110:0/2181503499 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fba680b04d0 0x7fba680ca660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:31.719 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:31.710+0000 7fba6ce5e700 1 --2- 172.21.15.110:0/2181503499 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fba680ae120 0x7fba680af4f0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:31.719 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:31.710+0000 7fba6ce5e700 1 -- 172.21.15.110:0/2181503499 >> 172.21.15.110:0/2181503499 conn(0x7fba680cc470 msgr2=0x7fba680c8410 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:29:31.720 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:31.710+0000 7fba6ce5e700 1 -- 172.21.15.110:0/2181503499 shutdown_connections 2023-10-06T03:29:31.720 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:31.710+0000 7fba6ce5e700 1 -- 172.21.15.110:0/2181503499 wait complete. 2023-10-06T03:29:31.720 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:31.714+0000 7fba6ce5e700 1 Processor -- start 2023-10-06T03:29:31.720 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:31.714+0000 7fba6ce5e700 1 -- start start 2023-10-06T03:29:31.722 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:31.714+0000 7fba6ce5e700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fba680add50 0x7fba680c5f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:31.723 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:31.714+0000 7fba6ce5e700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fba680ae120 0x7fba680c2c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:31.723 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:31.714+0000 7fba6ce5e700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fba680b04d0 0x7fba68154860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:31.723 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:31.714+0000 7fba6ce5e700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba680c31b0 con 0x7fba680add50 2023-10-06T03:29:31.723 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:31.714+0000 7fba6ce5e700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba680c33e0 con 0x7fba680ae120 2023-10-06T03:29:31.723 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:31.714+0000 7fba6ce5e700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba680c36c0 con 0x7fba680b04d0 2023-10-06T03:29:31.723 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:31.714+0000 7fba677fe700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fba680add50 0x7fba680c5f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:31.724 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:31.714+0000 7fba67fff700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fba680b04d0 0x7fba68154860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:31.724 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:31.714+0000 7fba66ffd700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fba680ae120 0x7fba680c2c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:31.724 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:31.714+0000 7fba677fe700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fba680add50 0x7fba680c5f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:36558/0 (socket says 172.21.15.110:36558) 2023-10-06T03:29:31.724 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:31.714+0000 7fba67fff700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fba680b04d0 0x7fba68154860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3301/0 says I am v2:172.21.15.110:33042/0 (socket says 172.21.15.110:33042) 2023-10-06T03:29:31.724 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:31.714+0000 7fba677fe700 1 -- 172.21.15.110:0/1402334399 learned_addr learned my addr 172.21.15.110:0/1402334399 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:29:31.725 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:31.714+0000 7fba67fff700 1 -- 172.21.15.110:0/1402334399 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fba680ae120 msgr2=0x7fba680c2c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:31.725 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:31.714+0000 7fba67fff700 1 --2- 172.21.15.110:0/1402334399 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fba680ae120 0x7fba680c2c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:31.725 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:31.718+0000 7fba67fff700 1 -- 172.21.15.110:0/1402334399 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fba680add50 msgr2=0x7fba680c5f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:31.725 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:31.718+0000 7fba67fff700 1 --2- 172.21.15.110:0/1402334399 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fba680add50 0x7fba680c5f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:31.725 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:31.718+0000 7fba67fff700 1 -- 172.21.15.110:0/1402334399 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba5800b590 con 0x7fba680b04d0 2023-10-06T03:29:31.725 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:31.718+0000 7fba67fff700 1 --2- 172.21.15.110:0/1402334399 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fba680b04d0 0x7fba68154860 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fba600101f0 tx=0x7fba6000f040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:29:31.725 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:31.718+0000 7fba4f7fe700 1 -- 172.21.15.110:0/1402334399 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba6001c070 con 0x7fba680b04d0 2023-10-06T03:29:31.726 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:31.718+0000 7fba4f7fe700 1 -- 172.21.15.110:0/1402334399 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba60013040 con 0x7fba680b04d0 2023-10-06T03:29:31.726 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:31.718+0000 7fba4f7fe700 1 -- 172.21.15.110:0/1402334399 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba60012020 con 0x7fba680b04d0 2023-10-06T03:29:31.726 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:31.718+0000 7fba6ce5e700 1 -- 172.21.15.110:0/1402334399 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba680c39a0 con 0x7fba680b04d0 2023-10-06T03:29:31.726 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:31.718+0000 7fba6ce5e700 1 -- 172.21.15.110:0/1402334399 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba68154da0 con 0x7fba680b04d0 2023-10-06T03:29:31.726 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:31.718+0000 7fba6ce5e700 1 -- 172.21.15.110:0/1402334399 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba68156f20 con 0x7fba680b04d0 2023-10-06T03:29:31.727 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:31.718+0000 7fba4f7fe700 1 -- 172.21.15.110:0/1402334399 <== mon.2 v2:172.21.15.192:3301/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7fba6000d2e0 con 0x7fba680b04d0 2023-10-06T03:29:31.727 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:31.722+0000 7fba4f7fe700 1 --2- 172.21.15.110:0/1402334399 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fba54060550 0x7fba54062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:31.727 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:31.722+0000 7fba4f7fe700 1 -- 172.21.15.110:0/1402334399 <== mon.2 v2:172.21.15.192:3301/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5206+0+0 (secure 0 0 0) 0x7fba60080980 con 0x7fba680b04d0 2023-10-06T03:29:31.727 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:31.722+0000 7fba677fe700 1 --2- 172.21.15.110:0/1402334399 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fba54060550 0x7fba54062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:31.728 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:31.722+0000 7fba677fe700 1 --2- 172.21.15.110:0/1402334399 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fba54060550 0x7fba54062a00 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fba5c000c00 tx=0x7fba5c007f30 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:29:31.740 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:31.734+0000 7fba4f7fe700 1 -- 172.21.15.110:0/1402334399 <== mon.2 v2:172.21.15.192:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fba60051160 con 0x7fba680b04d0 2023-10-06T03:29:31.906 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:31.898+0000 7fba6ce5e700 1 -- 172.21.15.110:0/1402334399 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7fba680c3e20 con 0x7fba680b04d0 2023-10-06T03:29:31.906 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:31.898+0000 7fba4f7fe700 1 -- 172.21.15.110:0/1402334399 <== mon.2 v2:172.21.15.192:3301/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v20) v1 ==== 84+0+126 (secure 0 0 0) 0x7fba60054780 con 0x7fba680b04d0 2023-10-06T03:29:31.906 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-10-06T03:29:31.907 INFO:teuthology.orchestra.run.smithi110.stdout:[ 2023-10-06T03:29:31.907 INFO:teuthology.orchestra.run.smithi110.stdout: { 2023-10-06T03:29:31.907 INFO:teuthology.orchestra.run.smithi110.stdout: "poolnum": 1, 2023-10-06T03:29:31.907 INFO:teuthology.orchestra.run.smithi110.stdout: "poolname": ".mgr" 2023-10-06T03:29:31.907 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2023-10-06T03:29:31.907 INFO:teuthology.orchestra.run.smithi110.stdout: { 2023-10-06T03:29:31.907 INFO:teuthology.orchestra.run.smithi110.stdout: "poolnum": 2, 2023-10-06T03:29:31.907 INFO:teuthology.orchestra.run.smithi110.stdout: "poolname": "rbd" 2023-10-06T03:29:31.908 INFO:teuthology.orchestra.run.smithi110.stdout: } 2023-10-06T03:29:31.908 INFO:teuthology.orchestra.run.smithi110.stdout:] 2023-10-06T03:29:31.909 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:31.902+0000 7fba6ce5e700 1 -- 172.21.15.110:0/1402334399 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fba54060550 msgr2=0x7fba54062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:31.909 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:31.902+0000 7fba6ce5e700 1 --2- 172.21.15.110:0/1402334399 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fba54060550 0x7fba54062a00 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fba5c000c00 tx=0x7fba5c007f30 comp rx=0 tx=0).stop 2023-10-06T03:29:31.909 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:31.902+0000 7fba6ce5e700 1 -- 172.21.15.110:0/1402334399 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fba680b04d0 msgr2=0x7fba68154860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:31.909 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:31.902+0000 7fba6ce5e700 1 --2- 172.21.15.110:0/1402334399 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fba680b04d0 0x7fba68154860 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fba600101f0 tx=0x7fba6000f040 comp rx=0 tx=0).stop 2023-10-06T03:29:31.909 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:31.902+0000 7fba6ce5e700 1 -- 172.21.15.110:0/1402334399 shutdown_connections 2023-10-06T03:29:31.909 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:31.902+0000 7fba6ce5e700 1 --2- 172.21.15.110:0/1402334399 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fba54060550 0x7fba54062a00 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:31.909 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:31.902+0000 7fba6ce5e700 1 --2- 172.21.15.110:0/1402334399 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fba680b04d0 0x7fba68154860 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:31.910 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:31.902+0000 7fba6ce5e700 1 --2- 172.21.15.110:0/1402334399 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fba680ae120 0x7fba680c2c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:31.910 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:31.902+0000 7fba6ce5e700 1 --2- 172.21.15.110:0/1402334399 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fba680add50 0x7fba680c5f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:31.910 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:31.902+0000 7fba6ce5e700 1 -- 172.21.15.110:0/1402334399 >> 172.21.15.110:0/1402334399 conn(0x7fba680cc470 msgr2=0x7fba680698e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:29:31.910 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:31.902+0000 7fba6ce5e700 1 -- 172.21.15.110:0/1402334399 shutdown_connections 2023-10-06T03:29:31.910 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:31.902+0000 7fba6ce5e700 1 -- 172.21.15.110:0/1402334399 wait complete. 2023-10-06T03:29:31.920 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2023-10-06T03:29:31.920 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2023-10-06T03:29:32.029 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:32.022+0000 7fe912c39700 1 Processor -- start 2023-10-06T03:29:32.030 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:32.022+0000 7fe912c39700 1 -- start start 2023-10-06T03:29:32.030 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:32.022+0000 7fe912c39700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe90c158de0 0x7fe90c0cab00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:32.030 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:32.022+0000 7fe912c39700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fe90c0cb040 0x7fe90c15cb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:32.031 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:32.022+0000 7fe912c39700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe90c15d4f0 con 0x7fe90c15d090 2023-10-06T03:29:32.031 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:32.022+0000 7fe912c39700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe90c15da50 con 0x7fe90c0cb040 2023-10-06T03:29:32.031 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:32.022+0000 7fe912c39700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe90c06b7c0 con 0x7fe90c158de0 2023-10-06T03:29:32.031 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:32.022+0000 7fe912438700 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7fe90c15d090 0x7fe90c1591b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:55490/0 (socket says 172.21.15.110:55490) 2023-10-06T03:29:32.031 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:32.022+0000 7fe912438700 1 -- 172.21.15.110:0/2606792537 learned_addr learned my addr 172.21.15.110:0/2606792537 (peer_addr_for_me v1:172.21.15.110:0/0) 2023-10-06T03:29:32.035 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:32.022+0000 7fe911436700 1 --2- 172.21.15.110:0/2606792537 >> v2:172.21.15.192:3300/0 conn(0x7fe90c0cb040 0x7fe90c15cb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:32.035 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:32.022+0000 7fe910c35700 1 -- 172.21.15.110:0/2606792537 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2101877896 0 0) 0x7fe904001490 con 0x7fe90c15d090 2023-10-06T03:29:32.035 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:32.022+0000 7fe911c37700 1 --2- 172.21.15.110:0/2606792537 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe90c158de0 0x7fe90c0cab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:32.035 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:32.022+0000 7fe910c35700 1 -- 172.21.15.110:0/2606792537 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe8f0003570 con 0x7fe90c15d090 2023-10-06T03:29:32.035 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:32.026+0000 7fe910c35700 1 -- 172.21.15.110:0/2606792537 <== mon.0 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 839019869 0 0) 0x7fe8f0003570 con 0x7fe90c15d090 2023-10-06T03:29:32.035 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:32.026+0000 7fe910c35700 1 -- 172.21.15.110:0/2606792537 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe90c158de0 msgr2=0x7fe90c0cab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:29:32.036 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:32.026+0000 7fe910c35700 1 --2- 172.21.15.110:0/2606792537 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe90c158de0 0x7fe90c0cab00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:32.036 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:32.026+0000 7fe910c35700 1 -- 172.21.15.110:0/2606792537 >> v2:172.21.15.192:3300/0 conn(0x7fe90c0cb040 msgr2=0x7fe90c15cb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:29:32.036 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:32.026+0000 7fe910c35700 1 --2- 172.21.15.110:0/2606792537 >> v2:172.21.15.192:3300/0 conn(0x7fe90c0cb040 0x7fe90c15cb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:32.036 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:32.026+0000 7fe910c35700 1 -- 172.21.15.110:0/2606792537 --> v1:172.21.15.110:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe90c06ba60 con 0x7fe90c15d090 2023-10-06T03:29:32.036 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:32.026+0000 7fe910c35700 1 -- 172.21.15.110:0/2606792537 <== mon.0 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 118603703 0 0) 0x7fe904001980 con 0x7fe90c15d090 2023-10-06T03:29:32.036 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:32.026+0000 7fe910c35700 1 -- 172.21.15.110:0/2606792537 >> v1:172.21.15.110:6789/0 conn(0x7fe90c15d090 legacy=0x7fe90c1591b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:32.037 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:32.026+0000 7fe910c35700 1 --2- 172.21.15.110:0/2606792537 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe8f0003a60 0x7fe8f0003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:32.037 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:32.026+0000 7fe910c35700 1 --2- 172.21.15.110:0/2606792537 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe8f0004810 0x7fe8f0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:32.037 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:32.026+0000 7fe910c35700 1 --2- 172.21.15.110:0/2606792537 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe8f00051c0 0x7fe8f000ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:32.037 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:32.026+0000 7fe910c35700 1 -- 172.21.15.110:0/2606792537 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe90c06b7c0 con 0x7fe8f0004810 2023-10-06T03:29:32.037 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:32.026+0000 7fe910c35700 1 -- 172.21.15.110:0/2606792537 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8f000c080 con 0x7fe8f00051c0 2023-10-06T03:29:32.037 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:32.026+0000 7fe910c35700 1 -- 172.21.15.110:0/2606792537 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8f000c240 con 0x7fe8f0003a60 2023-10-06T03:29:32.037 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:32.026+0000 7fe911c37700 1 --2- 172.21.15.110:0/2606792537 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe90c158de0 0x7fe90c0cab00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-06T03:29:32.037 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:32.026+0000 7fe911436700 1 --2- 172.21.15.110:0/2606792537 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe8f0004810 0x7fe8f0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:32.038 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:32.026+0000 7fe912438700 1 --2- 172.21.15.110:0/2606792537 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe8f00051c0 0x7fe8f000ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:32.038 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:32.026+0000 7fe911436700 1 -- 172.21.15.110:0/2606792537 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe8f0003a60 msgr2=0x7fe8f0003e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:29:32.038 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:32.026+0000 7fe911c37700 1 --2- 172.21.15.110:0/2606792537 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe8f0003a60 0x7fe8f0003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:32.038 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:32.026+0000 7fe911436700 1 --2- 172.21.15.110:0/2606792537 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe8f0003a60 0x7fe8f0003e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:32.038 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:32.026+0000 7fe911436700 1 -- 172.21.15.110:0/2606792537 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe8f00051c0 msgr2=0x7fe8f000ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:29:32.038 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:32.026+0000 7fe911436700 1 --2- 172.21.15.110:0/2606792537 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe8f00051c0 0x7fe8f000ba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:32.038 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:32.026+0000 7fe911436700 1 -- 172.21.15.110:0/2606792537 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8f0003570 con 0x7fe8f0004810 2023-10-06T03:29:32.038 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:32.026+0000 7fe911c37700 1 --2- 172.21.15.110:0/2606792537 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe8f0003a60 0x7fe8f0003e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-06T03:29:32.038 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:32.026+0000 7fe911436700 1 --2- 172.21.15.110:0/2606792537 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe8f0004810 0x7fe8f0004c80 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fe8f800bb90 tx=0x7fe8f8008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c8729479931c75b8 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:29:32.038 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:32.026+0000 7fe910c35700 1 -- 172.21.15.110:0/2606792537 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8f800d5d0 con 0x7fe8f0004810 2023-10-06T03:29:32.039 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:32.026+0000 7fe910c35700 1 -- 172.21.15.110:0/2606792537 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe8f800d730 con 0x7fe8f0004810 2023-10-06T03:29:32.039 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:32.026+0000 7fe910c35700 1 -- 172.21.15.110:0/2606792537 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8f800daf0 con 0x7fe8f0004810 2023-10-06T03:29:32.039 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:32.026+0000 7fe912c39700 1 -- 172.21.15.110:0/2606792537 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe8f0004810 msgr2=0x7fe8f0004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:32.039 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:32.026+0000 7fe912c39700 1 --2- 172.21.15.110:0/2606792537 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe8f0004810 0x7fe8f0004c80 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fe8f800bb90 tx=0x7fe8f8008040 comp rx=0 tx=0).stop 2023-10-06T03:29:32.039 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:32.026+0000 7fe912438700 1 -- 172.21.15.110:0/2606792537 reap_dead start 2023-10-06T03:29:32.039 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:32.026+0000 7fe912c39700 1 -- 172.21.15.110:0/2606792537 shutdown_connections 2023-10-06T03:29:32.040 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:32.026+0000 7fe912c39700 1 -- 172.21.15.110:0/2606792537 >> 172.21.15.110:0/2606792537 conn(0x7fe90c0c8d80 msgr2=0x7fe90c0c9180 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:29:32.040 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:32.026+0000 7fe912c39700 1 -- 172.21.15.110:0/2606792537 shutdown_connections 2023-10-06T03:29:32.040 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:32.026+0000 7fe912c39700 1 -- 172.21.15.110:0/2606792537 wait complete. 2023-10-06T03:29:32.040 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:32.026+0000 7fe912c39700 1 Processor -- start 2023-10-06T03:29:32.040 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:32.030+0000 7fe912c39700 1 -- start start 2023-10-06T03:29:32.042 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:32.030+0000 7fe912c39700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe8f0004810 0x7fe90c155820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:32.042 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:32.030+0000 7fe912c39700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe90c0abb10 0x7fe90c0ade70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:32.042 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:32.030+0000 7fe912c39700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe90c0c24d0 0x7fe90c0c2970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:32.043 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:32.030+0000 7fe912c39700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe90c0c2eb0 con 0x7fe90c0abb10 2023-10-06T03:29:32.043 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:32.030+0000 7fe912c39700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe90c0abf80 con 0x7fe8f0004810 2023-10-06T03:29:32.043 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:32.030+0000 7fe912c39700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe90c0ac260 con 0x7fe90c0c24d0 2023-10-06T03:29:32.043 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:32.030+0000 7fe911436700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe90c0abb10 0x7fe90c0ade70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:32.043 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:32.030+0000 7fe911436700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe90c0abb10 0x7fe90c0ade70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:36584/0 (socket says 172.21.15.110:36584) 2023-10-06T03:29:32.043 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:32.030+0000 7fe911436700 1 -- 172.21.15.110:0/605292194 learned_addr learned my addr 172.21.15.110:0/605292194 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:29:32.043 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:32.030+0000 7fe911c37700 1 --2- 172.21.15.110:0/605292194 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe8f0004810 0x7fe90c155820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:32.043 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:32.030+0000 7fe912438700 1 --2- 172.21.15.110:0/605292194 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe90c0c24d0 0x7fe90c0c2970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:32.043 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:32.030+0000 7fe911436700 1 -- 172.21.15.110:0/605292194 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe90c0c24d0 msgr2=0x7fe90c0c2970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:32.043 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:32.030+0000 7fe911436700 1 --2- 172.21.15.110:0/605292194 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe90c0c24d0 0x7fe90c0c2970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:32.044 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:32.030+0000 7fe911436700 1 -- 172.21.15.110:0/605292194 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe8f0004810 msgr2=0x7fe90c155820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:32.044 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:32.030+0000 7fe911436700 1 --2- 172.21.15.110:0/605292194 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe8f0004810 0x7fe90c155820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:32.044 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:32.030+0000 7fe911436700 1 -- 172.21.15.110:0/605292194 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe90c0ac540 con 0x7fe90c0abb10 2023-10-06T03:29:32.044 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:32.030+0000 7fe911c37700 1 --2- 172.21.15.110:0/605292194 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe8f0004810 0x7fe90c155820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-06T03:29:32.044 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:32.030+0000 7fe911436700 1 --2- 172.21.15.110:0/605292194 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe90c0abb10 0x7fe90c0ade70 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fe8f800bfd0 tx=0x7fe8f801b000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:29:32.044 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:32.030+0000 7fe909ffb700 1 -- 172.21.15.110:0/605292194 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8f8014740 con 0x7fe90c0abb10 2023-10-06T03:29:32.044 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:32.030+0000 7fe909ffb700 1 -- 172.21.15.110:0/605292194 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe8f80148c0 con 0x7fe90c0abb10 2023-10-06T03:29:32.044 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:32.030+0000 7fe909ffb700 1 -- 172.21.15.110:0/605292194 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8f8014c80 con 0x7fe90c0abb10 2023-10-06T03:29:32.044 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:32.030+0000 7fe912c39700 1 -- 172.21.15.110:0/605292194 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe90c0ac820 con 0x7fe90c0abb10 2023-10-06T03:29:32.044 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:32.030+0000 7fe912c39700 1 -- 172.21.15.110:0/605292194 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe90c0af8e0 con 0x7fe90c0abb10 2023-10-06T03:29:32.044 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:32.034+0000 7fe912c39700 1 -- 172.21.15.110:0/605292194 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe90c0afb90 con 0x7fe90c0abb10 2023-10-06T03:29:32.045 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:32.034+0000 7fe909ffb700 1 -- 172.21.15.110:0/605292194 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7fe8f801d610 con 0x7fe90c0abb10 2023-10-06T03:29:32.045 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:32.034+0000 7fe909ffb700 1 --2- 172.21.15.110:0/605292194 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fe8f4060550 0x7fe8f4062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:32.045 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:32.034+0000 7fe911c37700 1 --2- 172.21.15.110:0/605292194 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fe8f4060550 0x7fe8f4062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:32.045 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:32.034+0000 7fe909ffb700 1 -- 172.21.15.110:0/605292194 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5206+0+0 (secure 0 0 0) 0x7fe8f807df30 con 0x7fe90c0abb10 2023-10-06T03:29:32.045 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:32.034+0000 7fe911c37700 1 --2- 172.21.15.110:0/605292194 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fe8f4060550 0x7fe8f4062a00 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fe8fc006d60 tx=0x7fe8fc008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:29:32.048 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:32.042+0000 7fe909ffb700 1 -- 172.21.15.110:0/605292194 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fe90c0afb90 con 0x7fe90c0abb10 2023-10-06T03:29:32.208 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:32.202+0000 7fe912c39700 1 -- 172.21.15.110:0/605292194 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7fe90c0afb90 con 0x7fe90c0abb10 2023-10-06T03:29:32.511 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:32.502+0000 7fe909ffb700 1 -- 172.21.15.110:0/605292194 <== mon.0 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v21) v1 ==== 138+0+0 (secure 0 0 0) 0x7fe90c0afb90 con 0x7fe90c0abb10 2023-10-06T03:29:32.516 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:32.510+0000 7fe912c39700 1 -- 172.21.15.110:0/605292194 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fe8f4060550 msgr2=0x7fe8f4062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:32.516 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:32.510+0000 7fe912c39700 1 --2- 172.21.15.110:0/605292194 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fe8f4060550 0x7fe8f4062a00 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fe8fc006d60 tx=0x7fe8fc008040 comp rx=0 tx=0).stop 2023-10-06T03:29:32.516 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:32.510+0000 7fe912c39700 1 -- 172.21.15.110:0/605292194 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe90c0abb10 msgr2=0x7fe90c0ade70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:32.516 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:32.510+0000 7fe912c39700 1 --2- 172.21.15.110:0/605292194 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe90c0abb10 0x7fe90c0ade70 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fe8f800bfd0 tx=0x7fe8f801b000 comp rx=0 tx=0).stop 2023-10-06T03:29:32.517 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:32.510+0000 7fe912c39700 1 -- 172.21.15.110:0/605292194 shutdown_connections 2023-10-06T03:29:32.517 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:32.510+0000 7fe912c39700 1 --2- 172.21.15.110:0/605292194 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fe8f4060550 0x7fe8f4062a00 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:32.517 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:32.510+0000 7fe912c39700 1 --2- 172.21.15.110:0/605292194 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe90c0c24d0 0x7fe90c0c2970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:32.517 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:32.510+0000 7fe912c39700 1 --2- 172.21.15.110:0/605292194 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe90c0abb10 0x7fe90c0ade70 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:32.517 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:32.510+0000 7fe912c39700 1 --2- 172.21.15.110:0/605292194 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe8f0004810 0x7fe90c155820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:32.517 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:32.510+0000 7fe912c39700 1 -- 172.21.15.110:0/605292194 >> 172.21.15.110:0/605292194 conn(0x7fe90c0c8d80 msgr2=0x7fe90c062ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:29:32.517 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:32.510+0000 7fe912c39700 1 -- 172.21.15.110:0/605292194 shutdown_connections 2023-10-06T03:29:32.517 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:32.510+0000 7fe912c39700 1 -- 172.21.15.110:0/605292194 wait complete. 2023-10-06T03:29:32.518 INFO:teuthology.orchestra.run.smithi110.stderr:pool 'cephfs_metadata' created 2023-10-06T03:29:32.530 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.05 2023-10-06T03:29:32.651 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:32.642+0000 7f144aab9700 1 Processor -- start 2023-10-06T03:29:32.651 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:32.642+0000 7f144aab9700 1 -- start start 2023-10-06T03:29:32.652 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:32.646+0000 7f144aab9700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f144406cd40 0x7f14441558f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:32.652 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:32.646+0000 7f144aab9700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f1444155e30 0x7f14440afa80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:32.652 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:32.646+0000 7f144aab9700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14440af5f0 con 0x7f144406c970 2023-10-06T03:29:32.653 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:32.646+0000 7f144aab9700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f14440b1f00 con 0x7f1444155e30 2023-10-06T03:29:32.653 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:32.646+0000 7f144aab9700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14440aee10 con 0x7f144406cd40 2023-10-06T03:29:32.653 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:32.646+0000 7f1449ab7700 1 --1- >> v1:172.21.15.192:6789/0 conn(0x7f144406c970 0x7f14440ae700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.110:54326/0 (socket says 172.21.15.110:54326) 2023-10-06T03:29:32.653 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:32.646+0000 7f14492b6700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f144406cd40 0x7f14441558f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:32.653 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:32.646+0000 7f1449ab7700 1 -- 172.21.15.110:0/2767134100 learned_addr learned my addr 172.21.15.110:0/2767134100 (peer_addr_for_me v1:172.21.15.110:0/0) 2023-10-06T03:29:32.654 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:32.646+0000 7f144a2b8700 1 --2- 172.21.15.110:0/2767134100 >> v2:172.21.15.192:3300/0 conn(0x7f1444155e30 0x7f14440afa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:32.654 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:32.646+0000 7f14492b6700 1 -- 172.21.15.110:0/2767134100 >> v2:172.21.15.192:3300/0 conn(0x7f1444155e30 msgr2=0x7f14440afa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:29:32.654 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:32.646+0000 7f14492b6700 1 --2- 172.21.15.110:0/2767134100 >> v2:172.21.15.192:3300/0 conn(0x7f1444155e30 0x7f14440afa80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:32.654 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:32.646+0000 7f14492b6700 1 -- 172.21.15.110:0/2767134100 >> v1:172.21.15.192:6789/0 conn(0x7f144406c970 legacy=0x7f14440ae700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:32.654 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:32.646+0000 7f14492b6700 1 -- 172.21.15.110:0/2767134100 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14440aba00 con 0x7f144406cd40 2023-10-06T03:29:32.654 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:32.646+0000 7f1448ab5700 1 -- 172.21.15.110:0/2767134100 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2271019180 0 0) 0x7f14440af5f0 con 0x7f144406c970 2023-10-06T03:29:32.655 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:32.646+0000 7f14492b6700 1 --2- 172.21.15.110:0/2767134100 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f144406cd40 0x7f14441558f0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f14340098e0 tx=0x7f14340091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d2daa2ee54968d37 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:29:32.655 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:32.646+0000 7f1448ab5700 1 -- 172.21.15.110:0/2767134100 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f143400b640 con 0x7f144406cd40 2023-10-06T03:29:32.655 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:32.646+0000 7f1448ab5700 1 -- 172.21.15.110:0/2767134100 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f143400b7a0 con 0x7f144406cd40 2023-10-06T03:29:32.655 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:32.646+0000 7f1448ab5700 1 -- 172.21.15.110:0/2767134100 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f143400bb40 con 0x7f144406cd40 2023-10-06T03:29:32.656 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:32.646+0000 7f144aab9700 1 -- 172.21.15.110:0/2767134100 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f144406cd40 msgr2=0x7f14441558f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:32.656 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:32.646+0000 7f144aab9700 1 --2- 172.21.15.110:0/2767134100 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f144406cd40 0x7f14441558f0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f14340098e0 tx=0x7f14340091a0 comp rx=0 tx=0).stop 2023-10-06T03:29:32.656 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:32.650+0000 7f144aab9700 1 -- 172.21.15.110:0/2767134100 shutdown_connections 2023-10-06T03:29:32.656 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:32.650+0000 7f144aab9700 1 --2- 172.21.15.110:0/2767134100 >> v2:172.21.15.192:3300/0 conn(0x7f1444155e30 0x7f14440afa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:32.656 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:32.650+0000 7f144aab9700 1 --2- 172.21.15.110:0/2767134100 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f144406cd40 0x7f14441558f0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:32.656 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:32.650+0000 7f144aab9700 1 -- 172.21.15.110:0/2767134100 >> 172.21.15.110:0/2767134100 conn(0x7f144406ab30 msgr2=0x7f14440adff0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:29:32.656 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:32.650+0000 7f144aab9700 1 -- 172.21.15.110:0/2767134100 shutdown_connections 2023-10-06T03:29:32.656 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:32.650+0000 7f144aab9700 1 -- 172.21.15.110:0/2767134100 wait complete. 2023-10-06T03:29:32.657 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:32.650+0000 7f144aab9700 1 Processor -- start 2023-10-06T03:29:32.658 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:32.650+0000 7f144aab9700 1 -- start start 2023-10-06T03:29:32.659 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:32.654+0000 7f144aab9700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f144406c970 0x7f1444139fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:32.659 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:32.654+0000 7f144aab9700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f144406cd40 0x7f144415a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:32.659 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:32.654+0000 7f144aab9700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1444155e30 0x7f144415add0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:32.660 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:32.654+0000 7f144aab9700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f144414b440 con 0x7f1444155e30 2023-10-06T03:29:32.660 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:32.654+0000 7f144aab9700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f144414a830 con 0x7f144406cd40 2023-10-06T03:29:32.660 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:32.654+0000 7f144aab9700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f144414ab10 con 0x7f144406c970 2023-10-06T03:29:32.661 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:32.654+0000 7f144a2b8700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1444155e30 0x7f144415add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:32.662 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:32.654+0000 7f14492b6700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f144406cd40 0x7f144415a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:32.662 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:32.654+0000 7f1449ab7700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f144406c970 0x7f1444139fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:32.662 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:32.654+0000 7f14492b6700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f144406cd40 0x7f144415a890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.110:43248/0 (socket says 172.21.15.110:43248) 2023-10-06T03:29:32.663 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:32.654+0000 7f144a2b8700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1444155e30 0x7f144415add0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:36600/0 (socket says 172.21.15.110:36600) 2023-10-06T03:29:32.663 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:32.654+0000 7f14492b6700 1 -- 172.21.15.110:0/4028045287 learned_addr learned my addr 172.21.15.110:0/4028045287 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:29:32.663 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:32.654+0000 7f1449ab7700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f144406c970 0x7f1444139fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3301/0 says I am v2:172.21.15.110:33088/0 (socket says 172.21.15.110:33088) 2023-10-06T03:29:32.663 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:32.654+0000 7f14492b6700 1 -- 172.21.15.110:0/4028045287 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f144406c970 msgr2=0x7f1444139fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:32.663 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:32.654+0000 7f14492b6700 1 --2- 172.21.15.110:0/4028045287 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f144406c970 0x7f1444139fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:32.664 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:32.654+0000 7f14492b6700 1 -- 172.21.15.110:0/4028045287 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1444155e30 msgr2=0x7f144415add0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:32.664 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:32.654+0000 7f14492b6700 1 --2- 172.21.15.110:0/4028045287 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1444155e30 0x7f144415add0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:32.664 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:32.654+0000 7f14492b6700 1 -- 172.21.15.110:0/4028045287 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1434009420 con 0x7f144406cd40 2023-10-06T03:29:32.664 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:32.654+0000 7f1449ab7700 1 --2- 172.21.15.110:0/4028045287 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f144406c970 0x7f1444139fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-06T03:29:32.664 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:32.654+0000 7f14492b6700 1 --2- 172.21.15.110:0/4028045287 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f144406cd40 0x7f144415a890 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f14440af560 tx=0x7f1434017000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:29:32.664 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:32.654+0000 7f1439ffb700 1 -- 172.21.15.110:0/4028045287 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f143400bb20 con 0x7f144406cd40 2023-10-06T03:29:32.665 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:32.654+0000 7f1439ffb700 1 -- 172.21.15.110:0/4028045287 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1434000c90 con 0x7f144406cd40 2023-10-06T03:29:32.665 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:32.654+0000 7f144aab9700 1 -- 172.21.15.110:0/4028045287 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f144414adf0 con 0x7f144406cd40 2023-10-06T03:29:32.665 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:32.654+0000 7f1439ffb700 1 -- 172.21.15.110:0/4028045287 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1434011630 con 0x7f144406cd40 2023-10-06T03:29:32.665 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:32.654+0000 7f144aab9700 1 -- 172.21.15.110:0/4028045287 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f144413a520 con 0x7f144406cd40 2023-10-06T03:29:32.666 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:32.658+0000 7f144aab9700 1 -- 172.21.15.110:0/4028045287 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1444070ff0 con 0x7f144406cd40 2023-10-06T03:29:32.666 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:32.658+0000 7f1439ffb700 1 -- 172.21.15.110:0/4028045287 <== mon.1 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7f14340119a0 con 0x7f144406cd40 2023-10-06T03:29:32.667 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:32.658+0000 7f1439ffb700 1 --2- 172.21.15.110:0/4028045287 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f1428060580 0x7f1428062a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:32.668 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:32.658+0000 7f1449ab7700 1 --2- 172.21.15.110:0/4028045287 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f1428060580 0x7f1428062a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:32.668 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:32.658+0000 7f1439ffb700 1 -- 172.21.15.110:0/4028045287 <== mon.1 v2:172.21.15.192:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5576+0+0 (secure 0 0 0) 0x7f143407da80 con 0x7f144406cd40 2023-10-06T03:29:32.668 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:32.658+0000 7f1449ab7700 1 --2- 172.21.15.110:0/4028045287 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f1428060580 0x7f1428062a30 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f1440007fd0 tx=0x7f1440007d30 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:29:32.676 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:32.670+0000 7f1439ffb700 1 -- 172.21.15.110:0/4028045287 <== mon.1 v2:172.21.15.192:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f143404e050 con 0x7f144406cd40 2023-10-06T03:29:32.843 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:32.834+0000 7f144aab9700 1 -- 172.21.15.110:0/4028045287 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05} v 0) v1 -- 0x7f14440638b0 con 0x7f144406cd40 2023-10-06T03:29:33.525 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:33.518+0000 7f1439ffb700 1 -- 172.21.15.110:0/4028045287 <== mon.1 v2:172.21.15.192:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]=0 pool 'cephfs_data' created v22) v1 ==== 171+0+0 (secure 0 0 0) 0x7f1434051670 con 0x7f144406cd40 2023-10-06T03:29:33.550 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:33.542+0000 7f144aab9700 1 -- 172.21.15.110:0/4028045287 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f1428060580 msgr2=0x7f1428062a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:33.550 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:33.542+0000 7f144aab9700 1 --2- 172.21.15.110:0/4028045287 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f1428060580 0x7f1428062a30 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f1440007fd0 tx=0x7f1440007d30 comp rx=0 tx=0).stop 2023-10-06T03:29:33.550 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:33.542+0000 7f144aab9700 1 -- 172.21.15.110:0/4028045287 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f144406cd40 msgr2=0x7f144415a890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:33.550 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:33.542+0000 7f144aab9700 1 --2- 172.21.15.110:0/4028045287 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f144406cd40 0x7f144415a890 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f14440af560 tx=0x7f1434017000 comp rx=0 tx=0).stop 2023-10-06T03:29:33.551 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:33.542+0000 7f144aab9700 1 -- 172.21.15.110:0/4028045287 shutdown_connections 2023-10-06T03:29:33.551 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:33.542+0000 7f144aab9700 1 --2- 172.21.15.110:0/4028045287 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f1428060580 0x7f1428062a30 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:33.551 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:33.542+0000 7f144aab9700 1 --2- 172.21.15.110:0/4028045287 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1444155e30 0x7f144415add0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:33.551 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:33.542+0000 7f144aab9700 1 --2- 172.21.15.110:0/4028045287 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f144406cd40 0x7f144415a890 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:33.551 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:33.542+0000 7f144aab9700 1 --2- 172.21.15.110:0/4028045287 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f144406c970 0x7f1444139fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:33.551 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:33.542+0000 7f144aab9700 1 -- 172.21.15.110:0/4028045287 >> 172.21.15.110:0/4028045287 conn(0x7f144406ab30 msgr2=0x7f14440ac310 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:29:33.551 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:33.542+0000 7f144aab9700 1 -- 172.21.15.110:0/4028045287 shutdown_connections 2023-10-06T03:29:33.551 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:33.542+0000 7f144aab9700 1 -- 172.21.15.110:0/4028045287 wait complete. 2023-10-06T03:29:33.551 INFO:teuthology.orchestra.run.smithi110.stderr:pool 'cephfs_data' created 2023-10-06T03:29:33.561 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2023-10-06T03:29:33.666 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:33.658+0000 7fd773a8e700 1 Processor -- start 2023-10-06T03:29:33.667 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:33.658+0000 7fd773a8e700 1 -- start start 2023-10-06T03:29:33.668 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:33.662+0000 7fd773a8e700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fd76c0af070 0x7fd76c0af440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:33.668 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:33.662+0000 7fd773a8e700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd76c0cf5b0 con 0x7fd76c0cfa40 2023-10-06T03:29:33.668 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:33.662+0000 7fd773a8e700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd76c0ce9a0 con 0x7fd76c15d360 2023-10-06T03:29:33.668 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:33.662+0000 7fd773a8e700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd76c0cec80 con 0x7fd76c0af070 2023-10-06T03:29:33.668 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:33.662+0000 7fd771029700 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7fd76c0cfa40 0x7fd76c15cc50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:55498/0 (socket says 172.21.15.110:55498) 2023-10-06T03:29:33.668 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:33.662+0000 7fd771029700 1 -- 172.21.15.110:0/3014921273 learned_addr learned my addr 172.21.15.110:0/3014921273 (peer_addr_for_me v1:172.21.15.110:0/0) 2023-10-06T03:29:33.671 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:33.662+0000 7fd77182a700 1 --2- 172.21.15.110:0/3014921273 >> v2:172.21.15.192:3300/0 conn(0x7fd76c0af070 0x7fd76c0af440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:33.671 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:33.662+0000 7fd770828700 1 -- 172.21.15.110:0/3014921273 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 613320816 0 0) 0x7fd76c0cf5b0 con 0x7fd76c0cfa40 2023-10-06T03:29:33.672 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:33.662+0000 7fd770828700 1 -- 172.21.15.110:0/3014921273 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd758003570 con 0x7fd76c0cfa40 2023-10-06T03:29:33.672 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:33.662+0000 7fd770828700 1 -- 172.21.15.110:0/3014921273 <== mon.0 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 252469700 0 0) 0x7fd758003570 con 0x7fd76c0cfa40 2023-10-06T03:29:33.672 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:33.662+0000 7fd770828700 1 -- 172.21.15.110:0/3014921273 >> v2:172.21.15.192:3300/0 conn(0x7fd76c0af070 msgr2=0x7fd76c0af440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:29:33.672 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:33.662+0000 7fd770828700 1 --2- 172.21.15.110:0/3014921273 >> v2:172.21.15.192:3300/0 conn(0x7fd76c0af070 0x7fd76c0af440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:33.672 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:33.662+0000 7fd770828700 1 -- 172.21.15.110:0/3014921273 >> v1:172.21.15.192:6789/0 conn(0x7fd76c15d360 legacy=0x7fd76c0afbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:33.672 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:33.662+0000 7fd770828700 1 -- 172.21.15.110:0/3014921273 --> v1:172.21.15.110:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd76c0cef60 con 0x7fd76c0cfa40 2023-10-06T03:29:33.672 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:33.662+0000 7fd770828700 1 -- 172.21.15.110:0/3014921273 <== mon.0 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 118603703 0 0) 0x7fd768001470 con 0x7fd76c0cfa40 2023-10-06T03:29:33.672 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:33.662+0000 7fd770828700 1 -- 172.21.15.110:0/3014921273 >> v1:172.21.15.110:6789/0 conn(0x7fd76c0cfa40 legacy=0x7fd76c15cc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:33.672 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:33.662+0000 7fd770828700 1 --2- 172.21.15.110:0/3014921273 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd758003a60 0x7fd758003e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:33.672 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:33.662+0000 7fd770828700 1 --2- 172.21.15.110:0/3014921273 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd758004810 0x7fd758004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:33.673 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:33.662+0000 7fd770828700 1 --2- 172.21.15.110:0/3014921273 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd7580051c0 0x7fd75800ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:33.673 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:33.662+0000 7fd770828700 1 -- 172.21.15.110:0/3014921273 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd76c0cec80 con 0x7fd7580051c0 2023-10-06T03:29:33.673 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:33.662+0000 7fd770828700 1 -- 172.21.15.110:0/3014921273 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd75800c0e0 con 0x7fd758004810 2023-10-06T03:29:33.673 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:33.662+0000 7fd770828700 1 -- 172.21.15.110:0/3014921273 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd75800c2a0 con 0x7fd758003a60 2023-10-06T03:29:33.673 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:33.662+0000 7fd77202b700 1 --2- 172.21.15.110:0/3014921273 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd7580051c0 0x7fd75800ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:33.673 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:33.662+0000 7fd771029700 1 --2- 172.21.15.110:0/3014921273 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd758004810 0x7fd758004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:33.673 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:33.662+0000 7fd77182a700 1 --2- 172.21.15.110:0/3014921273 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd758003a60 0x7fd758003e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:33.673 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:33.662+0000 7fd77202b700 1 -- 172.21.15.110:0/3014921273 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd758003a60 msgr2=0x7fd758003e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:29:33.673 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:33.662+0000 7fd77202b700 1 --2- 172.21.15.110:0/3014921273 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd758003a60 0x7fd758003e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:33.673 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:33.662+0000 7fd77202b700 1 -- 172.21.15.110:0/3014921273 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd758004810 msgr2=0x7fd758004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:29:33.673 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:33.662+0000 7fd77202b700 1 --2- 172.21.15.110:0/3014921273 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd758004810 0x7fd758004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:33.674 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:33.662+0000 7fd77202b700 1 -- 172.21.15.110:0/3014921273 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd758003570 con 0x7fd7580051c0 2023-10-06T03:29:33.674 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:33.662+0000 7fd77202b700 1 -- 172.21.15.110:0/3014921273 reap_dead start 2023-10-06T03:29:33.674 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:33.662+0000 7fd77202b700 1 --2- 172.21.15.110:0/3014921273 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd7580051c0 0x7fd75800ba60 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fd764002070 tx=0x7fd7640016d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=203f0add14c8df42 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:29:33.674 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:33.662+0000 7fd770828700 1 -- 172.21.15.110:0/3014921273 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7640019e0 con 0x7fd7580051c0 2023-10-06T03:29:33.674 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:33.662+0000 7fd770828700 1 -- 172.21.15.110:0/3014921273 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd76400c8b0 con 0x7fd7580051c0 2023-10-06T03:29:33.674 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:33.662+0000 7fd770828700 1 -- 172.21.15.110:0/3014921273 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd76400cc70 con 0x7fd7580051c0 2023-10-06T03:29:33.674 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:33.662+0000 7fd773a8e700 1 -- 172.21.15.110:0/3014921273 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd7580051c0 msgr2=0x7fd75800ba60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:33.674 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:33.662+0000 7fd773a8e700 1 --2- 172.21.15.110:0/3014921273 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd7580051c0 0x7fd75800ba60 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fd764002070 tx=0x7fd7640016d0 comp rx=0 tx=0).stop 2023-10-06T03:29:33.674 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:33.662+0000 7fd773a8e700 1 -- 172.21.15.110:0/3014921273 shutdown_connections 2023-10-06T03:29:33.674 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:33.662+0000 7fd773a8e700 1 --2- 172.21.15.110:0/3014921273 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd7580051c0 0x7fd75800ba60 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:33.674 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:33.662+0000 7fd773a8e700 1 -- 172.21.15.110:0/3014921273 >> 172.21.15.110:0/3014921273 conn(0x7fd76c06e840 msgr2=0x7fd76c0d0010 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:29:33.675 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:33.662+0000 7fd773a8e700 1 -- 172.21.15.110:0/3014921273 shutdown_connections 2023-10-06T03:29:33.675 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:33.662+0000 7fd773a8e700 1 -- 172.21.15.110:0/3014921273 wait complete. 2023-10-06T03:29:33.677 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:33.666+0000 7fd773a8e700 1 Processor -- start 2023-10-06T03:29:33.677 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:33.666+0000 7fd773a8e700 1 -- start start 2023-10-06T03:29:33.678 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:33.666+0000 7fd773a8e700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd7580051c0 0x7fd76c159c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:33.678 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:33.666+0000 7fd773a8e700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd76c1415d0 0x7fd76c0b1350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:33.678 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:33.666+0000 7fd773a8e700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd76c141a40 0x7fd76c0b2db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:33.678 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:33.666+0000 7fd773a8e700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd76c0b0060 con 0x7fd76c141a40 2023-10-06T03:29:33.678 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:33.666+0000 7fd773a8e700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd76c0b0ec0 con 0x7fd7580051c0 2023-10-06T03:29:33.678 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:33.666+0000 7fd773a8e700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd76c0b02b0 con 0x7fd76c1415d0 2023-10-06T03:29:33.678 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:33.666+0000 7fd77202b700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd76c141a40 0x7fd76c0b2db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:33.678 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:33.666+0000 7fd77182a700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd7580051c0 0x7fd76c159c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:33.678 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:33.666+0000 7fd77202b700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd76c141a40 0x7fd76c0b2db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:36616/0 (socket says 172.21.15.110:36616) 2023-10-06T03:29:33.678 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:33.666+0000 7fd77202b700 1 -- 172.21.15.110:0/3081181690 learned_addr learned my addr 172.21.15.110:0/3081181690 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:29:33.678 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:33.666+0000 7fd771029700 1 --2- 172.21.15.110:0/3081181690 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd76c1415d0 0x7fd76c0b1350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:33.679 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:33.666+0000 7fd77182a700 1 -- 172.21.15.110:0/3081181690 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd76c1415d0 msgr2=0x7fd76c0b1350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:33.679 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:33.666+0000 7fd77182a700 1 --2- 172.21.15.110:0/3081181690 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd76c1415d0 0x7fd76c0b1350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:33.679 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:33.666+0000 7fd77182a700 1 -- 172.21.15.110:0/3081181690 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd76c141a40 msgr2=0x7fd76c0b2db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:33.679 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:33.666+0000 7fd77182a700 1 --2- 172.21.15.110:0/3081181690 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd76c141a40 0x7fd76c0b2db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:33.679 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:33.666+0000 7fd77182a700 1 -- 172.21.15.110:0/3081181690 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd76c0b0590 con 0x7fd7580051c0 2023-10-06T03:29:33.679 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:33.666+0000 7fd77202b700 1 --2- 172.21.15.110:0/3081181690 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd76c141a40 0x7fd76c0b2db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-06T03:29:33.680 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:33.666+0000 7fd77182a700 1 --2- 172.21.15.110:0/3081181690 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd7580051c0 0x7fd76c159c50 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fd7600116e0 tx=0x7fd760008040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:29:33.680 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:33.666+0000 7fd75dffb700 1 -- 172.21.15.110:0/3081181690 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd760015670 con 0x7fd7580051c0 2023-10-06T03:29:33.680 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:33.670+0000 7fd75dffb700 1 -- 172.21.15.110:0/3081181690 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd7600157d0 con 0x7fd7580051c0 2023-10-06T03:29:33.680 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:33.670+0000 7fd75dffb700 1 -- 172.21.15.110:0/3081181690 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd760015b70 con 0x7fd7580051c0 2023-10-06T03:29:33.680 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:33.670+0000 7fd773a8e700 1 -- 172.21.15.110:0/3081181690 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd76c0b0870 con 0x7fd7580051c0 2023-10-06T03:29:33.680 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:33.670+0000 7fd773a8e700 1 -- 172.21.15.110:0/3081181690 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd76c1552b0 con 0x7fd7580051c0 2023-10-06T03:29:33.680 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:33.670+0000 7fd773a8e700 1 -- 172.21.15.110:0/3081181690 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd76c06ff60 con 0x7fd7580051c0 2023-10-06T03:29:33.680 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:33.670+0000 7fd75dffb700 1 -- 172.21.15.110:0/3081181690 <== mon.1 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7fd760013410 con 0x7fd7580051c0 2023-10-06T03:29:33.681 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:33.670+0000 7fd75dffb700 1 --2- 172.21.15.110:0/3081181690 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fd750060550 0x7fd750062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:33.681 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:33.670+0000 7fd75dffb700 1 -- 172.21.15.110:0/3081181690 <== mon.1 v2:172.21.15.192:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5958+0+0 (secure 0 0 0) 0x7fd76007cb50 con 0x7fd7580051c0 2023-10-06T03:29:33.681 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:33.670+0000 7fd771029700 1 --2- 172.21.15.110:0/3081181690 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fd750060550 0x7fd750062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:33.681 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:33.670+0000 7fd771029700 1 --2- 172.21.15.110:0/3081181690 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fd750060550 0x7fd750062a00 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fd76c0b5e90 tx=0x7fd768001e70 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:29:33.688 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:33.682+0000 7fd75dffb700 1 -- 172.21.15.110:0/3081181690 <== mon.1 v2:172.21.15.192:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fd76004d040 con 0x7fd7580051c0 2023-10-06T03:29:33.893 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:33.886+0000 7fd773a8e700 1 -- 172.21.15.110:0/3081181690 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7fd76c1569c0 con 0x7fd7580051c0 2023-10-06T03:29:34.538 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:34.530+0000 7fd75dffb700 1 -- 172.21.15.110:0/3081181690 <== mon.1 v2:172.21.15.192:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 3 and data pool 4 v4) v1 ==== 172+0+0 (secure 0 0 0) 0x7fd760050660 con 0x7fd7580051c0 2023-10-06T03:29:34.550 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:34.542+0000 7fd773a8e700 1 -- 172.21.15.110:0/3081181690 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fd750060550 msgr2=0x7fd750062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:34.550 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:34.542+0000 7fd773a8e700 1 --2- 172.21.15.110:0/3081181690 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fd750060550 0x7fd750062a00 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fd76c0b5e90 tx=0x7fd768001e70 comp rx=0 tx=0).stop 2023-10-06T03:29:34.551 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:34.542+0000 7fd773a8e700 1 -- 172.21.15.110:0/3081181690 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd7580051c0 msgr2=0x7fd76c159c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:34.551 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:34.542+0000 7fd773a8e700 1 --2- 172.21.15.110:0/3081181690 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd7580051c0 0x7fd76c159c50 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fd7600116e0 tx=0x7fd760008040 comp rx=0 tx=0).stop 2023-10-06T03:29:34.551 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:34.542+0000 7fd773a8e700 1 -- 172.21.15.110:0/3081181690 shutdown_connections 2023-10-06T03:29:34.551 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:34.542+0000 7fd773a8e700 1 --2- 172.21.15.110:0/3081181690 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fd750060550 0x7fd750062a00 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:34.551 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:34.542+0000 7fd773a8e700 1 --2- 172.21.15.110:0/3081181690 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd76c141a40 0x7fd76c0b2db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:34.551 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:34.542+0000 7fd773a8e700 1 --2- 172.21.15.110:0/3081181690 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd76c1415d0 0x7fd76c0b1350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:34.552 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:34.542+0000 7fd773a8e700 1 --2- 172.21.15.110:0/3081181690 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd7580051c0 0x7fd76c159c50 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:34.552 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:34.542+0000 7fd773a8e700 1 -- 172.21.15.110:0/3081181690 >> 172.21.15.110:0/3081181690 conn(0x7fd76c06e840 msgr2=0x7fd76c155880 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:29:34.552 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:34.546+0000 7fd773a8e700 1 -- 172.21.15.110:0/3081181690 shutdown_connections 2023-10-06T03:29:34.552 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:34.546+0000 7fd773a8e700 1 -- 172.21.15.110:0/3081181690 wait complete. 2023-10-06T03:29:34.554 INFO:teuthology.orchestra.run.smithi110.stderr:new fs with metadata pool 3 and data pool 4 2023-10-06T03:29:34.567 DEBUG:tasks.cephfs.filesystem:EC profile is ['m=2', 'k=2', 'crush-failure-domain=osd'] 2023-10-06T03:29:34.568 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd erasure-code-profile set cephfs_data_ec m=2 k=2 crush-failure-domain=osd 2023-10-06T03:29:34.668 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:34.662+0000 7f2ce32a8700 1 Processor -- start 2023-10-06T03:29:34.668 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:34.662+0000 7f2ce32a8700 1 -- start start 2023-10-06T03:29:34.669 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:34.662+0000 7f2ce32a8700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2cdc0abd90 0x7f2cdc0ac210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:34.669 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:34.662+0000 7f2ce32a8700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2cdc1574c0 con 0x7f2cdc0ab9c0 2023-10-06T03:29:34.669 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:34.662+0000 7f2ce32a8700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2cdc0ae8e0 con 0x7f2cdc0ad760 2023-10-06T03:29:34.669 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:34.662+0000 7f2ce32a8700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2cdc0adcd0 con 0x7f2cdc0abd90 2023-10-06T03:29:34.669 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:34.662+0000 7f2ce1044700 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f2cdc0ab9c0 0x7f2cdc0ce620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:55504/0 (socket says 172.21.15.110:55504) 2023-10-06T03:29:34.669 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:34.662+0000 7f2ce1044700 1 -- 172.21.15.110:0/2012150341 learned_addr learned my addr 172.21.15.110:0/2012150341 (peer_addr_for_me v1:172.21.15.110:0/0) 2023-10-06T03:29:34.672 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:34.662+0000 7f2ce1845700 1 --2- 172.21.15.110:0/2012150341 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2cdc0abd90 0x7f2cdc0ac210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:34.672 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:34.662+0000 7f2cd3fff700 1 -- 172.21.15.110:0/2012150341 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4229982250 0 0) 0x7f2cdc1574c0 con 0x7f2cdc0ab9c0 2023-10-06T03:29:34.672 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:34.662+0000 7f2cd3fff700 1 -- 172.21.15.110:0/2012150341 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2cc8003570 con 0x7f2cdc0ab9c0 2023-10-06T03:29:34.672 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:34.662+0000 7f2cd3fff700 1 -- 172.21.15.110:0/2012150341 <== mon.0 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2227828962 0 0) 0x7f2cc8003570 con 0x7f2cdc0ab9c0 2023-10-06T03:29:34.673 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:34.662+0000 7f2cd3fff700 1 -- 172.21.15.110:0/2012150341 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2cdc0abd90 msgr2=0x7f2cdc0ac210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:29:34.673 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:34.662+0000 7f2cd3fff700 1 --2- 172.21.15.110:0/2012150341 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2cdc0abd90 0x7f2cdc0ac210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:34.673 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:34.662+0000 7f2cd3fff700 1 -- 172.21.15.110:0/2012150341 >> v1:172.21.15.192:6789/0 conn(0x7f2cdc0ad760 legacy=0x7f2cdc0aed70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:34.673 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:34.662+0000 7f2cd3fff700 1 -- 172.21.15.110:0/2012150341 --> v1:172.21.15.110:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cdc0adfb0 con 0x7f2cdc0ab9c0 2023-10-06T03:29:34.673 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:34.662+0000 7f2cd3fff700 1 -- 172.21.15.110:0/2012150341 <== mon.0 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 118603703 0 0) 0x7f2ccc001470 con 0x7f2cdc0ab9c0 2023-10-06T03:29:34.673 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:34.662+0000 7f2cd3fff700 1 -- 172.21.15.110:0/2012150341 >> v1:172.21.15.110:6789/0 conn(0x7f2cdc0ab9c0 legacy=0x7f2cdc0ce620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:34.673 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:34.662+0000 7f2cd3fff700 1 --2- 172.21.15.110:0/2012150341 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2cdc0abd90 0x7f2cc8003a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:34.673 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:34.662+0000 7f2cd3fff700 1 --2- 172.21.15.110:0/2012150341 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2cc8004010 0x7f2cc8004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:34.673 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:34.662+0000 7f2cd3fff700 1 --2- 172.21.15.110:0/2012150341 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2cc8004d50 0x7f2cc80051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:34.673 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:34.662+0000 7f2cd3fff700 1 -- 172.21.15.110:0/2012150341 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cdc0adcd0 con 0x7f2cc8004d50 2023-10-06T03:29:34.673 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:34.662+0000 7f2cd3fff700 1 -- 172.21.15.110:0/2012150341 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cc800bbf0 con 0x7f2cc8004010 2023-10-06T03:29:34.673 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:34.662+0000 7f2cd3fff700 1 -- 172.21.15.110:0/2012150341 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2cc800bdb0 con 0x7f2cdc0abd90 2023-10-06T03:29:34.674 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:34.662+0000 7f2ce1845700 1 --2- 172.21.15.110:0/2012150341 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2cc8004d50 0x7f2cc80051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:34.674 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:34.662+0000 7f2ce1044700 1 --2- 172.21.15.110:0/2012150341 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2cdc0abd90 0x7f2cc8003a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:34.674 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:34.662+0000 7f2ce1845700 1 -- 172.21.15.110:0/2012150341 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2cdc0abd90 msgr2=0x7f2cc8003a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:29:34.674 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:34.662+0000 7f2ce1845700 1 --2- 172.21.15.110:0/2012150341 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2cdc0abd90 0x7f2cc8003a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:34.674 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:34.662+0000 7f2ce1845700 1 -- 172.21.15.110:0/2012150341 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2cc8004010 msgr2=0x7f2cc8004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:29:34.674 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:34.662+0000 7f2ce0843700 1 --2- 172.21.15.110:0/2012150341 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2cc8004010 0x7f2cc8004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:34.674 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:34.662+0000 7f2ce1845700 1 --2- 172.21.15.110:0/2012150341 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2cc8004010 0x7f2cc8004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:34.674 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:34.662+0000 7f2ce1845700 1 -- 172.21.15.110:0/2012150341 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cc8003570 con 0x7f2cc8004d50 2023-10-06T03:29:34.674 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:34.662+0000 7f2ce0843700 1 --2- 172.21.15.110:0/2012150341 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2cc8004010 0x7f2cc8004810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-06T03:29:34.674 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:34.666+0000 7f2ce1845700 1 --2- 172.21.15.110:0/2012150341 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2cc8004d50 0x7f2cc80051c0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f2cd40094d0 tx=0x7f2cd400b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=56b57da16fc154b9 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:29:34.675 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:34.666+0000 7f2cd3fff700 1 -- 172.21.15.110:0/2012150341 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cd400ea90 con 0x7f2cc8004d50 2023-10-06T03:29:34.675 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:34.666+0000 7f2cd3fff700 1 -- 172.21.15.110:0/2012150341 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2cd400f040 con 0x7f2cc8004d50 2023-10-06T03:29:34.675 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:34.666+0000 7f2cd3fff700 1 -- 172.21.15.110:0/2012150341 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cd400ee20 con 0x7f2cc8004d50 2023-10-06T03:29:34.675 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:34.666+0000 7f2ce32a8700 1 -- 172.21.15.110:0/2012150341 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2cc8004d50 msgr2=0x7f2cc80051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:34.675 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:34.666+0000 7f2ce32a8700 1 --2- 172.21.15.110:0/2012150341 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2cc8004d50 0x7f2cc80051c0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f2cd40094d0 tx=0x7f2cd400b040 comp rx=0 tx=0).stop 2023-10-06T03:29:34.675 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:34.666+0000 7f2ce1845700 1 -- 172.21.15.110:0/2012150341 reap_dead start 2023-10-06T03:29:34.675 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:34.666+0000 7f2ce32a8700 1 -- 172.21.15.110:0/2012150341 shutdown_connections 2023-10-06T03:29:34.675 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:34.666+0000 7f2ce32a8700 1 -- 172.21.15.110:0/2012150341 >> 172.21.15.110:0/2012150341 conn(0x7f2cdc06bfa0 msgr2=0x7f2cdc0cdea0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:29:34.675 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:34.666+0000 7f2ce32a8700 1 -- 172.21.15.110:0/2012150341 shutdown_connections 2023-10-06T03:29:34.675 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:34.666+0000 7f2ce32a8700 1 -- 172.21.15.110:0/2012150341 wait complete. 2023-10-06T03:29:34.677 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:34.666+0000 7f2ce32a8700 1 Processor -- start 2023-10-06T03:29:34.677 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:34.666+0000 7f2ce32a8700 1 -- start start 2023-10-06T03:29:34.677 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:34.670+0000 7f2ce32a8700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2cc8004d50 0x7f2cdc06c8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:34.678 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:34.670+0000 7f2ce32a8700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2cdc157f90 0x7f2cdc159ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:34.678 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:34.670+0000 7f2ce32a8700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2cdc158400 0x7f2cdc15a410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:34.678 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:34.670+0000 7f2ce32a8700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cdc16de10 con 0x7f2cdc157f90 2023-10-06T03:29:34.678 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:34.670+0000 7f2ce32a8700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cdc154e90 con 0x7f2cc8004d50 2023-10-06T03:29:34.678 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:34.670+0000 7f2ce32a8700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2cdc155170 con 0x7f2cdc158400 2023-10-06T03:29:34.678 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:34.670+0000 7f2ce0843700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2cdc157f90 0x7f2cdc159ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:34.678 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:34.670+0000 7f2ce0843700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2cdc157f90 0x7f2cdc159ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:36630/0 (socket says 172.21.15.110:36630) 2023-10-06T03:29:34.678 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:34.670+0000 7f2ce0843700 1 -- 172.21.15.110:0/391956668 learned_addr learned my addr 172.21.15.110:0/391956668 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:29:34.678 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:34.670+0000 7f2ce1044700 1 --2- 172.21.15.110:0/391956668 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2cc8004d50 0x7f2cdc06c8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:34.678 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:34.670+0000 7f2ce1845700 1 --2- 172.21.15.110:0/391956668 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2cdc158400 0x7f2cdc15a410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:34.678 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:34.670+0000 7f2ce0843700 1 -- 172.21.15.110:0/391956668 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2cdc158400 msgr2=0x7f2cdc15a410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:34.679 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:34.670+0000 7f2ce0843700 1 --2- 172.21.15.110:0/391956668 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2cdc158400 0x7f2cdc15a410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:34.679 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:34.670+0000 7f2ce0843700 1 -- 172.21.15.110:0/391956668 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2cc8004d50 msgr2=0x7f2cdc06c8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:34.679 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:34.670+0000 7f2ce0843700 1 --2- 172.21.15.110:0/391956668 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2cc8004d50 0x7f2cdc06c8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:34.679 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:34.670+0000 7f2ce0843700 1 -- 172.21.15.110:0/391956668 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cdc155450 con 0x7f2cdc157f90 2023-10-06T03:29:34.679 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:34.670+0000 7f2ce1845700 1 --2- 172.21.15.110:0/391956668 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2cdc158400 0x7f2cdc15a410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-06T03:29:34.679 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:34.670+0000 7f2ce0843700 1 --2- 172.21.15.110:0/391956668 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2cdc157f90 0x7f2cdc159ed0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f2cd800d620 tx=0x7f2cd8009040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:29:34.679 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:34.670+0000 7f2cd17fa700 1 -- 172.21.15.110:0/391956668 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cd8006820 con 0x7f2cdc157f90 2023-10-06T03:29:34.679 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:34.670+0000 7f2cd17fa700 1 -- 172.21.15.110:0/391956668 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2cd8006980 con 0x7f2cdc157f90 2023-10-06T03:29:34.679 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:34.670+0000 7f2ce32a8700 1 -- 172.21.15.110:0/391956668 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2cdc155730 con 0x7f2cdc157f90 2023-10-06T03:29:34.679 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:34.670+0000 7f2cd17fa700 1 -- 172.21.15.110:0/391956668 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cd8006d40 con 0x7f2cdc157f90 2023-10-06T03:29:34.679 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:34.670+0000 7f2ce32a8700 1 -- 172.21.15.110:0/391956668 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2cdc06ce10 con 0x7f2cdc157f90 2023-10-06T03:29:34.682 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:34.670+0000 7f2ce32a8700 1 -- 172.21.15.110:0/391956668 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2cdc1570f0 con 0x7f2cdc157f90 2023-10-06T03:29:34.682 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:34.674+0000 7f2cd17fa700 1 -- 172.21.15.110:0/391956668 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7f2cd8007230 con 0x7f2cdc157f90 2023-10-06T03:29:34.682 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:34.674+0000 7f2cd17fa700 1 --2- 172.21.15.110:0/391956668 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f2cc0060580 0x7f2cc0062a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:34.682 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:34.674+0000 7f2cd17fa700 1 -- 172.21.15.110:0/391956668 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6058+0+0 (secure 0 0 0) 0x7f2cd807c620 con 0x7f2cdc157f90 2023-10-06T03:29:34.682 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:34.674+0000 7f2ce1044700 1 --2- 172.21.15.110:0/391956668 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f2cc0060580 0x7f2cc0062a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:34.682 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:34.674+0000 7f2ce1044700 1 --2- 172.21.15.110:0/391956668 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f2cc0060580 0x7f2cc0062a30 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f2ccc001c20 tx=0x7f2ccc009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:29:34.689 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:34.682+0000 7f2cd17fa700 1 -- 172.21.15.110:0/391956668 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f2cdc1570f0 con 0x7f2cdc157f90 2023-10-06T03:29:34.861 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:34.854+0000 7f2ce32a8700 1 -- 172.21.15.110:0/391956668 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]} v 0) v1 -- 0x7f2cdc1570f0 con 0x7f2cdc157f90 2023-10-06T03:29:35.504 INFO:tasks.ceph.mds.a.smithi110.stderr:2023-10-06T03:29:35.498+0000 7f8399c1b700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-06T03:29:35.525 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:35.518+0000 7f2cd17fa700 1 -- 172.21.15.110:0/391956668 <== mon.0 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]=0 v24) v1 ==== 157+0+0 (secure 0 0 0) 0x7f2cdc1570f0 con 0x7f2cdc157f90 2023-10-06T03:29:35.531 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:35.522+0000 7f2ce32a8700 1 -- 172.21.15.110:0/391956668 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f2cc0060580 msgr2=0x7f2cc0062a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:35.532 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:35.522+0000 7f2ce32a8700 1 --2- 172.21.15.110:0/391956668 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f2cc0060580 0x7f2cc0062a30 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f2ccc001c20 tx=0x7f2ccc009040 comp rx=0 tx=0).stop 2023-10-06T03:29:35.532 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:35.522+0000 7f2ce32a8700 1 -- 172.21.15.110:0/391956668 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2cdc157f90 msgr2=0x7f2cdc159ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:35.532 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:35.522+0000 7f2ce32a8700 1 --2- 172.21.15.110:0/391956668 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2cdc157f90 0x7f2cdc159ed0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f2cd800d620 tx=0x7f2cd8009040 comp rx=0 tx=0).stop 2023-10-06T03:29:35.532 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:35.522+0000 7f2ce32a8700 1 -- 172.21.15.110:0/391956668 shutdown_connections 2023-10-06T03:29:35.532 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:35.522+0000 7f2ce32a8700 1 --2- 172.21.15.110:0/391956668 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f2cc0060580 0x7f2cc0062a30 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:35.532 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:35.522+0000 7f2ce32a8700 1 --2- 172.21.15.110:0/391956668 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2cdc158400 0x7f2cdc15a410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:35.533 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:35.522+0000 7f2ce32a8700 1 --2- 172.21.15.110:0/391956668 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2cdc157f90 0x7f2cdc159ed0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:35.533 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:35.522+0000 7f2ce32a8700 1 --2- 172.21.15.110:0/391956668 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2cc8004d50 0x7f2cdc06c8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:35.533 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:35.522+0000 7f2ce32a8700 1 -- 172.21.15.110:0/391956668 >> 172.21.15.110:0/391956668 conn(0x7f2cdc06bfa0 msgr2=0x7f2cdc067a70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:29:35.533 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:35.526+0000 7f2ce32a8700 1 -- 172.21.15.110:0/391956668 shutdown_connections 2023-10-06T03:29:35.533 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:35.526+0000 7f2ce32a8700 1 -- 172.21.15.110:0/391956668 wait complete. 2023-10-06T03:29:35.544 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data_ec erasure cephfs_data_ec --pg_num_min 64 --target_size_ratio 0.9 2023-10-06T03:29:35.658 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:35.650+0000 7fc916a7c700 1 Processor -- start 2023-10-06T03:29:35.658 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:35.650+0000 7fc916a7c700 1 -- start start 2023-10-06T03:29:35.659 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:35.654+0000 7fc916a7c700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc91015cce0 0x7fc9100cac20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:35.659 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:35.654+0000 7fc916a7c700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fc9100cc3b0 0x7fc91015d0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:35.660 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:35.654+0000 7fc916a7c700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9100cbf30 con 0x7fc910158b20 2023-10-06T03:29:35.660 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:35.654+0000 7fc916a7c700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc91015dbe0 con 0x7fc9100cc3b0 2023-10-06T03:29:35.660 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:35.654+0000 7fc916a7c700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc91015d5f0 con 0x7fc91015cce0 2023-10-06T03:29:35.661 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:35.654+0000 7fc90ffff700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fc9100cc3b0 0x7fc91015d0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:35.661 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:35.654+0000 7fc915019700 1 --1- >> v1:172.21.15.192:6789/0 conn(0x7fc910158b20 0x7fc910158fa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.110:54352/0 (socket says 172.21.15.110:54352) 2023-10-06T03:29:35.661 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:35.654+0000 7fc914818700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc91015cce0 0x7fc9100cac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:35.661 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:35.654+0000 7fc915019700 1 -- 172.21.15.110:0/3462348364 learned_addr learned my addr 172.21.15.110:0/3462348364 (peer_addr_for_me v1:172.21.15.110:0/0) 2023-10-06T03:29:35.661 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:35.654+0000 7fc90f7fe700 1 -- 172.21.15.110:0/3462348364 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3431832584 0 0) 0x7fc9100cbf30 con 0x7fc910158b20 2023-10-06T03:29:35.661 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:35.654+0000 7fc90f7fe700 1 -- 172.21.15.110:0/3462348364 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc8f8003570 con 0x7fc910158b20 2023-10-06T03:29:35.661 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:35.654+0000 7fc90ffff700 1 -- 172.21.15.110:0/3462348364 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc91015cce0 msgr2=0x7fc9100cac20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:29:35.661 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:35.654+0000 7fc90ffff700 1 --2- 172.21.15.110:0/3462348364 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc91015cce0 0x7fc9100cac20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:35.661 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:35.654+0000 7fc90ffff700 1 -- 172.21.15.110:0/3462348364 >> v1:172.21.15.192:6789/0 conn(0x7fc910158b20 legacy=0x7fc910158fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:35.661 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:35.654+0000 7fc90ffff700 1 -- 172.21.15.110:0/3462348364 --> v2:172.21.15.192:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9100cb320 con 0x7fc9100cc3b0 2023-10-06T03:29:35.662 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:35.654+0000 7fc914818700 1 --2- 172.21.15.110:0/3462348364 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc91015cce0 0x7fc9100cac20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-06T03:29:35.663 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:35.654+0000 7fc90ffff700 1 --2- 172.21.15.110:0/3462348364 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc9100cc3b0 0x7fc91015d0b0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fc9040098e0 tx=0x7fc9040091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c19baa7d9f40d7dc server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:29:35.663 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:35.654+0000 7fc90f7fe700 1 -- 172.21.15.110:0/3462348364 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc904011630 con 0x7fc9100cc3b0 2023-10-06T03:29:35.663 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:35.654+0000 7fc90f7fe700 1 -- 172.21.15.110:0/3462348364 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc904011790 con 0x7fc9100cc3b0 2023-10-06T03:29:35.663 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:35.654+0000 7fc90f7fe700 1 -- 172.21.15.110:0/3462348364 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc904011b30 con 0x7fc9100cc3b0 2023-10-06T03:29:35.663 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:35.654+0000 7fc916a7c700 1 -- 172.21.15.110:0/3462348364 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc9100cc3b0 msgr2=0x7fc91015d0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:35.663 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:35.654+0000 7fc916a7c700 1 --2- 172.21.15.110:0/3462348364 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc9100cc3b0 0x7fc91015d0b0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fc9040098e0 tx=0x7fc9040091a0 comp rx=0 tx=0).stop 2023-10-06T03:29:35.663 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:35.654+0000 7fc916a7c700 1 -- 172.21.15.110:0/3462348364 shutdown_connections 2023-10-06T03:29:35.663 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:35.654+0000 7fc916a7c700 1 --2- 172.21.15.110:0/3462348364 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc9100cc3b0 0x7fc91015d0b0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:35.664 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:35.654+0000 7fc916a7c700 1 --2- 172.21.15.110:0/3462348364 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc91015cce0 0x7fc9100cac20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:35.664 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:35.654+0000 7fc916a7c700 1 -- 172.21.15.110:0/3462348364 >> 172.21.15.110:0/3462348364 conn(0x7fc9100d0940 msgr2=0x7fc9100cc940 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:29:35.664 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:35.654+0000 7fc916a7c700 1 -- 172.21.15.110:0/3462348364 shutdown_connections 2023-10-06T03:29:35.664 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:35.654+0000 7fc916a7c700 1 -- 172.21.15.110:0/3462348364 wait complete. 2023-10-06T03:29:35.665 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:35.658+0000 7fc916a7c700 1 Processor -- start 2023-10-06T03:29:35.665 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:35.658+0000 7fc916a7c700 1 -- start start 2023-10-06T03:29:35.667 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:35.658+0000 7fc916a7c700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc9100cc3b0 0x7fc910144ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:35.667 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:35.658+0000 7fc916a7c700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc910158b20 0x7fc91015b400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:35.667 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:35.658+0000 7fc916a7c700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc91015cce0 0x7fc9100b46a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:35.667 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:35.658+0000 7fc916a7c700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc91015ca40 con 0x7fc9100cc3b0 2023-10-06T03:29:35.668 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:35.658+0000 7fc916a7c700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc91015bec0 con 0x7fc91015cce0 2023-10-06T03:29:35.668 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:35.658+0000 7fc916a7c700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc91015c1a0 con 0x7fc910158b20 2023-10-06T03:29:35.668 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:35.662+0000 7fc914818700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc9100cc3b0 0x7fc910144ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:35.668 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:35.662+0000 7fc914818700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc9100cc3b0 0x7fc910144ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:36638/0 (socket says 172.21.15.110:36638) 2023-10-06T03:29:35.668 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:35.662+0000 7fc914818700 1 -- 172.21.15.110:0/1760953002 learned_addr learned my addr 172.21.15.110:0/1760953002 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:29:35.668 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:35.662+0000 7fc90ffff700 1 --2- 172.21.15.110:0/1760953002 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc910158b20 0x7fc91015b400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:35.668 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:35.662+0000 7fc915019700 1 --2- 172.21.15.110:0/1760953002 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc91015cce0 0x7fc9100b46a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:35.669 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:35.662+0000 7fc914818700 1 -- 172.21.15.110:0/1760953002 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc910158b20 msgr2=0x7fc91015b400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:35.669 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:35.662+0000 7fc914818700 1 --2- 172.21.15.110:0/1760953002 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc910158b20 0x7fc91015b400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:35.669 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:35.662+0000 7fc914818700 1 -- 172.21.15.110:0/1760953002 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc91015cce0 msgr2=0x7fc9100b46a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:35.669 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:35.662+0000 7fc914818700 1 --2- 172.21.15.110:0/1760953002 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc91015cce0 0x7fc9100b46a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:35.670 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:35.662+0000 7fc914818700 1 -- 172.21.15.110:0/1760953002 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc900009420 con 0x7fc9100cc3b0 2023-10-06T03:29:35.670 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:35.662+0000 7fc914818700 1 --2- 172.21.15.110:0/1760953002 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc9100cc3b0 0x7fc910144ec0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fc90000ec80 tx=0x7fc9000091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:29:35.670 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:35.662+0000 7fc90cff9700 1 -- 172.21.15.110:0/1760953002 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc90000c850 con 0x7fc9100cc3b0 2023-10-06T03:29:35.670 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:35.662+0000 7fc90cff9700 1 -- 172.21.15.110:0/1760953002 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc90000c9b0 con 0x7fc9100cc3b0 2023-10-06T03:29:35.670 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:35.662+0000 7fc916a7c700 1 -- 172.21.15.110:0/1760953002 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc904009420 con 0x7fc9100cc3b0 2023-10-06T03:29:35.670 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:35.662+0000 7fc90cff9700 1 -- 172.21.15.110:0/1760953002 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc90000cd30 con 0x7fc9100cc3b0 2023-10-06T03:29:35.670 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:35.662+0000 7fc916a7c700 1 -- 172.21.15.110:0/1760953002 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc91015c760 con 0x7fc9100cc3b0 2023-10-06T03:29:35.670 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:35.662+0000 7fc916a7c700 1 -- 172.21.15.110:0/1760953002 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc9100c6f40 con 0x7fc9100cc3b0 2023-10-06T03:29:35.671 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:35.666+0000 7fc90cff9700 1 -- 172.21.15.110:0/1760953002 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7fc900013680 con 0x7fc9100cc3b0 2023-10-06T03:29:35.671 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:35.666+0000 7fc90cff9700 1 --2- 172.21.15.110:0/1760953002 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fc8fc060550 0x7fc8fc062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:35.672 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:35.666+0000 7fc90cff9700 1 -- 172.21.15.110:0/1760953002 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6284+0+0 (secure 0 0 0) 0x7fc90007cfb0 con 0x7fc9100cc3b0 2023-10-06T03:29:35.672 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:35.666+0000 7fc90ffff700 1 --2- 172.21.15.110:0/1760953002 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fc8fc060550 0x7fc8fc062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:35.672 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:35.666+0000 7fc90ffff700 1 --2- 172.21.15.110:0/1760953002 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fc8fc060550 0x7fc8fc062a00 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fc9040091e0 tx=0x7fc904017000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:29:35.682 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:35.674+0000 7fc90cff9700 1 -- 172.21.15.110:0/1760953002 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fc9000acbe0 con 0x7fc9100cc3b0 2023-10-06T03:29:35.837 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:35.830+0000 7fc916a7c700 1 -- 172.21.15.110:0/1760953002 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7fc910070ff0 con 0x7fc9100cc3b0 2023-10-06T03:29:37.589 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:37.582+0000 7fc90cff9700 1 -- 172.21.15.110:0/1760953002 <== mon.0 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data_ec' created v26) v1 ==== 228+0+0 (secure 0 0 0) 0x7fc900017100 con 0x7fc9100cc3b0 2023-10-06T03:29:37.602 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:37.594+0000 7fc916a7c700 1 -- 172.21.15.110:0/1760953002 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fc8fc060550 msgr2=0x7fc8fc062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:37.602 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:37.594+0000 7fc916a7c700 1 --2- 172.21.15.110:0/1760953002 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fc8fc060550 0x7fc8fc062a00 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fc9040091e0 tx=0x7fc904017000 comp rx=0 tx=0).stop 2023-10-06T03:29:37.603 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:37.594+0000 7fc916a7c700 1 -- 172.21.15.110:0/1760953002 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc9100cc3b0 msgr2=0x7fc910144ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:37.603 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:37.594+0000 7fc916a7c700 1 --2- 172.21.15.110:0/1760953002 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc9100cc3b0 0x7fc910144ec0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fc90000ec80 tx=0x7fc9000091a0 comp rx=0 tx=0).stop 2023-10-06T03:29:37.603 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:37.594+0000 7fc916a7c700 1 -- 172.21.15.110:0/1760953002 shutdown_connections 2023-10-06T03:29:37.603 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:37.594+0000 7fc916a7c700 1 --2- 172.21.15.110:0/1760953002 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fc8fc060550 0x7fc8fc062a00 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:37.603 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:37.594+0000 7fc916a7c700 1 --2- 172.21.15.110:0/1760953002 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc91015cce0 0x7fc9100b46a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:37.603 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:37.594+0000 7fc916a7c700 1 --2- 172.21.15.110:0/1760953002 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc910158b20 0x7fc91015b400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:37.603 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:37.594+0000 7fc916a7c700 1 --2- 172.21.15.110:0/1760953002 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc9100cc3b0 0x7fc910144ec0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:37.603 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:37.594+0000 7fc916a7c700 1 -- 172.21.15.110:0/1760953002 >> 172.21.15.110:0/1760953002 conn(0x7fc9100d0940 msgr2=0x7fc910157390 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:29:37.603 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:37.594+0000 7fc916a7c700 1 -- 172.21.15.110:0/1760953002 shutdown_connections 2023-10-06T03:29:37.603 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:37.594+0000 7fc916a7c700 1 -- 172.21.15.110:0/1760953002 wait complete. 2023-10-06T03:29:37.604 INFO:teuthology.orchestra.run.smithi110.stderr:pool 'cephfs_data_ec' created 2023-10-06T03:29:37.614 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool set cephfs_data_ec allow_ec_overwrites true 2023-10-06T03:29:37.723 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:37.718+0000 7ffacf59e700 1 Processor -- start 2023-10-06T03:29:37.724 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:37.718+0000 7ffacf59e700 1 -- start start 2023-10-06T03:29:37.724 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:37.718+0000 7ffacf59e700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7ffad0156810 0x7ffad00ced70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:37.725 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:37.718+0000 7ffacf59e700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ffad0156be0 0x7ffad00b5820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:37.727 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:37.718+0000 7ffacf59e700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffad00b5d60 con 0x7ffad015caf0 2023-10-06T03:29:37.727 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:37.718+0000 7ffacf59e700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffad00cf3a0 con 0x7ffad0156810 2023-10-06T03:29:37.727 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:37.718+0000 7ffacf59e700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffad00cf680 con 0x7ffad0156be0 2023-10-06T03:29:37.727 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:37.718+0000 7ffaced9d700 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7ffad015caf0 0x7ffad015cf10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:55518/0 (socket says 172.21.15.110:55518) 2023-10-06T03:29:37.727 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:37.718+0000 7ffaced9d700 1 -- 172.21.15.110:0/2455574160 learned_addr learned my addr 172.21.15.110:0/2455574160 (peer_addr_for_me v1:172.21.15.110:0/0) 2023-10-06T03:29:37.727 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:37.718+0000 7fface59c700 1 --2- 172.21.15.110:0/2455574160 >> v2:172.21.15.192:3300/0 conn(0x7ffad0156810 0x7ffad00ced70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:37.727 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:37.718+0000 7ffacdd9b700 1 --2- 172.21.15.110:0/2455574160 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ffad0156be0 0x7ffad00b5820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:37.727 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:37.718+0000 7ffacd59a700 1 -- 172.21.15.110:0/2455574160 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 676097762 0 0) 0x7ffac8001490 con 0x7ffad015caf0 2023-10-06T03:29:37.727 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:37.718+0000 7ffacd59a700 1 -- 172.21.15.110:0/2455574160 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffabc003570 con 0x7ffad015caf0 2023-10-06T03:29:37.727 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:37.718+0000 7ffacd59a700 1 -- 172.21.15.110:0/2455574160 <== mon.0 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3033640211 0 0) 0x7ffabc003570 con 0x7ffad015caf0 2023-10-06T03:29:37.727 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:37.718+0000 7ffacdd9b700 1 -- 172.21.15.110:0/2455574160 >> v2:172.21.15.192:3300/0 conn(0x7ffad0156810 msgr2=0x7ffad00ced70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:29:37.728 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:37.718+0000 7ffacdd9b700 1 --2- 172.21.15.110:0/2455574160 >> v2:172.21.15.192:3300/0 conn(0x7ffad0156810 0x7ffad00ced70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:37.728 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:37.718+0000 7ffacdd9b700 1 -- 172.21.15.110:0/2455574160 >> v1:172.21.15.110:6789/0 conn(0x7ffad015caf0 legacy=0x7ffad015cf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:37.728 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:37.718+0000 7ffacdd9b700 1 -- 172.21.15.110:0/2455574160 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffad00cf960 con 0x7ffad0156be0 2023-10-06T03:29:37.728 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:37.718+0000 7fface59c700 1 --2- 172.21.15.110:0/2455574160 >> v2:172.21.15.192:3300/0 conn(0x7ffad0156810 0x7ffad00ced70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-06T03:29:37.728 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:37.718+0000 7ffacdd9b700 1 --2- 172.21.15.110:0/2455574160 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ffad0156be0 0x7ffad00b5820 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7ffac00098e0 tx=0x7ffac00091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=884e835f13ededd2 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:29:37.728 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:37.718+0000 7ffacd59a700 1 -- 172.21.15.110:0/2455574160 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffac8001980 con 0x7ffad0156be0 2023-10-06T03:29:37.728 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:37.718+0000 7ffacd59a700 1 -- 172.21.15.110:0/2455574160 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffac0011420 con 0x7ffad0156be0 2023-10-06T03:29:37.729 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:37.722+0000 7ffacf59e700 1 -- 172.21.15.110:0/2455574160 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ffad0156be0 msgr2=0x7ffad00b5820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:37.729 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:37.722+0000 7ffacf59e700 1 --2- 172.21.15.110:0/2455574160 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ffad0156be0 0x7ffad00b5820 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7ffac00098e0 tx=0x7ffac00091a0 comp rx=0 tx=0).stop 2023-10-06T03:29:37.729 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:37.722+0000 7ffacf59e700 1 -- 172.21.15.110:0/2455574160 shutdown_connections 2023-10-06T03:29:37.729 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:37.722+0000 7ffacf59e700 1 --2- 172.21.15.110:0/2455574160 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ffad0156be0 0x7ffad00b5820 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:37.729 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:37.722+0000 7ffacf59e700 1 --2- 172.21.15.110:0/2455574160 >> v2:172.21.15.192:3300/0 conn(0x7ffad0156810 0x7ffad00ced70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:37.729 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:37.722+0000 7ffacf59e700 1 -- 172.21.15.110:0/2455574160 >> 172.21.15.110:0/2455574160 conn(0x7ffad00cfee0 msgr2=0x7ffad00d02e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:29:37.729 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:37.722+0000 7ffacf59e700 1 -- 172.21.15.110:0/2455574160 shutdown_connections 2023-10-06T03:29:37.730 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:37.722+0000 7ffacf59e700 1 -- 172.21.15.110:0/2455574160 wait complete. 2023-10-06T03:29:37.730 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:37.722+0000 7ffacf59e700 1 Processor -- start 2023-10-06T03:29:37.730 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:37.722+0000 7ffacf59e700 1 -- start start 2023-10-06T03:29:37.733 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:37.726+0000 7ffacf59e700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ffad0156810 0x7ffad0144c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:37.733 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:37.726+0000 7ffacf59e700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ffad0156be0 0x7ffad00b55b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:37.733 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:37.726+0000 7ffacf59e700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ffad015caf0 0x7ffad015b780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:37.733 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:37.726+0000 7ffacf59e700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffad01451a0 con 0x7ffad0156810 2023-10-06T03:29:37.733 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:37.726+0000 7ffacf59e700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffad00b5af0 con 0x7ffad015caf0 2023-10-06T03:29:37.733 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:37.726+0000 7ffacf59e700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffad015a6e0 con 0x7ffad0156be0 2023-10-06T03:29:37.733 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:37.726+0000 7fface59c700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ffad0156810 0x7ffad0144c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:37.733 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:37.726+0000 7fface59c700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ffad0156810 0x7ffad0144c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:36652/0 (socket says 172.21.15.110:36652) 2023-10-06T03:29:37.734 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:37.726+0000 7fface59c700 1 -- 172.21.15.110:0/3784131549 learned_addr learned my addr 172.21.15.110:0/3784131549 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:29:37.734 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:37.726+0000 7ffaced9d700 1 --2- 172.21.15.110:0/3784131549 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ffad015caf0 0x7ffad015b780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:37.734 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:37.726+0000 7ffacdd9b700 1 --2- 172.21.15.110:0/3784131549 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ffad0156be0 0x7ffad00b55b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:37.734 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:37.726+0000 7fface59c700 1 -- 172.21.15.110:0/3784131549 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ffad0156be0 msgr2=0x7ffad00b55b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:37.734 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:37.726+0000 7fface59c700 1 --2- 172.21.15.110:0/3784131549 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ffad0156be0 0x7ffad00b55b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:37.734 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:37.726+0000 7fface59c700 1 -- 172.21.15.110:0/3784131549 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ffad015caf0 msgr2=0x7ffad015b780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:37.734 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:37.726+0000 7fface59c700 1 --2- 172.21.15.110:0/3784131549 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ffad015caf0 0x7ffad015b780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:37.734 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:37.726+0000 7fface59c700 1 -- 172.21.15.110:0/3784131549 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffab8009420 con 0x7ffad0156810 2023-10-06T03:29:37.735 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:37.726+0000 7fface59c700 1 --2- 172.21.15.110:0/3784131549 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ffad0156810 0x7ffad0144c60 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7ffab800ec80 tx=0x7ffab80091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:29:37.735 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:37.726+0000 7ffac67fc700 1 -- 172.21.15.110:0/3784131549 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffab800c920 con 0x7ffad0156810 2023-10-06T03:29:37.735 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:37.726+0000 7ffac67fc700 1 -- 172.21.15.110:0/3784131549 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffab800ca80 con 0x7ffad0156810 2023-10-06T03:29:37.735 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:37.726+0000 7ffac67fc700 1 -- 172.21.15.110:0/3784131549 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffab800ce00 con 0x7ffad0156810 2023-10-06T03:29:37.735 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:37.726+0000 7ffacf59e700 1 -- 172.21.15.110:0/3784131549 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffac0009420 con 0x7ffad0156810 2023-10-06T03:29:37.736 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:37.726+0000 7ffacf59e700 1 -- 172.21.15.110:0/3784131549 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffad015aca0 con 0x7ffad0156810 2023-10-06T03:29:37.736 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:37.726+0000 7ffacf59e700 1 -- 172.21.15.110:0/3784131549 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffad0070ff0 con 0x7ffad0156810 2023-10-06T03:29:37.736 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:37.730+0000 7ffac67fc700 1 -- 172.21.15.110:0/3784131549 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7ffab8013680 con 0x7ffad0156810 2023-10-06T03:29:37.736 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:37.730+0000 7ffac67fc700 1 --2- 172.21.15.110:0/3784131549 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7ffab0060550 0x7ffab0062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:37.736 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:37.730+0000 7ffac67fc700 1 -- 172.21.15.110:0/3784131549 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6777+0+0 (secure 0 0 0) 0x7ffab807d2a0 con 0x7ffad0156810 2023-10-06T03:29:37.737 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:37.730+0000 7ffacdd9b700 1 --2- 172.21.15.110:0/3784131549 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7ffab0060550 0x7ffab0062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:37.737 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:37.730+0000 7ffacdd9b700 1 --2- 172.21.15.110:0/3784131549 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7ffab0060550 0x7ffab0062a00 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7ffac00091e0 tx=0x7ffac0016000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:29:37.745 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:37.738+0000 7ffac67fc700 1 -- 172.21.15.110:0/3784131549 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7ffab804d460 con 0x7ffad0156810 2023-10-06T03:29:37.778 DEBUG:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-06T03:29:37.780 DEBUG:teuthology.orchestra.run.smithi192:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-06T03:29:37.905 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:37.898+0000 7ffacf59e700 1 -- 172.21.15.110:0/3784131549 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"} v 0) v1 -- 0x7ffad006b860 con 0x7ffad0156810 2023-10-06T03:29:38.596 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:38.590+0000 7ffac67fc700 1 -- 172.21.15.110:0/3784131549 <== mon.0 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]=0 set pool 5 allow_ec_overwrites to true v27) v1 ==== 169+0+0 (secure 0 0 0) 0x7ffab8050a80 con 0x7ffad0156810 2023-10-06T03:29:38.606 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:38.598+0000 7ffacf59e700 1 -- 172.21.15.110:0/3784131549 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7ffab0060550 msgr2=0x7ffab0062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:38.606 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:38.598+0000 7ffacf59e700 1 --2- 172.21.15.110:0/3784131549 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7ffab0060550 0x7ffab0062a00 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7ffac00091e0 tx=0x7ffac0016000 comp rx=0 tx=0).stop 2023-10-06T03:29:38.607 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:38.598+0000 7ffacf59e700 1 -- 172.21.15.110:0/3784131549 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ffad0156810 msgr2=0x7ffad0144c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:38.607 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:38.598+0000 7ffacf59e700 1 --2- 172.21.15.110:0/3784131549 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ffad0156810 0x7ffad0144c60 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7ffab800ec80 tx=0x7ffab80091a0 comp rx=0 tx=0).stop 2023-10-06T03:29:38.607 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:38.598+0000 7ffacf59e700 1 -- 172.21.15.110:0/3784131549 shutdown_connections 2023-10-06T03:29:38.607 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:38.598+0000 7ffacf59e700 1 --2- 172.21.15.110:0/3784131549 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7ffab0060550 0x7ffab0062a00 secure :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7ffac00091e0 tx=0x7ffac0016000 comp rx=0 tx=0).stop 2023-10-06T03:29:38.607 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:38.598+0000 7ffacf59e700 1 --2- 172.21.15.110:0/3784131549 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ffad015caf0 0x7ffad015b780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:38.607 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:38.598+0000 7ffacf59e700 1 --2- 172.21.15.110:0/3784131549 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ffad0156be0 0x7ffad00b55b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:38.607 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:38.598+0000 7ffacf59e700 1 --2- 172.21.15.110:0/3784131549 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ffad0156810 0x7ffad0144c60 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:38.607 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:38.598+0000 7ffacf59e700 1 -- 172.21.15.110:0/3784131549 >> 172.21.15.110:0/3784131549 conn(0x7ffad00cfee0 msgr2=0x7ffad01509c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:29:38.609 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:38.602+0000 7ffacf59e700 1 -- 172.21.15.110:0/3784131549 shutdown_connections 2023-10-06T03:29:38.610 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:38.602+0000 7ffacf59e700 1 -- 172.21.15.110:0/3784131549 wait complete. 2023-10-06T03:29:38.617 INFO:teuthology.orchestra.run.smithi110.stderr:set pool 5 allow_ec_overwrites to true 2023-10-06T03:29:38.634 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs add_data_pool cephfs cephfs_data_ec 2023-10-06T03:29:38.732 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:38.726+0000 7f5771aa1700 1 Processor -- start 2023-10-06T03:29:38.732 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:38.726+0000 7f5771aa1700 1 -- start start 2023-10-06T03:29:38.734 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:38.726+0000 7f5771aa1700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f576c14ff40 0x7f576c156c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:38.734 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:38.726+0000 7f5771aa1700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f576c06daf0 0x7f576c0aea70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:38.734 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:38.726+0000 7f5771aa1700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f576c0ae5e0 con 0x7f576c150310 2023-10-06T03:29:38.734 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:38.726+0000 7f5771aa1700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f576c1571e0 con 0x7f576c06daf0 2023-10-06T03:29:38.734 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:38.726+0000 7f5771aa1700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f576c0ada10 con 0x7f576c14ff40 2023-10-06T03:29:38.734 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:38.726+0000 7f576bfff700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f576c06daf0 0x7f576c0aea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:38.735 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:38.726+0000 7f57712a0700 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f576c150310 0x7f576c06c8d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:55524/0 (socket says 172.21.15.110:55524) 2023-10-06T03:29:38.735 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:38.726+0000 7f576bfff700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f576c06daf0 0x7f576c0aea70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:36656/0 (socket says 172.21.15.110:36656) 2023-10-06T03:29:38.735 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:38.726+0000 7f57712a0700 1 -- 172.21.15.110:0/1698770495 learned_addr learned my addr 172.21.15.110:0/1698770495 (peer_addr_for_me v1:172.21.15.110:0/0) 2023-10-06T03:29:38.735 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:38.726+0000 7f5770a9f700 1 --2- 172.21.15.110:0/1698770495 >> v2:172.21.15.192:3300/0 conn(0x7f576c14ff40 0x7f576c156c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:38.735 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:38.726+0000 7f576bfff700 1 -- 172.21.15.110:0/1698770495 >> v2:172.21.15.192:3300/0 conn(0x7f576c14ff40 msgr2=0x7f576c156c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:29:38.735 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:38.726+0000 7f576bfff700 1 --2- 172.21.15.110:0/1698770495 >> v2:172.21.15.192:3300/0 conn(0x7f576c14ff40 0x7f576c156c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:38.735 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:38.726+0000 7f576bfff700 1 -- 172.21.15.110:0/1698770495 >> v1:172.21.15.110:6789/0 conn(0x7f576c150310 legacy=0x7f576c06c8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:38.735 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:38.726+0000 7f576b7fe700 1 -- 172.21.15.110:0/1698770495 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3958852340 0 0) 0x7f576c0ae5e0 con 0x7f576c150310 2023-10-06T03:29:38.735 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:38.726+0000 7f576bfff700 1 -- 172.21.15.110:0/1698770495 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f576c0adcb0 con 0x7f576c06daf0 2023-10-06T03:29:38.735 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:38.726+0000 7f576bfff700 1 --2- 172.21.15.110:0/1698770495 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f576c06daf0 0x7f576c0aea70 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f576c06b560 tx=0x7f57540091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=287b11882d8c5622 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:29:38.736 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:38.726+0000 7f576b7fe700 1 -- 172.21.15.110:0/1698770495 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f575400b6b0 con 0x7f576c06daf0 2023-10-06T03:29:38.736 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:38.726+0000 7f576b7fe700 1 -- 172.21.15.110:0/1698770495 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f575400b810 con 0x7f576c06daf0 2023-10-06T03:29:38.737 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:38.726+0000 7f576b7fe700 1 -- 172.21.15.110:0/1698770495 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f575400bbb0 con 0x7f576c06daf0 2023-10-06T03:29:38.737 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:38.730+0000 7f5771aa1700 1 -- 172.21.15.110:0/1698770495 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f576c06daf0 msgr2=0x7f576c0aea70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:38.737 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:38.730+0000 7f5771aa1700 1 --2- 172.21.15.110:0/1698770495 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f576c06daf0 0x7f576c0aea70 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f576c06b560 tx=0x7f57540091a0 comp rx=0 tx=0).stop 2023-10-06T03:29:38.737 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:38.730+0000 7f5771aa1700 1 -- 172.21.15.110:0/1698770495 shutdown_connections 2023-10-06T03:29:38.737 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:38.730+0000 7f5771aa1700 1 --2- 172.21.15.110:0/1698770495 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f576c06daf0 0x7f576c0aea70 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:38.737 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:38.730+0000 7f5771aa1700 1 --2- 172.21.15.110:0/1698770495 >> v2:172.21.15.192:3300/0 conn(0x7f576c14ff40 0x7f576c156c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:38.737 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:38.730+0000 7f5771aa1700 1 -- 172.21.15.110:0/1698770495 >> 172.21.15.110:0/1698770495 conn(0x7f576c06b140 msgr2=0x7f576c0670a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:29:38.737 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:38.730+0000 7f5771aa1700 1 -- 172.21.15.110:0/1698770495 shutdown_connections 2023-10-06T03:29:38.737 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:38.730+0000 7f5771aa1700 1 -- 172.21.15.110:0/1698770495 wait complete. 2023-10-06T03:29:38.738 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:38.730+0000 7f5771aa1700 1 Processor -- start 2023-10-06T03:29:38.738 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:38.730+0000 7f5771aa1700 1 -- start start 2023-10-06T03:29:38.741 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:38.730+0000 7f5771aa1700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f576c06daf0 0x7f576c142100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:38.741 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:38.734+0000 7f5771aa1700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f576c14ff40 0x7f576c0ae910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:38.742 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:38.734+0000 7f5771aa1700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f576c150310 0x7f576c0aee50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:38.742 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:38.734+0000 7f5771aa1700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f576c0c2480 con 0x7f576c150310 2023-10-06T03:29:38.742 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:38.734+0000 7f5771aa1700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f576c0c2d20 con 0x7f576c06daf0 2023-10-06T03:29:38.742 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:38.734+0000 7f5771aa1700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f576c151310 con 0x7f576c14ff40 2023-10-06T03:29:38.742 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:38.734+0000 7f57712a0700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f576c150310 0x7f576c0aee50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:38.742 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:38.734+0000 7f57712a0700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f576c150310 0x7f576c0aee50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:36668/0 (socket says 172.21.15.110:36668) 2023-10-06T03:29:38.742 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:38.734+0000 7f57712a0700 1 -- 172.21.15.110:0/1452255466 learned_addr learned my addr 172.21.15.110:0/1452255466 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:29:38.742 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:38.734+0000 7f5770a9f700 1 --2- 172.21.15.110:0/1452255466 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f576c06daf0 0x7f576c142100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:38.743 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:38.734+0000 7f576bfff700 1 --2- 172.21.15.110:0/1452255466 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f576c14ff40 0x7f576c0ae910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:38.743 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:38.734+0000 7f57712a0700 1 -- 172.21.15.110:0/1452255466 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f576c14ff40 msgr2=0x7f576c0ae910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:38.743 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:38.734+0000 7f57712a0700 1 --2- 172.21.15.110:0/1452255466 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f576c14ff40 0x7f576c0ae910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:38.743 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:38.734+0000 7f57712a0700 1 -- 172.21.15.110:0/1452255466 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f576c06daf0 msgr2=0x7f576c142100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:38.743 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:38.734+0000 7f57712a0700 1 --2- 172.21.15.110:0/1452255466 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f576c06daf0 0x7f576c142100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:38.743 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:38.734+0000 7f57712a0700 1 -- 172.21.15.110:0/1452255466 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5754009420 con 0x7f576c150310 2023-10-06T03:29:38.743 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:38.734+0000 7f57712a0700 1 --2- 172.21.15.110:0/1452255466 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f576c150310 0x7f576c0aee50 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f576400cc20 tx=0x7f57640019b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:29:38.743 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:38.734+0000 7f5768ff9700 1 -- 172.21.15.110:0/1452255466 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f576400a870 con 0x7f576c150310 2023-10-06T03:29:38.743 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:38.734+0000 7f5768ff9700 1 -- 172.21.15.110:0/1452255466 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f576400a9f0 con 0x7f576c150310 2023-10-06T03:29:38.743 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:38.734+0000 7f5771aa1700 1 -- 172.21.15.110:0/1452255466 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f576c0c26d0 con 0x7f576c150310 2023-10-06T03:29:38.744 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:38.734+0000 7f5768ff9700 1 -- 172.21.15.110:0/1452255466 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f576400adb0 con 0x7f576c150310 2023-10-06T03:29:38.744 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:38.734+0000 7f5771aa1700 1 -- 172.21.15.110:0/1452255466 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f576c0afc70 con 0x7f576c150310 2023-10-06T03:29:38.744 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:38.734+0000 7f5771aa1700 1 -- 172.21.15.110:0/1452255466 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f576c070ff0 con 0x7f576c150310 2023-10-06T03:29:38.745 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:38.734+0000 7f5768ff9700 1 -- 172.21.15.110:0/1452255466 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7f5764007300 con 0x7f576c150310 2023-10-06T03:29:38.745 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:38.738+0000 7f5768ff9700 1 --2- 172.21.15.110:0/1452255466 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f574c060550 0x7f574c062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:38.745 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:38.738+0000 7f5768ff9700 1 -- 172.21.15.110:0/1452255466 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6777+0+0 (secure 0 0 0) 0x7f576407cde0 con 0x7f576c150310 2023-10-06T03:29:38.745 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:38.738+0000 7f5770a9f700 1 --2- 172.21.15.110:0/1452255466 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f574c060550 0x7f574c062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:38.745 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:38.738+0000 7f5770a9f700 1 --2- 172.21.15.110:0/1452255466 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f574c060550 0x7f574c062a00 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f575c006fd0 tx=0x7f575c008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:29:38.750 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:38.742+0000 7f5768ff9700 1 -- 172.21.15.110:0/1452255466 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f576404d020 con 0x7f576c150310 2023-10-06T03:29:38.952 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:38.946+0000 7f5771aa1700 1 -- 172.21.15.110:0/1452255466 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"} v 0) v1 -- 0x7f576c0adf90 con 0x7f576c150310 2023-10-06T03:29:39.624 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:39.618+0000 7f5768ff9700 1 -- 172.21.15.110:0/1452255466 <== mon.0 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]=0 added data pool 5 to fsmap v6) v1 ==== 137+0+0 (secure 0 0 0) 0x7f5764050640 con 0x7f576c150310 2023-10-06T03:29:39.628 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:39.622+0000 7f5771aa1700 1 -- 172.21.15.110:0/1452255466 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f574c060550 msgr2=0x7f574c062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:39.629 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:39.622+0000 7f5771aa1700 1 --2- 172.21.15.110:0/1452255466 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f574c060550 0x7f574c062a00 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f575c006fd0 tx=0x7f575c008040 comp rx=0 tx=0).stop 2023-10-06T03:29:39.629 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:39.622+0000 7f5771aa1700 1 -- 172.21.15.110:0/1452255466 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f576c150310 msgr2=0x7f576c0aee50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:39.629 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:39.622+0000 7f5771aa1700 1 --2- 172.21.15.110:0/1452255466 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f576c150310 0x7f576c0aee50 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f576400cc20 tx=0x7f57640019b0 comp rx=0 tx=0).stop 2023-10-06T03:29:39.629 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:39.622+0000 7f5771aa1700 1 -- 172.21.15.110:0/1452255466 shutdown_connections 2023-10-06T03:29:39.629 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:39.622+0000 7f5771aa1700 1 --2- 172.21.15.110:0/1452255466 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f574c060550 0x7f574c062a00 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:39.629 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:39.622+0000 7f5771aa1700 1 --2- 172.21.15.110:0/1452255466 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f576c150310 0x7f576c0aee50 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:39.629 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:39.622+0000 7f5771aa1700 1 --2- 172.21.15.110:0/1452255466 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f576c14ff40 0x7f576c0ae910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:39.629 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:39.622+0000 7f5771aa1700 1 --2- 172.21.15.110:0/1452255466 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f576c06daf0 0x7f576c142100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:39.629 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:39.622+0000 7f5771aa1700 1 -- 172.21.15.110:0/1452255466 >> 172.21.15.110:0/1452255466 conn(0x7f576c06b140 msgr2=0x7f576c144e30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:29:39.630 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:39.622+0000 7f5771aa1700 1 -- 172.21.15.110:0/1452255466 shutdown_connections 2023-10-06T03:29:39.630 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:39.622+0000 7f5771aa1700 1 -- 172.21.15.110:0/1452255466 wait complete. 2023-10-06T03:29:39.630 INFO:teuthology.orchestra.run.smithi110.stderr:added data pool 5 to fsmap 2023-10-06T03:29:39.641 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-06T03:29:39.739 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:39.730+0000 7f390df8a700 1 Processor -- start 2023-10-06T03:29:39.739 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:39.730+0000 7f390df8a700 1 -- start start 2023-10-06T03:29:39.743 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:39.734+0000 7f390df8a700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f39081559d0 0x7f390806cc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:39.743 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:39.734+0000 7f390df8a700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f3908159e80 0x7f39080c5f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:39.743 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:39.734+0000 7f390df8a700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39080caad0 con 0x7f3908155da0 2023-10-06T03:29:39.743 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:39.734+0000 7f390df8a700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f39080ae3f0 con 0x7f3908159e80 2023-10-06T03:29:39.743 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:39.734+0000 7f390df8a700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39080b1060 con 0x7f39081559d0 2023-10-06T03:29:39.743 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:39.734+0000 7f390d789700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f3908159e80 0x7f39080c5f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:39.744 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:39.734+0000 7f390d789700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f3908159e80 0x7f39080c5f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:36674/0 (socket says 172.21.15.110:36674) 2023-10-06T03:29:39.744 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:39.734+0000 7f390d789700 1 -- 172.21.15.110:0/3784990056 learned_addr learned my addr 172.21.15.110:0/3784990056 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:29:39.744 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:39.734+0000 7f390cf88700 1 --2- 172.21.15.110:0/3784990056 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f39081559d0 0x7f390806cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:39.744 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:39.734+0000 7f390d789700 1 -- 172.21.15.110:0/3784990056 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f39081559d0 msgr2=0x7f390806cc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:29:39.745 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:39.734+0000 7f390d789700 1 --2- 172.21.15.110:0/3784990056 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f39081559d0 0x7f390806cc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:39.745 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:39.734+0000 7f390d789700 1 -- 172.21.15.110:0/3784990056 >> v1:172.21.15.192:6789/0 conn(0x7f3908155da0 legacy=0x7f3908159770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:39.745 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:39.734+0000 7f390d789700 1 -- 172.21.15.110:0/3784990056 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39080ca1a0 con 0x7f3908159e80 2023-10-06T03:29:39.745 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:39.734+0000 7f390d789700 1 --2- 172.21.15.110:0/3784990056 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3908159e80 0x7f39080c5f40 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f3900009f20 tx=0x7f3900009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=40fa075cba0bbf7c server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:29:39.745 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:39.734+0000 7f39077fe700 1 -- 172.21.15.110:0/3784990056 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f390000e070 con 0x7f3908159e80 2023-10-06T03:29:39.745 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:39.734+0000 7f39077fe700 1 -- 172.21.15.110:0/3784990056 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f390000ba80 con 0x7f3908159e80 2023-10-06T03:29:39.746 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:39.734+0000 7f39077fe700 1 -- 172.21.15.110:0/3784990056 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39000092e0 con 0x7f3908159e80 2023-10-06T03:29:39.746 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:39.734+0000 7f390df8a700 1 -- 172.21.15.110:0/3784990056 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3908159e80 msgr2=0x7f39080c5f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:39.746 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:39.734+0000 7f390df8a700 1 --2- 172.21.15.110:0/3784990056 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3908159e80 0x7f39080c5f40 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f3900009f20 tx=0x7f3900009840 comp rx=0 tx=0).stop 2023-10-06T03:29:39.746 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:39.734+0000 7f390df8a700 1 -- 172.21.15.110:0/3784990056 shutdown_connections 2023-10-06T03:29:39.746 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:39.734+0000 7f390df8a700 1 --2- 172.21.15.110:0/3784990056 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3908159e80 0x7f39080c5f40 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:39.746 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:39.734+0000 7f390df8a700 1 --2- 172.21.15.110:0/3784990056 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f39081559d0 0x7f390806cc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:39.746 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:39.734+0000 7f390df8a700 1 -- 172.21.15.110:0/3784990056 >> 172.21.15.110:0/3784990056 conn(0x7f39080cc470 msgr2=0x7f39080c8410 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:29:39.746 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:39.734+0000 7f390df8a700 1 -- 172.21.15.110:0/3784990056 shutdown_connections 2023-10-06T03:29:39.746 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:39.734+0000 7f390df8a700 1 -- 172.21.15.110:0/3784990056 wait complete. 2023-10-06T03:29:39.749 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:39.738+0000 7f390df8a700 1 Processor -- start 2023-10-06T03:29:39.749 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:39.738+0000 7f390df8a700 1 -- start start 2023-10-06T03:29:39.749 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:39.738+0000 7f390df8a700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f39081559d0 0x7f390814cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:39.749 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:39.738+0000 7f390df8a700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3908155da0 0x7f3908157ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:39.749 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:39.738+0000 7f390df8a700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3908159e80 0x7f39081694e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:39.750 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:39.738+0000 7f390df8a700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39080c3830 con 0x7f39081559d0 2023-10-06T03:29:39.750 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:39.738+0000 7f390df8a700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39080c3b10 con 0x7f3908155da0 2023-10-06T03:29:39.750 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:39.738+0000 7f390df8a700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39080c3df0 con 0x7f3908159e80 2023-10-06T03:29:39.750 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:39.738+0000 7f390cf88700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f39081559d0 0x7f390814cfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:39.751 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:39.738+0000 7f390cf88700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f39081559d0 0x7f390814cfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:36690/0 (socket says 172.21.15.110:36690) 2023-10-06T03:29:39.751 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:39.738+0000 7f390cf88700 1 -- 172.21.15.110:0/1554851783 learned_addr learned my addr 172.21.15.110:0/1554851783 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:29:39.751 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:39.738+0000 7f390d789700 1 --2- 172.21.15.110:0/1554851783 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3908159e80 0x7f39081694e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:39.751 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:39.738+0000 7f3907fff700 1 --2- 172.21.15.110:0/1554851783 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3908155da0 0x7f3908157ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:39.751 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:39.738+0000 7f390cf88700 1 -- 172.21.15.110:0/1554851783 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3908159e80 msgr2=0x7f39081694e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:39.751 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:39.738+0000 7f390cf88700 1 --2- 172.21.15.110:0/1554851783 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3908159e80 0x7f39081694e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:39.751 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:39.738+0000 7f390cf88700 1 -- 172.21.15.110:0/1554851783 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3908155da0 msgr2=0x7f3908157ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:39.751 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:39.738+0000 7f390cf88700 1 --2- 172.21.15.110:0/1554851783 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3908155da0 0x7f3908157ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:39.751 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:39.738+0000 7f390cf88700 1 -- 172.21.15.110:0/1554851783 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3900009ae0 con 0x7f39081559d0 2023-10-06T03:29:39.751 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:39.738+0000 7f390d789700 1 --2- 172.21.15.110:0/1554851783 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3908159e80 0x7f39081694e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-06T03:29:39.751 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:39.738+0000 7f3907fff700 1 --2- 172.21.15.110:0/1554851783 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3908155da0 0x7f3908157ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-06T03:29:39.752 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:39.738+0000 7f390cf88700 1 --2- 172.21.15.110:0/1554851783 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f39081559d0 0x7f390814cfd0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f38f800d890 tx=0x7f38f8008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:29:39.752 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:39.738+0000 7f3904ff9700 1 -- 172.21.15.110:0/1554851783 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38f800b600 con 0x7f39081559d0 2023-10-06T03:29:39.752 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:39.738+0000 7f3904ff9700 1 -- 172.21.15.110:0/1554851783 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38f800b760 con 0x7f39081559d0 2023-10-06T03:29:39.752 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:39.738+0000 7f3904ff9700 1 -- 172.21.15.110:0/1554851783 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38f800bae0 con 0x7f39081559d0 2023-10-06T03:29:39.752 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:39.738+0000 7f390df8a700 1 -- 172.21.15.110:0/1554851783 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39080c40d0 con 0x7f39081559d0 2023-10-06T03:29:39.752 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:39.738+0000 7f390df8a700 1 -- 172.21.15.110:0/1554851783 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f390814d510 con 0x7f39081559d0 2023-10-06T03:29:39.752 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:39.742+0000 7f390df8a700 1 -- 172.21.15.110:0/1554851783 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3908070ff0 con 0x7f39081559d0 2023-10-06T03:29:39.753 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:39.742+0000 7f3904ff9700 1 -- 172.21.15.110:0/1554851783 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7f38f801e460 con 0x7f39081559d0 2023-10-06T03:29:39.753 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:39.742+0000 7f3904ff9700 1 --2- 172.21.15.110:0/1554851783 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f38f4060550 0x7f38f4062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:39.753 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:39.742+0000 7f3904ff9700 1 -- 172.21.15.110:0/1554851783 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6809+0+0 (secure 0 0 0) 0x7f38f807d3d0 con 0x7f39081559d0 2023-10-06T03:29:39.753 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:39.746+0000 7f3907fff700 1 --2- 172.21.15.110:0/1554851783 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f38f4060550 0x7f38f4062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:39.753 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:39.746+0000 7f3907fff700 1 --2- 172.21.15.110:0/1554851783 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f38f4060550 0x7f38f4062a00 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f38fc007fd0 tx=0x7f38fc009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:29:39.757 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:39.750+0000 7f3904ff9700 1 -- 172.21.15.110:0/1554851783 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f38f804d4d0 con 0x7f39081559d0 2023-10-06T03:29:39.950 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:39.942+0000 7f390df8a700 1 -- 172.21.15.110:0/1554851783 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f39080ca480 con 0x7f39081559d0 2023-10-06T03:29:39.951 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:39.946+0000 7f3904ff9700 1 -- 172.21.15.110:0/1554851783 <== mon.0 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) v1 ==== 93+0+4602 (secure 0 0 0) 0x7f38f8050af0 con 0x7f39081559d0 2023-10-06T03:29:39.952 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-10-06T03:29:39.952 INFO:teuthology.orchestra.run.smithi110.stdout:{"epoch":6,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4279,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6833/1434505700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6832","nonce":1434505700},{"type":"v1","addr":"172.21.15.192:6833","nonce":1434505700}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4281,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/1168370789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6834","nonce":1168370789},{"type":"v1","addr":"172.21.15.110:6835","nonce":1168370789}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4288,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6835/1271810086","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":1271810086},{"type":"v1","addr":"172.21.15.192:6835","nonce":1271810086}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":6,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-06T03:29:33.894998+0000","modified":"2023-10-06T03:29:39.611370+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4293},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4293":{"gid":4293,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.110:6837/540320464","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6836","nonce":540320464},{"type":"v1","addr":"172.21.15.110:6837","nonce":540320464}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2023-10-06T03:29:39.954 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:39.946+0000 7f390df8a700 1 -- 172.21.15.110:0/1554851783 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f38f4060550 msgr2=0x7f38f4062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:39.954 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:39.946+0000 7f390df8a700 1 --2- 172.21.15.110:0/1554851783 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f38f4060550 0x7f38f4062a00 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f38fc007fd0 tx=0x7f38fc009040 comp rx=0 tx=0).stop 2023-10-06T03:29:39.954 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:39.946+0000 7f390df8a700 1 -- 172.21.15.110:0/1554851783 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f39081559d0 msgr2=0x7f390814cfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:39.954 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:39.946+0000 7f390df8a700 1 --2- 172.21.15.110:0/1554851783 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f39081559d0 0x7f390814cfd0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f38f800d890 tx=0x7f38f8008040 comp rx=0 tx=0).stop 2023-10-06T03:29:39.954 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:39.946+0000 7f390df8a700 1 -- 172.21.15.110:0/1554851783 shutdown_connections 2023-10-06T03:29:39.954 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:39.946+0000 7f390df8a700 1 --2- 172.21.15.110:0/1554851783 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f38f4060550 0x7f38f4062a00 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:39.954 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:39.946+0000 7f390df8a700 1 --2- 172.21.15.110:0/1554851783 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3908159e80 0x7f39081694e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:39.954 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:39.946+0000 7f390df8a700 1 --2- 172.21.15.110:0/1554851783 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3908155da0 0x7f3908157ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:39.955 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:39.946+0000 7f390df8a700 1 --2- 172.21.15.110:0/1554851783 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f39081559d0 0x7f390814cfd0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:39.955 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:39.946+0000 7f390df8a700 1 -- 172.21.15.110:0/1554851783 >> 172.21.15.110:0/1554851783 conn(0x7f39080cc470 msgr2=0x7f39080c5e60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:29:39.955 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:39.946+0000 7f390df8a700 1 -- 172.21.15.110:0/1554851783 shutdown_connections 2023-10-06T03:29:39.955 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:39.946+0000 7f390df8a700 1 -- 172.21.15.110:0/1554851783 wait complete. 2023-10-06T03:29:39.955 INFO:teuthology.orchestra.run.smithi110.stderr:dumped fsmap epoch 6 2023-10-06T03:29:39.965 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-06T03:29:40.071 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:40.062+0000 7fd6a682d700 1 Processor -- start 2023-10-06T03:29:40.071 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:40.062+0000 7fd6a682d700 1 -- start start 2023-10-06T03:29:40.072 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:40.066+0000 7fd6a682d700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd6a0159000 0x7fd6a006c180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:40.072 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:40.066+0000 7fd6a682d700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6a00c7710 con 0x7fd6a0158c30 2023-10-06T03:29:40.072 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:40.066+0000 7fd6a682d700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6a006cf80 con 0x7fd6a00c7ba0 2023-10-06T03:29:40.072 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:40.066+0000 7fd6a682d700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6a006cb20 con 0x7fd6a0159000 2023-10-06T03:29:40.073 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:40.066+0000 7fd6a582b700 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7fd6a0158c30 0x7fd6a00670a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:55528/0 (socket says 172.21.15.110:55528) 2023-10-06T03:29:40.073 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:40.066+0000 7fd6a582b700 1 -- 172.21.15.110:0/2695975843 learned_addr learned my addr 172.21.15.110:0/2695975843 (peer_addr_for_me v1:172.21.15.110:0/0) 2023-10-06T03:29:40.073 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:40.066+0000 7fd6a502a700 1 --2- 172.21.15.110:0/2695975843 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd6a0159000 0x7fd6a006c180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:40.082 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:40.066+0000 7fd6a4829700 1 -- 172.21.15.110:0/2695975843 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2149342285 0 0) 0x7fd6a00c7710 con 0x7fd6a0158c30 2023-10-06T03:29:40.082 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:40.066+0000 7fd6a4829700 1 -- 172.21.15.110:0/2695975843 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd68c003570 con 0x7fd6a0158c30 2023-10-06T03:29:40.083 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:40.066+0000 7fd6a502a700 1 -- 172.21.15.110:0/2695975843 >> v1:172.21.15.192:6789/0 conn(0x7fd6a00c7ba0 legacy=0x7fd6a015cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:40.083 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:40.066+0000 7fd6a502a700 1 -- 172.21.15.110:0/2695975843 >> v1:172.21.15.110:6789/0 conn(0x7fd6a0158c30 legacy=0x7fd6a00670a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:40.083 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:40.066+0000 7fd6a4829700 1 -- 172.21.15.110:0/2695975843 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1682863317 0 0) 0x7fd6a006cf80 con 0x7fd6a00c7ba0 2023-10-06T03:29:40.083 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:40.066+0000 7fd6a502a700 1 -- 172.21.15.110:0/2695975843 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6a00c6e20 con 0x7fd6a0159000 2023-10-06T03:29:40.083 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:40.066+0000 7fd6a502a700 1 --2- 172.21.15.110:0/2695975843 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd6a0159000 0x7fd6a006c180 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fd690009910 tx=0x7fd6900091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=324a28f4fc65ecb4 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:29:40.083 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:40.066+0000 7fd6a4829700 1 -- 172.21.15.110:0/2695975843 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd69000b680 con 0x7fd6a0159000 2023-10-06T03:29:40.083 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:40.066+0000 7fd6a4829700 1 -- 172.21.15.110:0/2695975843 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd69000b7e0 con 0x7fd6a0159000 2023-10-06T03:29:40.083 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:40.066+0000 7fd6a4829700 1 -- 172.21.15.110:0/2695975843 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd69000bba0 con 0x7fd6a0159000 2023-10-06T03:29:40.084 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:40.066+0000 7fd6a682d700 1 -- 172.21.15.110:0/2695975843 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd6a0159000 msgr2=0x7fd6a006c180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:40.084 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:40.066+0000 7fd6a682d700 1 --2- 172.21.15.110:0/2695975843 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd6a0159000 0x7fd6a006c180 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fd690009910 tx=0x7fd6900091a0 comp rx=0 tx=0).stop 2023-10-06T03:29:40.084 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:40.066+0000 7fd6a682d700 1 -- 172.21.15.110:0/2695975843 shutdown_connections 2023-10-06T03:29:40.084 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:40.066+0000 7fd6a682d700 1 --2- 172.21.15.110:0/2695975843 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd6a0159000 0x7fd6a006c180 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:40.084 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:40.066+0000 7fd6a682d700 1 -- 172.21.15.110:0/2695975843 >> 172.21.15.110:0/2695975843 conn(0x7fd6a00c8d00 msgr2=0x7fd6a00c9100 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:29:40.084 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:40.066+0000 7fd6a682d700 1 -- 172.21.15.110:0/2695975843 shutdown_connections 2023-10-06T03:29:40.084 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:40.066+0000 7fd6a682d700 1 -- 172.21.15.110:0/2695975843 wait complete. 2023-10-06T03:29:40.084 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:40.070+0000 7fd6a682d700 1 Processor -- start 2023-10-06T03:29:40.084 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:40.070+0000 7fd6a682d700 1 -- start start 2023-10-06T03:29:40.084 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:40.070+0000 7fd6a682d700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd6a00c7ba0 0x7fd6a01696f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:40.085 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:40.070+0000 7fd6a682d700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd6a0158c30 0x7fd6a00aace0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:40.085 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:40.070+0000 7fd6a682d700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd6a0159000 0x7fd6a00ae4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:40.085 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:40.070+0000 7fd6a682d700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6a00ab6d0 con 0x7fd6a0159000 2023-10-06T03:29:40.085 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:40.070+0000 7fd6a682d700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6a00c3a30 con 0x7fd6a0158c30 2023-10-06T03:29:40.085 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:40.070+0000 7fd6a682d700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6a00ae9f0 con 0x7fd6a00c7ba0 2023-10-06T03:29:40.085 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:40.070+0000 7fd6a602c700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd6a0159000 0x7fd6a00ae4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:40.085 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:40.070+0000 7fd6a602c700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd6a0159000 0x7fd6a00ae4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:36694/0 (socket says 172.21.15.110:36694) 2023-10-06T03:29:40.085 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:40.070+0000 7fd6a602c700 1 -- 172.21.15.110:0/2458040802 learned_addr learned my addr 172.21.15.110:0/2458040802 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:29:40.086 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:40.070+0000 7fd6a582b700 1 --2- 172.21.15.110:0/2458040802 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd6a00c7ba0 0x7fd6a01696f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:40.086 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:40.070+0000 7fd6a502a700 1 --2- 172.21.15.110:0/2458040802 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd6a0158c30 0x7fd6a00aace0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:40.086 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:40.070+0000 7fd6a602c700 1 -- 172.21.15.110:0/2458040802 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd6a00c7ba0 msgr2=0x7fd6a01696f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:40.086 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:40.070+0000 7fd6a602c700 1 --2- 172.21.15.110:0/2458040802 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd6a00c7ba0 0x7fd6a01696f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:40.086 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:40.070+0000 7fd6a602c700 1 -- 172.21.15.110:0/2458040802 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd6a0158c30 msgr2=0x7fd6a00aace0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:40.086 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:40.070+0000 7fd6a602c700 1 --2- 172.21.15.110:0/2458040802 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd6a0158c30 0x7fd6a00aace0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:40.086 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:40.070+0000 7fd6a602c700 1 -- 172.21.15.110:0/2458040802 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd690009420 con 0x7fd6a0159000 2023-10-06T03:29:40.086 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:40.070+0000 7fd6a502a700 1 --2- 172.21.15.110:0/2458040802 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd6a0158c30 0x7fd6a00aace0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-06T03:29:40.087 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:40.070+0000 7fd6a582b700 1 --2- 172.21.15.110:0/2458040802 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd6a00c7ba0 0x7fd6a01696f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-06T03:29:40.087 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:40.070+0000 7fd6a602c700 1 --2- 172.21.15.110:0/2458040802 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd6a0159000 0x7fd6a00ae4b0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fd69800cb60 tx=0x7fd698001770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:29:40.087 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:40.070+0000 7fd695ffb700 1 -- 172.21.15.110:0/2458040802 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd69800a770 con 0x7fd6a0159000 2023-10-06T03:29:40.087 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:40.070+0000 7fd695ffb700 1 -- 172.21.15.110:0/2458040802 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd69800a8d0 con 0x7fd6a0159000 2023-10-06T03:29:40.087 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:40.070+0000 7fd695ffb700 1 -- 172.21.15.110:0/2458040802 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd69800ac90 con 0x7fd6a0159000 2023-10-06T03:29:40.087 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:40.070+0000 7fd6a682d700 1 -- 172.21.15.110:0/2458040802 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6a00ab3f0 con 0x7fd6a0159000 2023-10-06T03:29:40.087 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:40.070+0000 7fd6a682d700 1 -- 172.21.15.110:0/2458040802 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6a00c36c0 con 0x7fd6a0159000 2023-10-06T03:29:40.087 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:40.070+0000 7fd6a682d700 1 -- 172.21.15.110:0/2458040802 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd6a0070ff0 con 0x7fd6a0159000 2023-10-06T03:29:40.087 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:40.074+0000 7fd695ffb700 1 -- 172.21.15.110:0/2458040802 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7fd698012630 con 0x7fd6a0159000 2023-10-06T03:29:40.087 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:40.074+0000 7fd695ffb700 1 --2- 172.21.15.110:0/2458040802 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fd684060550 0x7fd684062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:40.088 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:40.074+0000 7fd695ffb700 1 -- 172.21.15.110:0/2458040802 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6809+0+0 (secure 0 0 0) 0x7fd69807d780 con 0x7fd6a0159000 2023-10-06T03:29:40.088 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:40.074+0000 7fd6a582b700 1 --2- 172.21.15.110:0/2458040802 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fd684060550 0x7fd684062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:40.088 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:40.074+0000 7fd6a582b700 1 --2- 172.21.15.110:0/2458040802 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fd684060550 0x7fd684062a00 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fd69c000c00 tx=0x7fd69c009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:29:40.088 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:40.078+0000 7fd695ffb700 1 -- 172.21.15.110:0/2458040802 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fd69804d9a0 con 0x7fd6a0159000 2023-10-06T03:29:40.245 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:40.238+0000 7fd6a682d700 1 -- 172.21.15.110:0/2458040802 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd6a00635e0 con 0x7fd6a0159000 2023-10-06T03:29:40.247 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:40.242+0000 7fd695ffb700 1 -- 172.21.15.110:0/2458040802 <== mon.0 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v28) v1 ==== 74+0+18755 (secure 0 0 0) 0x7fd698050fc0 con 0x7fd6a0159000 2023-10-06T03:29:40.248 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-10-06T03:29:40.248 INFO:teuthology.orchestra.run.smithi110.stdout:{"epoch":28,"fsid":"93b8f6ac-4c92-4b45-b8e4-b59617024aa2","created":"2023-10-06T03:29:10.519583+0000","modified":"2023-10-06T03:29:38.951740+0000","last_up_change":"2023-10-06T03:29:21.960511+0000","last_in_change":"2023-10-06T03:29:12.560436+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-06T03:29:20.703743+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2023-10-06T03:29:28.307281+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2023-10-06T03:29:32.247251+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data","create_time":"2023-10-06T03:29:32.880086+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}}},{"pool":5,"pool_name":"cephfs_data_ec","create_time":"2023-10-06T03:29:36.613779+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"28","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"5866eb6c-2adc-46e4-be7a-e5bcda0f4065","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6824","nonce":33394},{"type":"v1","addr":"172.21.15.110:6825","nonce":33394}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6826","nonce":33394},{"type":"v1","addr":"172.21.15.110:6827","nonce":33394}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6830","nonce":33394},{"type":"v1","addr":"172.21.15.110:6831","nonce":33394}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6828","nonce":33394},{"type":"v1","addr":"172.21.15.110:6829","nonce":33394}]},"public_addr":"172.21.15.110:6825/33394","cluster_addr":"172.21.15.110:6827/33394","heartbeat_back_addr":"172.21.15.110:6831/33394","heartbeat_front_addr":"172.21.15.110:6829/33394","state":["exists","up"]},{"osd":1,"uuid":"a38de118-062b-4a05-9782-2ceddf6ff4df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6800","nonce":33396},{"type":"v1","addr":"172.21.15.110:6801","nonce":33396}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6802","nonce":33396},{"type":"v1","addr":"172.21.15.110:6803","nonce":33396}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6806","nonce":33396},{"type":"v1","addr":"172.21.15.110:6807","nonce":33396}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6804","nonce":33396},{"type":"v1","addr":"172.21.15.110:6805","nonce":33396}]},"public_addr":"172.21.15.110:6801/33396","cluster_addr":"172.21.15.110:6803/33396","heartbeat_back_addr":"172.21.15.110:6807/33396","heartbeat_front_addr":"172.21.15.110:6805/33396","state":["exists","up"]},{"osd":2,"uuid":"56e39be2-3407-4b69-877e-cf16617b06d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6816","nonce":33397},{"type":"v1","addr":"172.21.15.110:6817","nonce":33397}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6818","nonce":33397},{"type":"v1","addr":"172.21.15.110:6819","nonce":33397}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6822","nonce":33397},{"type":"v1","addr":"172.21.15.110:6823","nonce":33397}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6820","nonce":33397},{"type":"v1","addr":"172.21.15.110:6821","nonce":33397}]},"public_addr":"172.21.15.110:6817/33397","cluster_addr":"172.21.15.110:6819/33397","heartbeat_back_addr":"172.21.15.110:6823/33397","heartbeat_front_addr":"172.21.15.110:6821/33397","state":["exists","up"]},{"osd":3,"uuid":"01c82d13-c1d4-4bde-b55e-3e0089b40115","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6808","nonce":33395},{"type":"v1","addr":"172.21.15.110:6809","nonce":33395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6810","nonce":33395},{"type":"v1","addr":"172.21.15.110:6811","nonce":33395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6814","nonce":33395},{"type":"v1","addr":"172.21.15.110:6815","nonce":33395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6812","nonce":33395},{"type":"v1","addr":"172.21.15.110:6813","nonce":33395}]},"public_addr":"172.21.15.110:6809/33395","cluster_addr":"172.21.15.110:6811/33395","heartbeat_back_addr":"172.21.15.110:6815/33395","heartbeat_front_addr":"172.21.15.110:6813/33395","state":["exists","up"]},{"osd":4,"uuid":"86983430-5fa3-4d17-998f-03e2cd843237","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":33683},{"type":"v1","addr":"172.21.15.192:6817","nonce":33683}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":33683},{"type":"v1","addr":"172.21.15.192:6819","nonce":33683}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":33683},{"type":"v1","addr":"172.21.15.192:6823","nonce":33683}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6820","nonce":33683},{"type":"v1","addr":"172.21.15.192:6821","nonce":33683}]},"public_addr":"172.21.15.192:6817/33683","cluster_addr":"172.21.15.192:6819/33683","heartbeat_back_addr":"172.21.15.192:6823/33683","heartbeat_front_addr":"172.21.15.192:6821/33683","state":["exists","up"]},{"osd":5,"uuid":"6945257f-821d-4619-a63f-ebe857eba6f4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":33684},{"type":"v1","addr":"172.21.15.192:6801","nonce":33684}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":33684},{"type":"v1","addr":"172.21.15.192:6803","nonce":33684}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":33684},{"type":"v1","addr":"172.21.15.192:6807","nonce":33684}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6804","nonce":33684},{"type":"v1","addr":"172.21.15.192:6805","nonce":33684}]},"public_addr":"172.21.15.192:6801/33684","cluster_addr":"172.21.15.192:6803/33684","heartbeat_back_addr":"172.21.15.192:6807/33684","heartbeat_front_addr":"172.21.15.192:6805/33684","state":["exists","up"]},{"osd":6,"uuid":"7854d1ae-7060-4780-8f97-73cea5e7a12f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6824","nonce":33686},{"type":"v1","addr":"172.21.15.192:6825","nonce":33686}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":33686},{"type":"v1","addr":"172.21.15.192:6827","nonce":33686}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":33686},{"type":"v1","addr":"172.21.15.192:6831","nonce":33686}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6828","nonce":33686},{"type":"v1","addr":"172.21.15.192:6829","nonce":33686}]},"public_addr":"172.21.15.192:6825/33686","cluster_addr":"172.21.15.192:6827/33686","heartbeat_back_addr":"172.21.15.192:6831/33686","heartbeat_front_addr":"172.21.15.192:6829/33686","state":["exists","up"]},{"osd":7,"uuid":"f1ebdfd5-b859-482b-9f97-c18c4a26f445","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6808","nonce":33685},{"type":"v1","addr":"172.21.15.192:6809","nonce":33685}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":33685},{"type":"v1","addr":"172.21.15.192:6811","nonce":33685}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":33685},{"type":"v1","addr":"172.21.15.192:6815","nonce":33685}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6812","nonce":33685},{"type":"v1","addr":"172.21.15.192:6813","nonce":33685}]},"public_addr":"172.21.15.192:6809/33685","cluster_addr":"172.21.15.192:6811/33685","heartbeat_back_addr":"172.21.15.192:6815/33685","heartbeat_front_addr":"172.21.15.192:6813/33685","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T03:29:18.002048+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T03:29:17.478576+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T03:29:18.034944+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T03:29:17.628513+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T03:29:18.047338+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T03:29:17.772349+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T03:29:17.883958+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T03:29:18.151838+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-06T03:29:40.250 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:40.242+0000 7fd6a682d700 1 -- 172.21.15.110:0/2458040802 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fd684060550 msgr2=0x7fd684062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:40.250 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:40.242+0000 7fd6a682d700 1 --2- 172.21.15.110:0/2458040802 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fd684060550 0x7fd684062a00 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fd69c000c00 tx=0x7fd69c009040 comp rx=0 tx=0).stop 2023-10-06T03:29:40.250 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:40.242+0000 7fd6a682d700 1 -- 172.21.15.110:0/2458040802 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd6a0159000 msgr2=0x7fd6a00ae4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:40.250 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:40.242+0000 7fd6a682d700 1 --2- 172.21.15.110:0/2458040802 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd6a0159000 0x7fd6a00ae4b0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fd69800cb60 tx=0x7fd698001770 comp rx=0 tx=0).stop 2023-10-06T03:29:40.250 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:40.242+0000 7fd6a682d700 1 -- 172.21.15.110:0/2458040802 shutdown_connections 2023-10-06T03:29:40.250 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:40.242+0000 7fd6a682d700 1 --2- 172.21.15.110:0/2458040802 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fd684060550 0x7fd684062a00 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:40.250 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:40.242+0000 7fd6a682d700 1 --2- 172.21.15.110:0/2458040802 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd6a0159000 0x7fd6a00ae4b0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:40.250 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:40.242+0000 7fd6a682d700 1 --2- 172.21.15.110:0/2458040802 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd6a0158c30 0x7fd6a00aace0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:40.251 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:40.242+0000 7fd6a682d700 1 --2- 172.21.15.110:0/2458040802 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd6a00c7ba0 0x7fd6a01696f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:40.251 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:40.242+0000 7fd6a682d700 1 -- 172.21.15.110:0/2458040802 >> 172.21.15.110:0/2458040802 conn(0x7fd6a00c8d00 msgr2=0x7fd6a00c1030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:29:40.251 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:40.242+0000 7fd6a682d700 1 -- 172.21.15.110:0/2458040802 shutdown_connections 2023-10-06T03:29:40.251 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:40.242+0000 7fd6a682d700 1 -- 172.21.15.110:0/2458040802 wait complete. 2023-10-06T03:29:40.261 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-06T03:29:40.364 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:40.358+0000 7f2955029700 1 Processor -- start 2023-10-06T03:29:40.364 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:40.358+0000 7f2955029700 1 -- start start 2023-10-06T03:29:40.365 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:40.358+0000 7f2955029700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f29500cd110 0x7f2950155d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:40.365 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:40.358+0000 7f2955029700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f29500ad8a0 0x7f29500add40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:40.365 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:40.358+0000 7f2955029700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2950159b60 con 0x7f2950155930 2023-10-06T03:29:40.365 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:40.358+0000 7f2955029700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f29500c7180 con 0x7f29500cd110 2023-10-06T03:29:40.366 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:40.358+0000 7f2955029700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29500ae580 con 0x7f29500ad8a0 2023-10-06T03:29:40.366 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:40.358+0000 7f294ffff700 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f2950155930 0x7f29500cca00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:55540/0 (socket says 172.21.15.110:55540) 2023-10-06T03:29:40.366 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:40.358+0000 7f294ffff700 1 -- 172.21.15.110:0/452215559 learned_addr learned my addr 172.21.15.110:0/452215559 (peer_addr_for_me v1:172.21.15.110:0/0) 2023-10-06T03:29:40.366 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:40.358+0000 7f294f7fe700 1 --2- 172.21.15.110:0/452215559 >> v2:172.21.15.192:3300/0 conn(0x7f29500cd110 0x7f2950155d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:40.366 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:40.358+0000 7f2954828700 1 --2- 172.21.15.110:0/452215559 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f29500ad8a0 0x7f29500add40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:40.366 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:40.358+0000 7f294effd700 1 -- 172.21.15.110:0/452215559 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1587619469 0 0) 0x7f2950159b60 con 0x7f2950155930 2023-10-06T03:29:40.366 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:40.358+0000 7f294effd700 1 -- 172.21.15.110:0/452215559 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f293c003570 con 0x7f2950155930 2023-10-06T03:29:40.366 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:40.358+0000 7f294effd700 1 -- 172.21.15.110:0/452215559 <== mon.0 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3036437409 0 0) 0x7f293c003570 con 0x7f2950155930 2023-10-06T03:29:40.366 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:40.358+0000 7f294f7fe700 1 -- 172.21.15.110:0/452215559 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f29500ad8a0 msgr2=0x7f29500add40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:29:40.366 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:40.358+0000 7f294f7fe700 1 --2- 172.21.15.110:0/452215559 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f29500ad8a0 0x7f29500add40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:40.367 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:40.358+0000 7f294f7fe700 1 -- 172.21.15.110:0/452215559 >> v1:172.21.15.110:6789/0 conn(0x7f2950155930 legacy=0x7f29500cca00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:40.367 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:40.358+0000 7f294f7fe700 1 -- 172.21.15.110:0/452215559 --> v2:172.21.15.192:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29500ae860 con 0x7f29500cd110 2023-10-06T03:29:40.368 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:40.358+0000 7f294f7fe700 1 --2- 172.21.15.110:0/452215559 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f29500cd110 0x7f2950155d00 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f2948009e60 tx=0x7f2948009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=515f56584ebf9926 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:29:40.368 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:40.358+0000 7f294effd700 1 -- 172.21.15.110:0/452215559 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f294800e070 con 0x7f29500cd110 2023-10-06T03:29:40.368 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:40.358+0000 7f294effd700 1 -- 172.21.15.110:0/452215559 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f294800b7a0 con 0x7f29500cd110 2023-10-06T03:29:40.368 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:40.358+0000 7f294effd700 1 -- 172.21.15.110:0/452215559 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2938001490 con 0x7f29500cd110 2023-10-06T03:29:40.368 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:40.358+0000 7f2955029700 1 -- 172.21.15.110:0/452215559 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f29500cd110 msgr2=0x7f2950155d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:40.368 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:40.358+0000 7f2955029700 1 --2- 172.21.15.110:0/452215559 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f29500cd110 0x7f2950155d00 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f2948009e60 tx=0x7f2948009780 comp rx=0 tx=0).stop 2023-10-06T03:29:40.368 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:40.358+0000 7f2955029700 1 -- 172.21.15.110:0/452215559 shutdown_connections 2023-10-06T03:29:40.368 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:40.358+0000 7f2955029700 1 --2- 172.21.15.110:0/452215559 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f29500ad8a0 0x7f29500add40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:40.369 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:40.358+0000 7f2955029700 1 --2- 172.21.15.110:0/452215559 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f29500cd110 0x7f2950155d00 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:40.369 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:40.358+0000 7f2955029700 1 -- 172.21.15.110:0/452215559 >> 172.21.15.110:0/452215559 conn(0x7f29500670a0 msgr2=0x7f29500674a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:29:40.369 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:40.358+0000 7f2955029700 1 -- 172.21.15.110:0/452215559 shutdown_connections 2023-10-06T03:29:40.369 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:40.358+0000 7f2955029700 1 -- 172.21.15.110:0/452215559 wait complete. 2023-10-06T03:29:40.369 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:40.362+0000 7f2955029700 1 Processor -- start 2023-10-06T03:29:40.369 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:40.362+0000 7f2955029700 1 -- start start 2023-10-06T03:29:40.371 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:40.362+0000 7f2955029700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f29500ad8a0 0x7f29500af980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:40.371 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:40.362+0000 7f2955029700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f29500cd110 0x7f2950158c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:40.371 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:40.362+0000 7f2955029700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2950155930 0x7f295013faa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:40.371 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:40.362+0000 7f2955029700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2950159160 con 0x7f29500ad8a0 2023-10-06T03:29:40.371 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:40.362+0000 7f2955029700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29500b0b70 con 0x7f2950155930 2023-10-06T03:29:40.372 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:40.362+0000 7f2955029700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29500aff60 con 0x7f29500cd110 2023-10-06T03:29:40.372 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:40.362+0000 7f294ffff700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f29500ad8a0 0x7f29500af980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:40.372 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:40.362+0000 7f294ffff700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f29500ad8a0 0x7f29500af980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:36706/0 (socket says 172.21.15.110:36706) 2023-10-06T03:29:40.372 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:40.362+0000 7f294ffff700 1 -- 172.21.15.110:0/1248391411 learned_addr learned my addr 172.21.15.110:0/1248391411 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:29:40.372 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:40.362+0000 7f294f7fe700 1 --2- 172.21.15.110:0/1248391411 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f29500cd110 0x7f2950158c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:40.372 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:40.362+0000 7f2954828700 1 --2- 172.21.15.110:0/1248391411 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2950155930 0x7f295013faa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:40.372 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:40.362+0000 7f294ffff700 1 -- 172.21.15.110:0/1248391411 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f29500cd110 msgr2=0x7f2950158c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:40.372 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:40.362+0000 7f294ffff700 1 --2- 172.21.15.110:0/1248391411 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f29500cd110 0x7f2950158c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:40.372 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:40.362+0000 7f294ffff700 1 -- 172.21.15.110:0/1248391411 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2950155930 msgr2=0x7f295013faa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:40.372 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:40.362+0000 7f294ffff700 1 --2- 172.21.15.110:0/1248391411 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2950155930 0x7f295013faa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:40.372 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:40.362+0000 7f294ffff700 1 -- 172.21.15.110:0/1248391411 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2948009a20 con 0x7f29500ad8a0 2023-10-06T03:29:40.373 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:40.362+0000 7f294f7fe700 1 --2- 172.21.15.110:0/1248391411 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f29500cd110 0x7f2950158c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-06T03:29:40.373 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:40.362+0000 7f294ffff700 1 --2- 172.21.15.110:0/1248391411 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f29500ad8a0 0x7f29500af980 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f293800cb90 tx=0x7f2938001a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:29:40.373 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:40.362+0000 7f2947fff700 1 -- 172.21.15.110:0/1248391411 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2938007150 con 0x7f29500ad8a0 2023-10-06T03:29:40.373 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:40.362+0000 7f2955029700 1 -- 172.21.15.110:0/1248391411 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29500b0240 con 0x7f29500ad8a0 2023-10-06T03:29:40.373 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:40.362+0000 7f2955029700 1 -- 172.21.15.110:0/1248391411 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29500b0800 con 0x7f29500ad8a0 2023-10-06T03:29:40.373 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:40.362+0000 7f2947fff700 1 -- 172.21.15.110:0/1248391411 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29380074c0 con 0x7f29500ad8a0 2023-10-06T03:29:40.373 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:40.362+0000 7f2947fff700 1 -- 172.21.15.110:0/1248391411 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2938004830 con 0x7f29500ad8a0 2023-10-06T03:29:40.373 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:40.366+0000 7f2955029700 1 -- 172.21.15.110:0/1248391411 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f29500c3880 con 0x7f29500ad8a0 2023-10-06T03:29:40.374 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:40.366+0000 7f2947fff700 1 -- 172.21.15.110:0/1248391411 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7f293800a660 con 0x7f29500ad8a0 2023-10-06T03:29:40.374 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:40.366+0000 7f2947fff700 1 --2- 172.21.15.110:0/1248391411 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f2930060550 0x7f2930062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:40.374 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:40.366+0000 7f2947fff700 1 -- 172.21.15.110:0/1248391411 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6809+0+0 (secure 0 0 0) 0x7f293807c7d0 con 0x7f29500ad8a0 2023-10-06T03:29:40.374 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:40.366+0000 7f294f7fe700 1 --2- 172.21.15.110:0/1248391411 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f2930060550 0x7f2930062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:40.374 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:40.366+0000 7f294f7fe700 1 --2- 172.21.15.110:0/1248391411 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f2930060550 0x7f2930062a00 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f29480099f0 tx=0x7f2948018000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:29:40.380 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:40.374+0000 7f2947fff700 1 -- 172.21.15.110:0/1248391411 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f29500c3880 con 0x7f29500ad8a0 2023-10-06T03:29:40.542 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:40.534+0000 7f2955029700 1 -- 172.21.15.110:0/1248391411 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f29500c3880 con 0x7f29500ad8a0 2023-10-06T03:29:40.543 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:40.538+0000 7f2947fff700 1 -- 172.21.15.110:0/1248391411 <== mon.0 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v28) v1 ==== 74+0+18755 (secure 0 0 0) 0x7f29500c3880 con 0x7f29500ad8a0 2023-10-06T03:29:40.544 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-10-06T03:29:40.544 INFO:teuthology.orchestra.run.smithi110.stdout:{"epoch":28,"fsid":"93b8f6ac-4c92-4b45-b8e4-b59617024aa2","created":"2023-10-06T03:29:10.519583+0000","modified":"2023-10-06T03:29:38.951740+0000","last_up_change":"2023-10-06T03:29:21.960511+0000","last_in_change":"2023-10-06T03:29:12.560436+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-06T03:29:20.703743+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2023-10-06T03:29:28.307281+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2023-10-06T03:29:32.247251+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data","create_time":"2023-10-06T03:29:32.880086+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}}},{"pool":5,"pool_name":"cephfs_data_ec","create_time":"2023-10-06T03:29:36.613779+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"28","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"5866eb6c-2adc-46e4-be7a-e5bcda0f4065","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6824","nonce":33394},{"type":"v1","addr":"172.21.15.110:6825","nonce":33394}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6826","nonce":33394},{"type":"v1","addr":"172.21.15.110:6827","nonce":33394}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6830","nonce":33394},{"type":"v1","addr":"172.21.15.110:6831","nonce":33394}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6828","nonce":33394},{"type":"v1","addr":"172.21.15.110:6829","nonce":33394}]},"public_addr":"172.21.15.110:6825/33394","cluster_addr":"172.21.15.110:6827/33394","heartbeat_back_addr":"172.21.15.110:6831/33394","heartbeat_front_addr":"172.21.15.110:6829/33394","state":["exists","up"]},{"osd":1,"uuid":"a38de118-062b-4a05-9782-2ceddf6ff4df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6800","nonce":33396},{"type":"v1","addr":"172.21.15.110:6801","nonce":33396}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6802","nonce":33396},{"type":"v1","addr":"172.21.15.110:6803","nonce":33396}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6806","nonce":33396},{"type":"v1","addr":"172.21.15.110:6807","nonce":33396}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6804","nonce":33396},{"type":"v1","addr":"172.21.15.110:6805","nonce":33396}]},"public_addr":"172.21.15.110:6801/33396","cluster_addr":"172.21.15.110:6803/33396","heartbeat_back_addr":"172.21.15.110:6807/33396","heartbeat_front_addr":"172.21.15.110:6805/33396","state":["exists","up"]},{"osd":2,"uuid":"56e39be2-3407-4b69-877e-cf16617b06d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6816","nonce":33397},{"type":"v1","addr":"172.21.15.110:6817","nonce":33397}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6818","nonce":33397},{"type":"v1","addr":"172.21.15.110:6819","nonce":33397}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6822","nonce":33397},{"type":"v1","addr":"172.21.15.110:6823","nonce":33397}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6820","nonce":33397},{"type":"v1","addr":"172.21.15.110:6821","nonce":33397}]},"public_addr":"172.21.15.110:6817/33397","cluster_addr":"172.21.15.110:6819/33397","heartbeat_back_addr":"172.21.15.110:6823/33397","heartbeat_front_addr":"172.21.15.110:6821/33397","state":["exists","up"]},{"osd":3,"uuid":"01c82d13-c1d4-4bde-b55e-3e0089b40115","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6808","nonce":33395},{"type":"v1","addr":"172.21.15.110:6809","nonce":33395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6810","nonce":33395},{"type":"v1","addr":"172.21.15.110:6811","nonce":33395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6814","nonce":33395},{"type":"v1","addr":"172.21.15.110:6815","nonce":33395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6812","nonce":33395},{"type":"v1","addr":"172.21.15.110:6813","nonce":33395}]},"public_addr":"172.21.15.110:6809/33395","cluster_addr":"172.21.15.110:6811/33395","heartbeat_back_addr":"172.21.15.110:6815/33395","heartbeat_front_addr":"172.21.15.110:6813/33395","state":["exists","up"]},{"osd":4,"uuid":"86983430-5fa3-4d17-998f-03e2cd843237","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":33683},{"type":"v1","addr":"172.21.15.192:6817","nonce":33683}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":33683},{"type":"v1","addr":"172.21.15.192:6819","nonce":33683}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":33683},{"type":"v1","addr":"172.21.15.192:6823","nonce":33683}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6820","nonce":33683},{"type":"v1","addr":"172.21.15.192:6821","nonce":33683}]},"public_addr":"172.21.15.192:6817/33683","cluster_addr":"172.21.15.192:6819/33683","heartbeat_back_addr":"172.21.15.192:6823/33683","heartbeat_front_addr":"172.21.15.192:6821/33683","state":["exists","up"]},{"osd":5,"uuid":"6945257f-821d-4619-a63f-ebe857eba6f4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":33684},{"type":"v1","addr":"172.21.15.192:6801","nonce":33684}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":33684},{"type":"v1","addr":"172.21.15.192:6803","nonce":33684}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":33684},{"type":"v1","addr":"172.21.15.192:6807","nonce":33684}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6804","nonce":33684},{"type":"v1","addr":"172.21.15.192:6805","nonce":33684}]},"public_addr":"172.21.15.192:6801/33684","cluster_addr":"172.21.15.192:6803/33684","heartbeat_back_addr":"172.21.15.192:6807/33684","heartbeat_front_addr":"172.21.15.192:6805/33684","state":["exists","up"]},{"osd":6,"uuid":"7854d1ae-7060-4780-8f97-73cea5e7a12f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6824","nonce":33686},{"type":"v1","addr":"172.21.15.192:6825","nonce":33686}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":33686},{"type":"v1","addr":"172.21.15.192:6827","nonce":33686}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":33686},{"type":"v1","addr":"172.21.15.192:6831","nonce":33686}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6828","nonce":33686},{"type":"v1","addr":"172.21.15.192:6829","nonce":33686}]},"public_addr":"172.21.15.192:6825/33686","cluster_addr":"172.21.15.192:6827/33686","heartbeat_back_addr":"172.21.15.192:6831/33686","heartbeat_front_addr":"172.21.15.192:6829/33686","state":["exists","up"]},{"osd":7,"uuid":"f1ebdfd5-b859-482b-9f97-c18c4a26f445","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6808","nonce":33685},{"type":"v1","addr":"172.21.15.192:6809","nonce":33685}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":33685},{"type":"v1","addr":"172.21.15.192:6811","nonce":33685}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":33685},{"type":"v1","addr":"172.21.15.192:6815","nonce":33685}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6812","nonce":33685},{"type":"v1","addr":"172.21.15.192:6813","nonce":33685}]},"public_addr":"172.21.15.192:6809/33685","cluster_addr":"172.21.15.192:6811/33685","heartbeat_back_addr":"172.21.15.192:6815/33685","heartbeat_front_addr":"172.21.15.192:6813/33685","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T03:29:18.002048+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T03:29:17.478576+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T03:29:18.034944+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T03:29:17.628513+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T03:29:18.047338+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T03:29:17.772349+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T03:29:17.883958+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T03:29:18.151838+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-06T03:29:40.546 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:40.538+0000 7f2955029700 1 -- 172.21.15.110:0/1248391411 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f2930060550 msgr2=0x7f2930062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:40.546 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:40.538+0000 7f2955029700 1 --2- 172.21.15.110:0/1248391411 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f2930060550 0x7f2930062a00 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f29480099f0 tx=0x7f2948018000 comp rx=0 tx=0).stop 2023-10-06T03:29:40.546 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:40.538+0000 7f2955029700 1 -- 172.21.15.110:0/1248391411 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f29500ad8a0 msgr2=0x7f29500af980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:40.547 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:40.538+0000 7f2955029700 1 --2- 172.21.15.110:0/1248391411 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f29500ad8a0 0x7f29500af980 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f293800cb90 tx=0x7f2938001a50 comp rx=0 tx=0).stop 2023-10-06T03:29:40.547 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:40.538+0000 7f2955029700 1 -- 172.21.15.110:0/1248391411 shutdown_connections 2023-10-06T03:29:40.547 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:40.538+0000 7f2955029700 1 --2- 172.21.15.110:0/1248391411 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f2930060550 0x7f2930062a00 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:40.547 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:40.538+0000 7f2955029700 1 --2- 172.21.15.110:0/1248391411 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2950155930 0x7f295013faa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:40.547 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:40.538+0000 7f2955029700 1 --2- 172.21.15.110:0/1248391411 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f29500cd110 0x7f2950158c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:40.547 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:40.538+0000 7f2955029700 1 --2- 172.21.15.110:0/1248391411 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f29500ad8a0 0x7f29500af980 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:40.547 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:40.538+0000 7f2955029700 1 -- 172.21.15.110:0/1248391411 >> 172.21.15.110:0/1248391411 conn(0x7f29500670a0 msgr2=0x7f2950154250 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:29:40.547 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:40.538+0000 7f2955029700 1 -- 172.21.15.110:0/1248391411 shutdown_connections 2023-10-06T03:29:40.547 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:40.538+0000 7f2955029700 1 -- 172.21.15.110:0/1248391411 wait complete. 2023-10-06T03:29:40.571 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-06T03:29:40.681 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:40.674+0000 7f433d3f9700 1 Processor -- start 2023-10-06T03:29:40.682 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:40.674+0000 7f433d3f9700 1 -- start start 2023-10-06T03:29:40.683 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:40.674+0000 7f433d3f9700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f4338158be0 0x7f4338158fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:40.683 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:40.674+0000 7f433d3f9700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f43380ca7f0 0x7f43380cac70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:40.683 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:40.674+0000 7f433d3f9700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43380cc1a0 con 0x7f43380b1ba0 2023-10-06T03:29:40.684 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:40.674+0000 7f433d3f9700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f43380cc630 con 0x7f4338158be0 2023-10-06T03:29:40.684 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:40.674+0000 7f433d3f9700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43380cb590 con 0x7f43380ca7f0 2023-10-06T03:29:40.684 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:40.674+0000 7f43377fe700 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f43380b1ba0 0x7f43380b1ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:55556/0 (socket says 172.21.15.110:55556) 2023-10-06T03:29:40.684 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:40.674+0000 7f43377fe700 1 -- 172.21.15.110:0/979996387 learned_addr learned my addr 172.21.15.110:0/979996387 (peer_addr_for_me v1:172.21.15.110:0/0) 2023-10-06T03:29:40.684 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:40.674+0000 7f4337fff700 1 --2- 172.21.15.110:0/979996387 >> v2:172.21.15.192:3300/0 conn(0x7f4338158be0 0x7f4338158fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:40.684 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:40.674+0000 7f433cbf8700 1 --2- 172.21.15.110:0/979996387 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f43380ca7f0 0x7f43380cac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:40.684 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:40.678+0000 7f4336ffd700 1 -- 172.21.15.110:0/979996387 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 402228927 0 0) 0x7f43380cc1a0 con 0x7f43380b1ba0 2023-10-06T03:29:40.684 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:40.678+0000 7f4336ffd700 1 -- 172.21.15.110:0/979996387 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4324003570 con 0x7f43380b1ba0 2023-10-06T03:29:40.684 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:40.678+0000 7f4337fff700 1 -- 172.21.15.110:0/979996387 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f43380ca7f0 msgr2=0x7f43380cac70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:29:40.685 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:40.678+0000 7f4337fff700 1 --2- 172.21.15.110:0/979996387 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f43380ca7f0 0x7f43380cac70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:40.685 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:40.678+0000 7f4337fff700 1 -- 172.21.15.110:0/979996387 >> v1:172.21.15.110:6789/0 conn(0x7f43380b1ba0 legacy=0x7f43380b1ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:40.685 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:40.678+0000 7f4337fff700 1 -- 172.21.15.110:0/979996387 --> v2:172.21.15.192:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43380cb870 con 0x7f4338158be0 2023-10-06T03:29:40.686 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:40.678+0000 7f4337fff700 1 --2- 172.21.15.110:0/979996387 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4338158be0 0x7f4338158fb0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f4330009e60 tx=0x7f4330009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b07bfa5774f3e603 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:29:40.686 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:40.678+0000 7f4336ffd700 1 -- 172.21.15.110:0/979996387 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f433000e070 con 0x7f4338158be0 2023-10-06T03:29:40.686 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:40.678+0000 7f4336ffd700 1 -- 172.21.15.110:0/979996387 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f433000bab0 con 0x7f4338158be0 2023-10-06T03:29:40.687 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:40.678+0000 7f4336ffd700 1 -- 172.21.15.110:0/979996387 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f433000be30 con 0x7f4338158be0 2023-10-06T03:29:40.687 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:40.678+0000 7f433d3f9700 1 -- 172.21.15.110:0/979996387 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4338158be0 msgr2=0x7f4338158fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:40.687 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:40.678+0000 7f433d3f9700 1 --2- 172.21.15.110:0/979996387 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4338158be0 0x7f4338158fb0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f4330009e60 tx=0x7f4330009780 comp rx=0 tx=0).stop 2023-10-06T03:29:40.687 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:40.678+0000 7f433d3f9700 1 -- 172.21.15.110:0/979996387 shutdown_connections 2023-10-06T03:29:40.687 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:40.678+0000 7f433d3f9700 1 --2- 172.21.15.110:0/979996387 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f43380ca7f0 0x7f43380cac70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:40.687 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:40.678+0000 7f433d3f9700 1 --2- 172.21.15.110:0/979996387 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4338158be0 0x7f4338158fb0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:40.687 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:40.678+0000 7f433d3f9700 1 -- 172.21.15.110:0/979996387 >> 172.21.15.110:0/979996387 conn(0x7f43380d08c0 msgr2=0x7f43380cc860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:29:40.688 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:40.678+0000 7f433d3f9700 1 -- 172.21.15.110:0/979996387 shutdown_connections 2023-10-06T03:29:40.688 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:40.678+0000 7f433d3f9700 1 -- 172.21.15.110:0/979996387 wait complete. 2023-10-06T03:29:40.691 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:40.678+0000 7f433d3f9700 1 Processor -- start 2023-10-06T03:29:40.691 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:40.678+0000 7f433d3f9700 1 -- start start 2023-10-06T03:29:40.691 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:40.682+0000 7f433d3f9700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f43380b1ba0 0x7f43380c8940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:40.691 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:40.682+0000 7f433d3f9700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f43380ca7f0 0x7f4338153810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:40.691 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:40.682+0000 7f433d3f9700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4338158be0 0x7f43380c6e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:40.692 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:40.682+0000 7f433d3f9700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43380c84b0 con 0x7f43380b1ba0 2023-10-06T03:29:40.692 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:40.682+0000 7f433d3f9700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43380c78a0 con 0x7f43380ca7f0 2023-10-06T03:29:40.692 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:40.682+0000 7f433d3f9700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43380c7b80 con 0x7f4338158be0 2023-10-06T03:29:40.692 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:40.682+0000 7f4337fff700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f43380b1ba0 0x7f43380c8940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:40.692 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:40.682+0000 7f4337fff700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f43380b1ba0 0x7f43380c8940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:36720/0 (socket says 172.21.15.110:36720) 2023-10-06T03:29:40.692 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:40.682+0000 7f4337fff700 1 -- 172.21.15.110:0/462721616 learned_addr learned my addr 172.21.15.110:0/462721616 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:29:40.692 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:40.682+0000 7f43377fe700 1 --2- 172.21.15.110:0/462721616 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f43380ca7f0 0x7f4338153810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:40.692 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:40.682+0000 7f433cbf8700 1 --2- 172.21.15.110:0/462721616 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4338158be0 0x7f43380c6e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:40.692 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:40.682+0000 7f4337fff700 1 -- 172.21.15.110:0/462721616 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4338158be0 msgr2=0x7f43380c6e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:40.692 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:40.682+0000 7f4337fff700 1 --2- 172.21.15.110:0/462721616 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4338158be0 0x7f43380c6e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:40.693 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:40.682+0000 7f4337fff700 1 -- 172.21.15.110:0/462721616 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f43380ca7f0 msgr2=0x7f4338153810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:40.693 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:40.682+0000 7f4337fff700 1 --2- 172.21.15.110:0/462721616 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f43380ca7f0 0x7f4338153810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:40.693 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:40.682+0000 7f4337fff700 1 -- 172.21.15.110:0/462721616 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4330009a20 con 0x7f43380b1ba0 2023-10-06T03:29:40.693 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:40.682+0000 7f43377fe700 1 --2- 172.21.15.110:0/462721616 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f43380ca7f0 0x7f4338153810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-06T03:29:40.693 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:40.682+0000 7f4337fff700 1 --2- 172.21.15.110:0/462721616 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f43380b1ba0 0x7f43380c8940 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f43300097c0 tx=0x7f4330019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:29:40.693 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:40.682+0000 7f432ffff700 1 -- 172.21.15.110:0/462721616 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43300128c0 con 0x7f43380b1ba0 2023-10-06T03:29:40.693 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:40.682+0000 7f432ffff700 1 -- 172.21.15.110:0/462721616 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4330021070 con 0x7f43380b1ba0 2023-10-06T03:29:40.693 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:40.682+0000 7f433d3f9700 1 -- 172.21.15.110:0/462721616 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43380c7e60 con 0x7f43380b1ba0 2023-10-06T03:29:40.693 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:40.682+0000 7f432ffff700 1 -- 172.21.15.110:0/462721616 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4330012c70 con 0x7f43380b1ba0 2023-10-06T03:29:40.693 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:40.682+0000 7f433d3f9700 1 -- 172.21.15.110:0/462721616 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4338153d50 con 0x7f43380b1ba0 2023-10-06T03:29:40.694 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:40.686+0000 7f433d3f9700 1 -- 172.21.15.110:0/462721616 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f43380ca180 con 0x7f43380b1ba0 2023-10-06T03:29:40.695 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:40.686+0000 7f432ffff700 1 -- 172.21.15.110:0/462721616 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7f4330023030 con 0x7f43380b1ba0 2023-10-06T03:29:40.695 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:40.686+0000 7f432ffff700 1 --2- 172.21.15.110:0/462721616 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f4318060580 0x7f4318062a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:40.695 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:40.686+0000 7f432ffff700 1 -- 172.21.15.110:0/462721616 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6809+0+0 (secure 0 0 0) 0x7f433007e8c0 con 0x7f43380b1ba0 2023-10-06T03:29:40.695 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:40.686+0000 7f43377fe700 1 --2- 172.21.15.110:0/462721616 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f4318060580 0x7f4318062a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:40.695 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:40.686+0000 7f43377fe700 1 --2- 172.21.15.110:0/462721616 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f4318060580 0x7f4318062a30 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f43380b0460 tx=0x7f4320009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:29:40.699 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:40.694+0000 7f432ffff700 1 -- 172.21.15.110:0/462721616 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f433004ea60 con 0x7f43380b1ba0 2023-10-06T03:29:40.896 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:40.886+0000 7f433d3f9700 1 -- 172.21.15.110:0/462721616 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4338070ff0 con 0x7f43380b1ba0 2023-10-06T03:29:40.897 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:40.890+0000 7f432ffff700 1 -- 172.21.15.110:0/462721616 <== mon.0 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) v1 ==== 93+0+4602 (secure 0 0 0) 0x7f4330052080 con 0x7f43380b1ba0 2023-10-06T03:29:40.897 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-10-06T03:29:40.898 INFO:teuthology.orchestra.run.smithi110.stdout:{"epoch":6,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4279,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6833/1434505700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6832","nonce":1434505700},{"type":"v1","addr":"172.21.15.192:6833","nonce":1434505700}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4281,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/1168370789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6834","nonce":1168370789},{"type":"v1","addr":"172.21.15.110:6835","nonce":1168370789}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4288,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6835/1271810086","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":1271810086},{"type":"v1","addr":"172.21.15.192:6835","nonce":1271810086}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":6,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-06T03:29:33.894998+0000","modified":"2023-10-06T03:29:39.611370+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4293},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4293":{"gid":4293,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.110:6837/540320464","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6836","nonce":540320464},{"type":"v1","addr":"172.21.15.110:6837","nonce":540320464}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2023-10-06T03:29:40.899 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:40.890+0000 7f433d3f9700 1 -- 172.21.15.110:0/462721616 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f4318060580 msgr2=0x7f4318062a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:40.900 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:40.890+0000 7f433d3f9700 1 --2- 172.21.15.110:0/462721616 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f4318060580 0x7f4318062a30 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f43380b0460 tx=0x7f4320009040 comp rx=0 tx=0).stop 2023-10-06T03:29:40.900 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:40.890+0000 7f433d3f9700 1 -- 172.21.15.110:0/462721616 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f43380b1ba0 msgr2=0x7f43380c8940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:40.900 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:40.890+0000 7f433d3f9700 1 --2- 172.21.15.110:0/462721616 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f43380b1ba0 0x7f43380c8940 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f43300097c0 tx=0x7f4330019000 comp rx=0 tx=0).stop 2023-10-06T03:29:40.900 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:40.890+0000 7f433d3f9700 1 -- 172.21.15.110:0/462721616 shutdown_connections 2023-10-06T03:29:40.900 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:40.890+0000 7f433d3f9700 1 --2- 172.21.15.110:0/462721616 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f4318060580 0x7f4318062a30 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:40.900 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:40.890+0000 7f433d3f9700 1 --2- 172.21.15.110:0/462721616 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4338158be0 0x7f43380c6e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:40.900 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:40.890+0000 7f433d3f9700 1 --2- 172.21.15.110:0/462721616 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f43380ca7f0 0x7f4338153810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:40.900 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:40.890+0000 7f433d3f9700 1 --2- 172.21.15.110:0/462721616 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f43380b1ba0 0x7f43380c8940 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:40.900 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:40.890+0000 7f433d3f9700 1 -- 172.21.15.110:0/462721616 >> 172.21.15.110:0/462721616 conn(0x7f43380d08c0 msgr2=0x7f4338154650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:29:40.901 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:40.890+0000 7f433d3f9700 1 -- 172.21.15.110:0/462721616 shutdown_connections 2023-10-06T03:29:40.901 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:40.890+0000 7f433d3f9700 1 -- 172.21.15.110:0/462721616 wait complete. 2023-10-06T03:29:40.901 INFO:teuthology.orchestra.run.smithi110.stderr:dumped fsmap epoch 6 2023-10-06T03:29:40.910 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-06T03:29:41.011 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.002+0000 7f325d151700 1 Processor -- start 2023-10-06T03:29:41.011 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.002+0000 7f325d151700 1 -- start start 2023-10-06T03:29:41.015 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.006+0000 7f325d151700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3258158ee0 0x7f325806dd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:41.015 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.006+0000 7f325d151700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f325806d8e0 con 0x7f3258158b10 2023-10-06T03:29:41.015 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.006+0000 7f325d151700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f325806ccd0 con 0x7f32580b22b0 2023-10-06T03:29:41.015 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.006+0000 7f325d151700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3258159330 con 0x7f3258158ee0 2023-10-06T03:29:41.015 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.006+0000 7f3257fff700 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f3258158b10 0x7f325806c180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:37214/0 (socket says 172.21.15.110:37214) 2023-10-06T03:29:41.015 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.006+0000 7f3257fff700 1 -- 172.21.15.110:0/2049351182 learned_addr learned my addr 172.21.15.110:0/2049351182 (peer_addr_for_me v1:172.21.15.110:0/0) 2023-10-06T03:29:41.015 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.006+0000 7f32577fe700 1 --2- 172.21.15.110:0/2049351182 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3258158ee0 0x7f325806dd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:41.016 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.006+0000 7f3256ffd700 1 -- 172.21.15.110:0/2049351182 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2085561577 0 0) 0x7f325806d8e0 con 0x7f3258158b10 2023-10-06T03:29:41.016 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.006+0000 7f3256ffd700 1 -- 172.21.15.110:0/2049351182 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3244003570 con 0x7f3258158b10 2023-10-06T03:29:41.016 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.006+0000 7f32577fe700 1 -- 172.21.15.110:0/2049351182 >> v1:172.21.15.192:6789/0 conn(0x7f32580b22b0 legacy=0x7f32580b0140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:41.016 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.006+0000 7f3256ffd700 1 -- 172.21.15.110:0/2049351182 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3530908051 0 0) 0x7f325806ccd0 con 0x7f32580b22b0 2023-10-06T03:29:41.016 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.006+0000 7f32577fe700 1 -- 172.21.15.110:0/2049351182 >> v1:172.21.15.110:6789/0 conn(0x7f3258158b10 legacy=0x7f325806c180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:41.016 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.006+0000 7f32577fe700 1 -- 172.21.15.110:0/2049351182 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f325806cfb0 con 0x7f3258158ee0 2023-10-06T03:29:41.016 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.006+0000 7f32577fe700 1 --2- 172.21.15.110:0/2049351182 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3258158ee0 0x7f325806dd70 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f3240009e90 tx=0x7f3240009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=71670cfc03707169 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:29:41.016 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.006+0000 7f3256ffd700 1 -- 172.21.15.110:0/2049351182 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f324000e070 con 0x7f3258158ee0 2023-10-06T03:29:41.016 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.006+0000 7f3256ffd700 1 -- 172.21.15.110:0/2049351182 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f324000bc30 con 0x7f3258158ee0 2023-10-06T03:29:41.016 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.006+0000 7f3256ffd700 1 -- 172.21.15.110:0/2049351182 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3240012430 con 0x7f3258158ee0 2023-10-06T03:29:41.016 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.006+0000 7f325d151700 1 -- 172.21.15.110:0/2049351182 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3258158ee0 msgr2=0x7f325806dd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:41.017 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.006+0000 7f325d151700 1 --2- 172.21.15.110:0/2049351182 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3258158ee0 0x7f325806dd70 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f3240009e90 tx=0x7f3240009780 comp rx=0 tx=0).stop 2023-10-06T03:29:41.017 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.006+0000 7f325d151700 1 -- 172.21.15.110:0/2049351182 shutdown_connections 2023-10-06T03:29:41.017 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.006+0000 7f325d151700 1 --2- 172.21.15.110:0/2049351182 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3258158ee0 0x7f325806dd70 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:41.017 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.006+0000 7f325d151700 1 -- 172.21.15.110:0/2049351182 >> 172.21.15.110:0/2049351182 conn(0x7f325806a540 msgr2=0x7f325806a910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:29:41.017 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.006+0000 7f325d151700 1 -- 172.21.15.110:0/2049351182 shutdown_connections 2023-10-06T03:29:41.017 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.006+0000 7f325d151700 1 -- 172.21.15.110:0/2049351182 wait complete. 2023-10-06T03:29:41.019 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.010+0000 7f325d151700 1 Processor -- start 2023-10-06T03:29:41.019 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.010+0000 7f325d151700 1 -- start start 2023-10-06T03:29:41.020 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.010+0000 7f325d151700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f32580b22b0 0x7f325814c790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:41.020 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.010+0000 7f325d151700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3258158b10 0x7f3258155b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:41.020 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.010+0000 7f325d151700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3258158ee0 0x7f3258151630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:41.020 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.010+0000 7f325d151700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f325814ccd0 con 0x7f3258158b10 2023-10-06T03:29:41.020 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.010+0000 7f325d151700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f325814cf20 con 0x7f32580b22b0 2023-10-06T03:29:41.020 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.010+0000 7f325d151700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f325814d4e0 con 0x7f3258158ee0 2023-10-06T03:29:41.020 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.010+0000 7f32577fe700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3258158b10 0x7f3258155b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:41.020 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.010+0000 7f32577fe700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3258158b10 0x7f3258155b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:60292/0 (socket says 172.21.15.110:60292) 2023-10-06T03:29:41.021 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.010+0000 7f32577fe700 1 -- 172.21.15.110:0/3933765589 learned_addr learned my addr 172.21.15.110:0/3933765589 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:29:41.021 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.010+0000 7f3257fff700 1 --2- 172.21.15.110:0/3933765589 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f32580b22b0 0x7f325814c790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:41.021 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.010+0000 7f325c950700 1 --2- 172.21.15.110:0/3933765589 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3258158ee0 0x7f3258151630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:41.021 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.010+0000 7f32577fe700 1 -- 172.21.15.110:0/3933765589 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3258158ee0 msgr2=0x7f3258151630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:41.021 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.010+0000 7f32577fe700 1 --2- 172.21.15.110:0/3933765589 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3258158ee0 0x7f3258151630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:41.021 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.010+0000 7f32577fe700 1 -- 172.21.15.110:0/3933765589 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f32580b22b0 msgr2=0x7f325814c790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:41.021 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.010+0000 7f32577fe700 1 --2- 172.21.15.110:0/3933765589 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f32580b22b0 0x7f325814c790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:41.021 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.010+0000 7f32577fe700 1 -- 172.21.15.110:0/3933765589 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3240009a20 con 0x7f3258158b10 2023-10-06T03:29:41.021 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.010+0000 7f3257fff700 1 --2- 172.21.15.110:0/3933765589 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f32580b22b0 0x7f325814c790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-06T03:29:41.021 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.010+0000 7f32577fe700 1 --2- 172.21.15.110:0/3933765589 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3258158b10 0x7f3258155b30 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f324000b6c0 tx=0x7f3240019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:29:41.021 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.010+0000 7f324ffff700 1 -- 172.21.15.110:0/3933765589 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3240026480 con 0x7f3258158b10 2023-10-06T03:29:41.021 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.010+0000 7f324ffff700 1 -- 172.21.15.110:0/3933765589 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32400265e0 con 0x7f3258158b10 2023-10-06T03:29:41.022 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.010+0000 7f325d151700 1 -- 172.21.15.110:0/3933765589 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32580ad070 con 0x7f3258158b10 2023-10-06T03:29:41.022 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.010+0000 7f324ffff700 1 -- 172.21.15.110:0/3933765589 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32400269a0 con 0x7f3258158b10 2023-10-06T03:29:41.022 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.010+0000 7f325d151700 1 -- 172.21.15.110:0/3933765589 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f325814d170 con 0x7f3258158b10 2023-10-06T03:29:41.022 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.010+0000 7f325d151700 1 -- 172.21.15.110:0/3933765589 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3258070ff0 con 0x7f3258158b10 2023-10-06T03:29:41.022 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.014+0000 7f324ffff700 1 -- 172.21.15.110:0/3933765589 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7f324001c400 con 0x7f3258158b10 2023-10-06T03:29:41.023 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.014+0000 7f324ffff700 1 --2- 172.21.15.110:0/3933765589 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f3238060550 0x7f3238062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:41.023 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.014+0000 7f3257fff700 1 --2- 172.21.15.110:0/3933765589 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f3238060550 0x7f3238062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:41.023 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.014+0000 7f324ffff700 1 -- 172.21.15.110:0/3933765589 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6809+0+0 (secure 0 0 0) 0x7f324007eea0 con 0x7f3258158b10 2023-10-06T03:29:41.023 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.014+0000 7f3257fff700 1 --2- 172.21.15.110:0/3933765589 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f3238060550 0x7f3238062a00 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f32480058d0 tx=0x7f3248009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:29:41.024 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.018+0000 7f324ffff700 1 -- 172.21.15.110:0/3933765589 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f3240052ad0 con 0x7f3258158b10 2023-10-06T03:29:41.186 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.178+0000 7f325d151700 1 -- 172.21.15.110:0/3933765589 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f325814d620 con 0x7f3258158b10 2023-10-06T03:29:41.188 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.182+0000 7f324ffff700 1 -- 172.21.15.110:0/3933765589 <== mon.0 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v28) v1 ==== 74+0+18755 (secure 0 0 0) 0x7f3240052660 con 0x7f3258158b10 2023-10-06T03:29:41.189 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-10-06T03:29:41.189 INFO:teuthology.orchestra.run.smithi110.stdout:{"epoch":28,"fsid":"93b8f6ac-4c92-4b45-b8e4-b59617024aa2","created":"2023-10-06T03:29:10.519583+0000","modified":"2023-10-06T03:29:38.951740+0000","last_up_change":"2023-10-06T03:29:21.960511+0000","last_in_change":"2023-10-06T03:29:12.560436+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-06T03:29:20.703743+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2023-10-06T03:29:28.307281+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2023-10-06T03:29:32.247251+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data","create_time":"2023-10-06T03:29:32.880086+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}}},{"pool":5,"pool_name":"cephfs_data_ec","create_time":"2023-10-06T03:29:36.613779+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"28","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"5866eb6c-2adc-46e4-be7a-e5bcda0f4065","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6824","nonce":33394},{"type":"v1","addr":"172.21.15.110:6825","nonce":33394}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6826","nonce":33394},{"type":"v1","addr":"172.21.15.110:6827","nonce":33394}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6830","nonce":33394},{"type":"v1","addr":"172.21.15.110:6831","nonce":33394}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6828","nonce":33394},{"type":"v1","addr":"172.21.15.110:6829","nonce":33394}]},"public_addr":"172.21.15.110:6825/33394","cluster_addr":"172.21.15.110:6827/33394","heartbeat_back_addr":"172.21.15.110:6831/33394","heartbeat_front_addr":"172.21.15.110:6829/33394","state":["exists","up"]},{"osd":1,"uuid":"a38de118-062b-4a05-9782-2ceddf6ff4df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6800","nonce":33396},{"type":"v1","addr":"172.21.15.110:6801","nonce":33396}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6802","nonce":33396},{"type":"v1","addr":"172.21.15.110:6803","nonce":33396}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6806","nonce":33396},{"type":"v1","addr":"172.21.15.110:6807","nonce":33396}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6804","nonce":33396},{"type":"v1","addr":"172.21.15.110:6805","nonce":33396}]},"public_addr":"172.21.15.110:6801/33396","cluster_addr":"172.21.15.110:6803/33396","heartbeat_back_addr":"172.21.15.110:6807/33396","heartbeat_front_addr":"172.21.15.110:6805/33396","state":["exists","up"]},{"osd":2,"uuid":"56e39be2-3407-4b69-877e-cf16617b06d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6816","nonce":33397},{"type":"v1","addr":"172.21.15.110:6817","nonce":33397}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6818","nonce":33397},{"type":"v1","addr":"172.21.15.110:6819","nonce":33397}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6822","nonce":33397},{"type":"v1","addr":"172.21.15.110:6823","nonce":33397}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6820","nonce":33397},{"type":"v1","addr":"172.21.15.110:6821","nonce":33397}]},"public_addr":"172.21.15.110:6817/33397","cluster_addr":"172.21.15.110:6819/33397","heartbeat_back_addr":"172.21.15.110:6823/33397","heartbeat_front_addr":"172.21.15.110:6821/33397","state":["exists","up"]},{"osd":3,"uuid":"01c82d13-c1d4-4bde-b55e-3e0089b40115","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6808","nonce":33395},{"type":"v1","addr":"172.21.15.110:6809","nonce":33395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6810","nonce":33395},{"type":"v1","addr":"172.21.15.110:6811","nonce":33395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6814","nonce":33395},{"type":"v1","addr":"172.21.15.110:6815","nonce":33395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6812","nonce":33395},{"type":"v1","addr":"172.21.15.110:6813","nonce":33395}]},"public_addr":"172.21.15.110:6809/33395","cluster_addr":"172.21.15.110:6811/33395","heartbeat_back_addr":"172.21.15.110:6815/33395","heartbeat_front_addr":"172.21.15.110:6813/33395","state":["exists","up"]},{"osd":4,"uuid":"86983430-5fa3-4d17-998f-03e2cd843237","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":33683},{"type":"v1","addr":"172.21.15.192:6817","nonce":33683}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":33683},{"type":"v1","addr":"172.21.15.192:6819","nonce":33683}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":33683},{"type":"v1","addr":"172.21.15.192:6823","nonce":33683}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6820","nonce":33683},{"type":"v1","addr":"172.21.15.192:6821","nonce":33683}]},"public_addr":"172.21.15.192:6817/33683","cluster_addr":"172.21.15.192:6819/33683","heartbeat_back_addr":"172.21.15.192:6823/33683","heartbeat_front_addr":"172.21.15.192:6821/33683","state":["exists","up"]},{"osd":5,"uuid":"6945257f-821d-4619-a63f-ebe857eba6f4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":33684},{"type":"v1","addr":"172.21.15.192:6801","nonce":33684}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":33684},{"type":"v1","addr":"172.21.15.192:6803","nonce":33684}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":33684},{"type":"v1","addr":"172.21.15.192:6807","nonce":33684}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6804","nonce":33684},{"type":"v1","addr":"172.21.15.192:6805","nonce":33684}]},"public_addr":"172.21.15.192:6801/33684","cluster_addr":"172.21.15.192:6803/33684","heartbeat_back_addr":"172.21.15.192:6807/33684","heartbeat_front_addr":"172.21.15.192:6805/33684","state":["exists","up"]},{"osd":6,"uuid":"7854d1ae-7060-4780-8f97-73cea5e7a12f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6824","nonce":33686},{"type":"v1","addr":"172.21.15.192:6825","nonce":33686}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":33686},{"type":"v1","addr":"172.21.15.192:6827","nonce":33686}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":33686},{"type":"v1","addr":"172.21.15.192:6831","nonce":33686}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6828","nonce":33686},{"type":"v1","addr":"172.21.15.192:6829","nonce":33686}]},"public_addr":"172.21.15.192:6825/33686","cluster_addr":"172.21.15.192:6827/33686","heartbeat_back_addr":"172.21.15.192:6831/33686","heartbeat_front_addr":"172.21.15.192:6829/33686","state":["exists","up"]},{"osd":7,"uuid":"f1ebdfd5-b859-482b-9f97-c18c4a26f445","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6808","nonce":33685},{"type":"v1","addr":"172.21.15.192:6809","nonce":33685}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":33685},{"type":"v1","addr":"172.21.15.192:6811","nonce":33685}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":33685},{"type":"v1","addr":"172.21.15.192:6815","nonce":33685}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6812","nonce":33685},{"type":"v1","addr":"172.21.15.192:6813","nonce":33685}]},"public_addr":"172.21.15.192:6809/33685","cluster_addr":"172.21.15.192:6811/33685","heartbeat_back_addr":"172.21.15.192:6815/33685","heartbeat_front_addr":"172.21.15.192:6813/33685","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T03:29:18.002048+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T03:29:17.478576+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T03:29:18.034944+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T03:29:17.628513+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T03:29:18.047338+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T03:29:17.772349+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T03:29:17.883958+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T03:29:18.151838+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-06T03:29:41.190 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.182+0000 7f325d151700 1 -- 172.21.15.110:0/3933765589 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f3238060550 msgr2=0x7f3238062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:41.190 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.182+0000 7f325d151700 1 --2- 172.21.15.110:0/3933765589 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f3238060550 0x7f3238062a00 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f32480058d0 tx=0x7f3248009040 comp rx=0 tx=0).stop 2023-10-06T03:29:41.190 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.182+0000 7f325d151700 1 -- 172.21.15.110:0/3933765589 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3258158b10 msgr2=0x7f3258155b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:41.190 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.182+0000 7f325d151700 1 --2- 172.21.15.110:0/3933765589 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3258158b10 0x7f3258155b30 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f324000b6c0 tx=0x7f3240019000 comp rx=0 tx=0).stop 2023-10-06T03:29:41.191 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.182+0000 7f325d151700 1 -- 172.21.15.110:0/3933765589 shutdown_connections 2023-10-06T03:29:41.191 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.182+0000 7f325d151700 1 --2- 172.21.15.110:0/3933765589 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f3238060550 0x7f3238062a00 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:41.191 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.182+0000 7f325d151700 1 --2- 172.21.15.110:0/3933765589 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3258158ee0 0x7f3258151630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:41.191 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.182+0000 7f325d151700 1 --2- 172.21.15.110:0/3933765589 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3258158b10 0x7f3258155b30 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:41.191 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.182+0000 7f325d151700 1 --2- 172.21.15.110:0/3933765589 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f32580b22b0 0x7f325814c790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:41.191 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.182+0000 7f325d151700 1 -- 172.21.15.110:0/3933765589 >> 172.21.15.110:0/3933765589 conn(0x7f325806a540 msgr2=0x7f32580c6c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:29:41.191 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.182+0000 7f325d151700 1 -- 172.21.15.110:0/3933765589 shutdown_connections 2023-10-06T03:29:41.192 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.182+0000 7f325d151700 1 -- 172.21.15.110:0/3933765589 wait complete. 2023-10-06T03:29:41.606 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 6, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-10-06T03:29:33.894998+0000', 'modified': '2023-10-06T03:29:39.611370+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4293}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4293': {'gid': 4293, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.110:6837/540320464', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.110:6836', 'nonce': 540320464}, {'type': 'v1', 'addr': '172.21.15.110:6837', 'nonce': 540320464}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138320759226367, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [4, 5], 'metadata_pool': 3, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2023-10-06T03:29:41.606 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2023-10-06T03:29:41.606 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2023-10-06T03:29:41.721 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.718+0000 7f3121f59700 1 Processor -- start 2023-10-06T03:29:41.721 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.718+0000 7f3121f59700 1 -- start start 2023-10-06T03:29:41.723 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.718+0000 7f3121f59700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f311c0c98d0 0x7f311c0c9ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:41.723 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.718+0000 7f3121f59700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f311c0cb540 0x7f311c0c7a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:41.723 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.718+0000 7f3121f59700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f311c06d030 con 0x7f311c159d90 2023-10-06T03:29:41.723 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.718+0000 7f3121f59700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f311c06c7d0 con 0x7f311c0cb540 2023-10-06T03:29:41.723 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.718+0000 7f3121f59700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f311c0cb970 con 0x7f311c0c98d0 2023-10-06T03:29:41.723 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.718+0000 7f311b7fe700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f311c0c98d0 0x7f311c0c9ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:41.723 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.718+0000 7f311affd700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f311c0cb540 0x7f311c0c7a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:41.723 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.718+0000 7f311b7fe700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f311c0c98d0 0x7f311c0c9ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3301/0 says I am v2:172.21.15.110:43760/0 (socket says 172.21.15.110:43760) 2023-10-06T03:29:41.724 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.718+0000 7f311b7fe700 1 -- 172.21.15.110:0/1771845452 learned_addr learned my addr 172.21.15.110:0/1771845452 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:29:41.725 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.718+0000 7f311a7fc700 1 -- 172.21.15.110:0/1771845452 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4100110956 0 0) 0x7f311c06d030 con 0x7f311c159d90 2023-10-06T03:29:41.725 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.718+0000 7f311a7fc700 1 -- 172.21.15.110:0/1771845452 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3108003570 con 0x7f311c159d90 2023-10-06T03:29:41.725 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.718+0000 7f311affd700 1 -- 172.21.15.110:0/1771845452 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f311c0c98d0 msgr2=0x7f311c0c9ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:29:41.726 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.718+0000 7f311affd700 1 --2- 172.21.15.110:0/1771845452 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f311c0c98d0 0x7f311c0c9ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:41.726 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.718+0000 7f311affd700 1 -- 172.21.15.110:0/1771845452 >> v1:172.21.15.192:6789/0 conn(0x7f311c159d90 legacy=0x7f311c15a1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:41.726 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.718+0000 7f311affd700 1 -- 172.21.15.110:0/1771845452 --> v2:172.21.15.192:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f311c0c82c0 con 0x7f311c0cb540 2023-10-06T03:29:41.726 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.718+0000 7f311b7fe700 1 --2- 172.21.15.110:0/1771845452 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f311c0c98d0 0x7f311c0c9ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-06T03:29:41.726 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.718+0000 7f311affd700 1 --2- 172.21.15.110:0/1771845452 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f311c0cb540 0x7f311c0c7a90 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f3104009e60 tx=0x7f3104009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9169adf69c93cd74 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:29:41.726 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.718+0000 7f311a7fc700 1 -- 172.21.15.110:0/1771845452 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f310400e070 con 0x7f311c0cb540 2023-10-06T03:29:41.726 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.718+0000 7f311a7fc700 1 -- 172.21.15.110:0/1771845452 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f310400bab0 con 0x7f311c0cb540 2023-10-06T03:29:41.726 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.718+0000 7f311a7fc700 1 -- 172.21.15.110:0/1771845452 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f310400be50 con 0x7f311c0cb540 2023-10-06T03:29:41.726 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.718+0000 7f3121f59700 1 -- 172.21.15.110:0/1771845452 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f311c0cb540 msgr2=0x7f311c0c7a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:41.726 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.718+0000 7f3121f59700 1 --2- 172.21.15.110:0/1771845452 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f311c0cb540 0x7f311c0c7a90 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f3104009e60 tx=0x7f3104009780 comp rx=0 tx=0).stop 2023-10-06T03:29:41.726 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.718+0000 7f3121f59700 1 -- 172.21.15.110:0/1771845452 shutdown_connections 2023-10-06T03:29:41.727 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.718+0000 7f3121f59700 1 --2- 172.21.15.110:0/1771845452 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f311c0cb540 0x7f311c0c7a90 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:41.727 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.718+0000 7f3121f59700 1 --2- 172.21.15.110:0/1771845452 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f311c0c98d0 0x7f311c0c9ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:41.727 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.718+0000 7f3121f59700 1 -- 172.21.15.110:0/1771845452 >> 172.21.15.110:0/1771845452 conn(0x7f311c0cc710 msgr2=0x7f311c0c8670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:29:41.727 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.718+0000 7f3121f59700 1 -- 172.21.15.110:0/1771845452 shutdown_connections 2023-10-06T03:29:41.727 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.718+0000 7f3121f59700 1 -- 172.21.15.110:0/1771845452 wait complete. 2023-10-06T03:29:41.727 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.722+0000 7f3121f59700 1 Processor -- start 2023-10-06T03:29:41.728 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.722+0000 7f3121f59700 1 -- start start 2023-10-06T03:29:41.729 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.722+0000 7f3121f59700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f311c0c98d0 0x7f311c0a4c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:41.729 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.722+0000 7f3121f59700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f311c0cb540 0x7f311c0c1a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:41.729 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.722+0000 7f3121f59700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f311c159d90 0x7f311c0c1f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:41.729 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.722+0000 7f3121f59700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f311c0c3b40 con 0x7f311c0c98d0 2023-10-06T03:29:41.729 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.722+0000 7f3121f59700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f311c0c4930 con 0x7f311c159d90 2023-10-06T03:29:41.730 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.722+0000 7f3121f59700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f311c0c4000 con 0x7f311c0cb540 2023-10-06T03:29:41.730 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.722+0000 7f311b7fe700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f311c0c98d0 0x7f311c0a4c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:41.730 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.722+0000 7f311b7fe700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f311c0c98d0 0x7f311c0a4c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:60298/0 (socket says 172.21.15.110:60298) 2023-10-06T03:29:41.730 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.722+0000 7f311b7fe700 1 -- 172.21.15.110:0/2848864128 learned_addr learned my addr 172.21.15.110:0/2848864128 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:29:41.730 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.722+0000 7f311affd700 1 --2- 172.21.15.110:0/2848864128 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f311c0cb540 0x7f311c0c1a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:41.730 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.722+0000 7f311bfff700 1 --2- 172.21.15.110:0/2848864128 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f311c159d90 0x7f311c0c1f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:41.730 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.722+0000 7f311b7fe700 1 -- 172.21.15.110:0/2848864128 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f311c0cb540 msgr2=0x7f311c0c1a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:41.730 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.722+0000 7f311b7fe700 1 --2- 172.21.15.110:0/2848864128 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f311c0cb540 0x7f311c0c1a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:41.730 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.722+0000 7f311b7fe700 1 -- 172.21.15.110:0/2848864128 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f311c159d90 msgr2=0x7f311c0c1f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:41.730 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.722+0000 7f311b7fe700 1 --2- 172.21.15.110:0/2848864128 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f311c159d90 0x7f311c0c1f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:41.731 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.722+0000 7f311b7fe700 1 -- 172.21.15.110:0/2848864128 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f310c0094f0 con 0x7f311c0c98d0 2023-10-06T03:29:41.731 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.722+0000 7f311affd700 1 --2- 172.21.15.110:0/2848864128 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f311c0cb540 0x7f311c0c1a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-06T03:29:41.732 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.722+0000 7f311b7fe700 1 --2- 172.21.15.110:0/2848864128 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f311c0c98d0 0x7f311c0a4c50 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f310c00ec50 tx=0x7f310c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:29:41.732 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.722+0000 7f31137fe700 1 -- 172.21.15.110:0/2848864128 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f310c00c850 con 0x7f311c0c98d0 2023-10-06T03:29:41.732 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.722+0000 7f31137fe700 1 -- 172.21.15.110:0/2848864128 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f310c00c9b0 con 0x7f311c0c98d0 2023-10-06T03:29:41.732 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.722+0000 7f3121f59700 1 -- 172.21.15.110:0/2848864128 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3104009a20 con 0x7f311c0c98d0 2023-10-06T03:29:41.732 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.722+0000 7f31137fe700 1 -- 172.21.15.110:0/2848864128 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f310c00cd30 con 0x7f311c0c98d0 2023-10-06T03:29:41.732 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.722+0000 7f3121f59700 1 -- 172.21.15.110:0/2848864128 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f311c0c45c0 con 0x7f311c0c98d0 2023-10-06T03:29:41.732 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.726+0000 7f31137fe700 1 -- 172.21.15.110:0/2848864128 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7f310c013680 con 0x7f311c0c98d0 2023-10-06T03:29:41.733 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.726+0000 7f31137fe700 1 --2- 172.21.15.110:0/2848864128 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f30fc060550 0x7f30fc062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:41.733 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.726+0000 7f31137fe700 1 -- 172.21.15.110:0/2848864128 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6809+0+0 (secure 0 0 0) 0x7f310c07cf50 con 0x7f311c0c98d0 2023-10-06T03:29:41.734 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.726+0000 7f311affd700 1 --2- 172.21.15.110:0/2848864128 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f30fc060550 0x7f30fc062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:41.734 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.726+0000 7f311affd700 1 --2- 172.21.15.110:0/2848864128 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f30fc060550 0x7f30fc062a00 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f311c0c2660 tx=0x7f3104019000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:29:41.735 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.730+0000 7f3124202740 1 Processor -- start 2023-10-06T03:29:41.735 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.730+0000 7f3124202740 1 -- start start 2023-10-06T03:29:41.738 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.730+0000 7f3124202740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x22b8ae0 0x253a450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:41.738 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.730+0000 7f3124202740 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x22b8710 0x253a990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:41.738 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.730+0000 7f3124202740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x2303790 0x253aed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:41.738 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.730+0000 7f3124202740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x2463f30 con 0x22b8ae0 2023-10-06T03:29:41.738 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.730+0000 7f3124202740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x245fed0 con 0x2303790 2023-10-06T03:29:41.738 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.730+0000 7f3124202740 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x24260b0 con 0x22b8710 2023-10-06T03:29:41.739 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.730+0000 7f311b7fe700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x22b8ae0 0x253a450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:41.739 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.730+0000 7f311b7fe700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x22b8ae0 0x253a450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:60306/0 (socket says 172.21.15.110:60306) 2023-10-06T03:29:41.739 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.730+0000 7f311b7fe700 1 -- 172.21.15.110:0/2333027956 learned_addr learned my addr 172.21.15.110:0/2333027956 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:29:41.739 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.730+0000 7f311affd700 1 --2- 172.21.15.110:0/2333027956 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x22b8710 0x253a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:41.739 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.730+0000 7f311bfff700 1 --2- 172.21.15.110:0/2333027956 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x2303790 0x253aed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:41.739 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.730+0000 7f311b7fe700 1 -- 172.21.15.110:0/2333027956 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x22b8710 msgr2=0x253a990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:29:41.739 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.730+0000 7f311b7fe700 1 --2- 172.21.15.110:0/2333027956 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x22b8710 0x253a990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:41.739 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.730+0000 7f311b7fe700 1 -- 172.21.15.110:0/2333027956 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x2303790 msgr2=0x253aed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:29:41.739 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.730+0000 7f311b7fe700 1 --2- 172.21.15.110:0/2333027956 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x2303790 0x253aed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:41.739 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.730+0000 7f311b7fe700 1 -- 172.21.15.110:0/2333027956 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x23049e0 con 0x22b8ae0 2023-10-06T03:29:41.739 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.730+0000 7f311b7fe700 1 --2- 172.21.15.110:0/2333027956 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x22b8ae0 0x253a450 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f310c050ca0 tx=0x7f310c07aee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b6d8af39bc574bb server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:29:41.740 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.730+0000 7f3110ff9700 1 -- 172.21.15.110:0/2333027956 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f310c0560f0 con 0x22b8ae0 2023-10-06T03:29:41.741 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.730+0000 7f3110ff9700 1 -- 172.21.15.110:0/2333027956 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f310c053fa0 con 0x22b8ae0 2023-10-06T03:29:41.741 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.730+0000 7f3110ff9700 1 -- 172.21.15.110:0/2333027956 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f310c0661c0 con 0x22b8ae0 2023-10-06T03:29:41.741 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.734+0000 7f3124202740 1 -- 172.21.15.110:0/2333027956 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x22b8ae0 msgr2=0x253a450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:41.741 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.734+0000 7f3124202740 1 --2- 172.21.15.110:0/2333027956 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x22b8ae0 0x253a450 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f310c050ca0 tx=0x7f310c07aee0 comp rx=0 tx=0).stop 2023-10-06T03:29:41.741 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.734+0000 7f3124202740 1 -- 172.21.15.110:0/2333027956 shutdown_connections 2023-10-06T03:29:41.741 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.734+0000 7f3124202740 1 --2- 172.21.15.110:0/2333027956 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x2303790 0x253aed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:41.741 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.734+0000 7f3124202740 1 --2- 172.21.15.110:0/2333027956 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x22b8710 0x253a990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:41.741 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.734+0000 7f3124202740 1 --2- 172.21.15.110:0/2333027956 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x22b8ae0 0x253a450 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:41.741 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.734+0000 7f3124202740 1 -- 172.21.15.110:0/2333027956 >> 172.21.15.110:0/2333027956 conn(0x22b8eb0 msgr2=0x24f2880 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:29:41.741 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.734+0000 7f3124202740 1 -- 172.21.15.110:0/2333027956 shutdown_connections 2023-10-06T03:29:41.742 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.734+0000 7f3124202740 1 -- 172.21.15.110:0/2333027956 wait complete. 2023-10-06T03:29:41.742 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.734+0000 7f3124202740 1 -- start start 2023-10-06T03:29:41.742 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.734+0000 7f3124202740 1 Processor -- start 2023-10-06T03:29:41.743 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.734+0000 7f3124202740 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x22b8710 0x251bc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:41.743 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.734+0000 7f3124202740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x22b8ae0 0x251c140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:41.743 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.734+0000 7f3124202740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x2303790 0x251c680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:41.743 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.734+0000 7f3124202740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x2425d50 con 0x22b8ae0 2023-10-06T03:29:41.743 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.734+0000 7f3124202740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x23fe970 con 0x2303790 2023-10-06T03:29:41.743 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.734+0000 7f3124202740 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x2426b00 con 0x22b8710 2023-10-06T03:29:41.744 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.734+0000 7f3124202740 10 client.0 _ll_register_callbacks cb 0x24a9b00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-06T03:29:41.744 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.734+0000 7f311affd700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x22b8ae0 0x251c140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:41.744 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.738+0000 7f311affd700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x22b8ae0 0x251c140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:60320/0 (socket says 172.21.15.110:60320) 2023-10-06T03:29:41.744 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.738+0000 7f311affd700 1 -- 172.21.15.110:0/250361896 learned_addr learned my addr 172.21.15.110:0/250361896 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:29:41.744 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.738+0000 7f311b7fe700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x22b8710 0x251bc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:41.744 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.738+0000 7f311bfff700 1 --2- 172.21.15.110:0/250361896 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x2303790 0x251c680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:41.746 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.738+0000 7f311affd700 1 -- 172.21.15.110:0/250361896 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x22b8710 msgr2=0x251bc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:29:41.746 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.738+0000 7f311affd700 1 --2- 172.21.15.110:0/250361896 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x22b8710 0x251bc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:41.746 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.738+0000 7f311affd700 1 -- 172.21.15.110:0/250361896 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x2303790 msgr2=0x251c680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:29:41.746 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.738+0000 7f311affd700 1 --2- 172.21.15.110:0/250361896 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x2303790 0x251c680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:41.746 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.738+0000 7f311affd700 1 -- 172.21.15.110:0/250361896 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f311c0c45c0 con 0x22b8ae0 2023-10-06T03:29:41.746 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.738+0000 7f311b7fe700 1 --2- 172.21.15.110:0/250361896 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x22b8710 0x251bc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-06T03:29:41.747 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.738+0000 7f311affd700 1 --2- 172.21.15.110:0/250361896 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x22b8ae0 0x251c140 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f310400b610 tx=0x7f3104022000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8067998f2188df97 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:29:41.747 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.738+0000 7f30f9ffb700 10 client.0 ms_handle_connect on v2:172.21.15.110:3300/0 2023-10-06T03:29:41.747 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.738+0000 7f30f9ffb700 1 -- 172.21.15.110:0/250361896 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3104031070 con 0x22b8ae0 2023-10-06T03:29:41.747 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.738+0000 7f3124202740 1 -- 172.21.15.110:0/250361896 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x23fecc0 con 0x22b8ae0 2023-10-06T03:29:41.747 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.738+0000 7f30f9ffb700 1 -- 172.21.15.110:0/250361896 <== mon.0 v2:172.21.15.110:3300/0 2 ==== osd_map(28..28 src has 1..28) v4 ==== 6809+0+0 (secure 0 0 0) 0x7f310402aac0 con 0x22b8ae0 2023-10-06T03:29:41.747 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.738+0000 7f30f9ffb700 1 -- 172.21.15.110:0/250361896 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_get_version_reply(handle=1 version=6) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3104022070 con 0x22b8ae0 2023-10-06T03:29:41.747 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.738+0000 7f3124202740 10 client.4425 fetch_fsmap learned FSMap version 6 2023-10-06T03:29:41.747 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.738+0000 7f3124202740 1 -- 172.21.15.110:0/250361896 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,fsmap=6,monmap=2+}) v3 -- 0x242e370 con 0x22b8ae0 2023-10-06T03:29:41.747 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.738+0000 7f30f9ffb700 1 -- 172.21.15.110:0/250361896 <== mon.0 v2:172.21.15.110:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x23fecc0 con 0x22b8ae0 2023-10-06T03:29:41.747 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.738+0000 7f30f9ffb700 1 -- 172.21.15.110:0/250361896 <== mon.0 v2:172.21.15.110:3300/0 5 ==== fsmap(e 6) v1 ==== 3108+0+0 (secure 0 0 0) 0x7f3104036400 con 0x22b8ae0 2023-10-06T03:29:41.747 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.738+0000 7f3124202740 10 client.4425 fetch_fsmap finished waiting for FSMap version 6 2023-10-06T03:29:41.748 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.738+0000 7f3124202740 10 client.4425 resolve_mds: resolved name 'a' to daemon mds.a 2023-10-06T03:29:41.748 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.738+0000 7f3124202740 20 client.4425 populate_metadata read hostname 'smithi110' 2023-10-06T03:29:41.748 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.738+0000 7f3124202740 1 --2- 172.21.15.110:0/250361896 >> [v2:172.21.15.110:6836/540320464,v1:172.21.15.110:6837/540320464] conn(0x2546ec0 0x25492a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:41.748 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.738+0000 7f3124202740 4 client.4425 mds_command: new command op to 4293 tid=0[{"prefix": "get_command_descriptions"}] 2023-10-06T03:29:41.748 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.738+0000 7f3124202740 1 -- 172.21.15.110:0/250361896 --> [v2:172.21.15.110:6836/540320464,v1:172.21.15.110:6837/540320464] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x24023e0 con 0x2546ec0 2023-10-06T03:29:41.748 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.738+0000 7f311b7fe700 1 --2- 172.21.15.110:0/250361896 >> [v2:172.21.15.110:6836/540320464,v1:172.21.15.110:6837/540320464] conn(0x2546ec0 0x25492a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:41.748 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.738+0000 7f30f9ffb700 10 client.4425 ms_handle_connect on v2:172.21.15.110:6836/540320464 2023-10-06T03:29:41.748 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.738+0000 7f311b7fe700 1 --2- 172.21.15.110:0/250361896 >> [v2:172.21.15.110:6836/540320464,v1:172.21.15.110:6837/540320464] conn(0x2546ec0 0x25492a0 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bdc445cc4a76a275 server_cookie=3a60ff6502bb4160 in_seq=0 out_seq=0 2023-10-06T03:29:41.749 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.738+0000 7f30f9ffb700 1 -- 172.21.15.110:0/250361896 <== mds.0 v2:172.21.15.110:6836/540320464 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+16954 (crc 0 0 0) 0x24023e0 con 0x2546ec0 2023-10-06T03:29:41.749 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.738+0000 7f30f9ffb700 10 client.4425 handle_command_reply: tid=0 2023-10-06T03:29:41.749 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.738+0000 7f3124202740 1 client.4425 shutdown 2023-10-06T03:29:41.750 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.742+0000 7f3124202740 1 -- 172.21.15.110:0/250361896 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x22b8ae0 msgr2=0x251c140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:41.750 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.742+0000 7f3124202740 1 --2- 172.21.15.110:0/250361896 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x22b8ae0 0x251c140 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f310400b610 tx=0x7f3104022000 comp rx=0 tx=0).stop 2023-10-06T03:29:41.750 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.742+0000 7f3124202740 1 -- 172.21.15.110:0/250361896 shutdown_connections 2023-10-06T03:29:41.750 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.742+0000 7f3124202740 1 --2- 172.21.15.110:0/250361896 >> [v2:172.21.15.110:6836/540320464,v1:172.21.15.110:6837/540320464] conn(0x2546ec0 0x25492a0 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:41.750 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.742+0000 7f3124202740 1 --2- 172.21.15.110:0/250361896 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x2303790 0x251c680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:41.750 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.742+0000 7f3124202740 1 --2- 172.21.15.110:0/250361896 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x22b8ae0 0x251c140 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:41.750 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.742+0000 7f30f9ffb700 0 client.4425 ms_handle_reset on v2:172.21.15.110:6836/540320464 2023-10-06T03:29:41.750 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.742+0000 7f3124202740 1 --2- 172.21.15.110:0/250361896 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x22b8710 0x251bc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:41.750 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.742+0000 7f3124202740 1 -- 172.21.15.110:0/250361896 >> 172.21.15.110:0/250361896 conn(0x22b8eb0 msgr2=0x24eb760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:29:41.750 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.742+0000 7f3124202740 1 -- 172.21.15.110:0/250361896 shutdown_connections 2023-10-06T03:29:41.751 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.742+0000 7f3124202740 1 -- 172.21.15.110:0/250361896 wait complete. 2023-10-06T03:29:41.751 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.742+0000 7f3124202740 20 client.4425 trim_cache size 0 max 16384 2023-10-06T03:29:41.764 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.758+0000 7f3124202740 1 Processor -- start 2023-10-06T03:29:41.764 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.758+0000 7f3124202740 1 -- start start 2023-10-06T03:29:41.765 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.758+0000 7f3124202740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x22b8710 0x251bda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:41.765 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.758+0000 7f3124202740 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x22b8ae0 0x251c2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:41.765 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.758+0000 7f3124202740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x2303790 0x24f1250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:41.765 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.758+0000 7f3124202740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x2462590 con 0x2303790 2023-10-06T03:29:41.765 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.758+0000 7f3124202740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x24cdbb0 con 0x22b8710 2023-10-06T03:29:41.765 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.758+0000 7f3124202740 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x245bb50 con 0x22b8ae0 2023-10-06T03:29:41.765 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.758+0000 7f311bfff700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x2303790 0x24f1250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:41.765 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.758+0000 7f311bfff700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x2303790 0x24f1250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:60322/0 (socket says 172.21.15.110:60322) 2023-10-06T03:29:41.765 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.758+0000 7f311affd700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x22b8ae0 0x251c2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:41.765 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.758+0000 7f311bfff700 1 -- 172.21.15.110:0/3777893465 learned_addr learned my addr 172.21.15.110:0/3777893465 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:29:41.766 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.758+0000 7f311b7fe700 1 --2- 172.21.15.110:0/3777893465 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x22b8710 0x251bda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:41.767 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.758+0000 7f311bfff700 1 -- 172.21.15.110:0/3777893465 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x22b8ae0 msgr2=0x251c2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:29:41.767 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.758+0000 7f311bfff700 1 --2- 172.21.15.110:0/3777893465 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x22b8ae0 0x251c2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:41.767 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.758+0000 7f311bfff700 1 -- 172.21.15.110:0/3777893465 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x22b8710 msgr2=0x251bda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:29:41.767 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.758+0000 7f311bfff700 1 --2- 172.21.15.110:0/3777893465 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x22b8710 0x251bda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:41.768 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.758+0000 7f311bfff700 1 -- 172.21.15.110:0/3777893465 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f311c154910 con 0x2303790 2023-10-06T03:29:41.768 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.758+0000 7f311b7fe700 1 --2- 172.21.15.110:0/3777893465 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x22b8710 0x251bda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-06T03:29:41.768 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.758+0000 7f311bfff700 1 --2- 172.21.15.110:0/3777893465 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x2303790 0x24f1250 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f3114003a20 tx=0x7f3114007cf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ad49e784e8e37fa server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:29:41.768 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.758+0000 7f31117fa700 1 -- 172.21.15.110:0/3777893465 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3114003720 con 0x2303790 2023-10-06T03:29:41.768 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.758+0000 7f31117fa700 1 -- 172.21.15.110:0/3777893465 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3114015a30 con 0x2303790 2023-10-06T03:29:41.768 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.758+0000 7f31117fa700 1 -- 172.21.15.110:0/3777893465 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3114015ba0 con 0x2303790 2023-10-06T03:29:41.768 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.762+0000 7f3124202740 1 -- 172.21.15.110:0/3777893465 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x2303790 msgr2=0x24f1250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:41.768 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.762+0000 7f3124202740 1 --2- 172.21.15.110:0/3777893465 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x2303790 0x24f1250 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f3114003a20 tx=0x7f3114007cf0 comp rx=0 tx=0).stop 2023-10-06T03:29:41.768 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.762+0000 7f3124202740 1 -- 172.21.15.110:0/3777893465 shutdown_connections 2023-10-06T03:29:41.768 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.762+0000 7f3124202740 1 --2- 172.21.15.110:0/3777893465 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x2303790 0x24f1250 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:41.768 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.762+0000 7f3124202740 1 --2- 172.21.15.110:0/3777893465 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x22b8ae0 0x251c2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:41.769 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.762+0000 7f3124202740 1 --2- 172.21.15.110:0/3777893465 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x22b8710 0x251bda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:41.769 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.762+0000 7f3124202740 1 -- 172.21.15.110:0/3777893465 >> 172.21.15.110:0/3777893465 conn(0x22b8eb0 msgr2=0x2543ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:29:41.769 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.762+0000 7f3124202740 1 -- 172.21.15.110:0/3777893465 shutdown_connections 2023-10-06T03:29:41.769 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.762+0000 7f3124202740 1 -- 172.21.15.110:0/3777893465 wait complete. 2023-10-06T03:29:41.769 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.762+0000 7f3124202740 1 -- start start 2023-10-06T03:29:41.769 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.762+0000 7f3124202740 1 Processor -- start 2023-10-06T03:29:41.771 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.762+0000 7f3124202740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x22b8710 0x2593cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:41.771 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.762+0000 7f3124202740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x22b8ae0 0x25941f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:41.771 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.762+0000 7f3124202740 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x2303790 0x2578320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:41.771 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.762+0000 7f3124202740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x2433040 con 0x22b8710 2023-10-06T03:29:41.771 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.762+0000 7f3124202740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x239c430 con 0x22b8ae0 2023-10-06T03:29:41.771 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.762+0000 7f3124202740 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x240bec0 con 0x2303790 2023-10-06T03:29:41.771 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.762+0000 7f311b7fe700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x22b8710 0x2593cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:41.771 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.762+0000 7f3124202740 10 client.0 _ll_register_callbacks cb 0x237c0e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-06T03:29:41.772 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.762+0000 7f311b7fe700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x22b8710 0x2593cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:60336/0 (socket says 172.21.15.110:60336) 2023-10-06T03:29:41.772 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.762+0000 7f311b7fe700 1 -- 172.21.15.110:0/1755383629 learned_addr learned my addr 172.21.15.110:0/1755383629 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:29:41.772 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.762+0000 7f311affd700 1 --2- 172.21.15.110:0/1755383629 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x22b8ae0 0x25941f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:41.772 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.762+0000 7f311bfff700 1 --2- 172.21.15.110:0/1755383629 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x2303790 0x2578320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:41.773 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.766+0000 7f311b7fe700 1 -- 172.21.15.110:0/1755383629 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x2303790 msgr2=0x2578320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:29:41.774 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.766+0000 7f311b7fe700 1 --2- 172.21.15.110:0/1755383629 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x2303790 0x2578320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:41.774 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.766+0000 7f311b7fe700 1 -- 172.21.15.110:0/1755383629 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x22b8ae0 msgr2=0x25941f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:29:41.774 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.766+0000 7f311b7fe700 1 --2- 172.21.15.110:0/1755383629 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x22b8ae0 0x25941f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:41.774 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.766+0000 7f311b7fe700 1 -- 172.21.15.110:0/1755383629 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3108003570 con 0x22b8710 2023-10-06T03:29:41.774 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.766+0000 7f311b7fe700 1 --2- 172.21.15.110:0/1755383629 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x22b8710 0x2593cb0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f310c050a10 tx=0x7f310c06a040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f3a30b40dc03c6d server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:29:41.774 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.766+0000 7f30f9ffb700 10 client.0 ms_handle_connect on v2:172.21.15.110:3300/0 2023-10-06T03:29:41.774 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.766+0000 7f30f9ffb700 1 -- 172.21.15.110:0/1755383629 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f310c07a040 con 0x22b8710 2023-10-06T03:29:41.774 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.766+0000 7f30f9ffb700 1 -- 172.21.15.110:0/1755383629 <== mon.0 v2:172.21.15.110:3300/0 2 ==== osd_map(28..28 src has 1..28) v4 ==== 6809+0+0 (secure 0 0 0) 0x7f310c056070 con 0x22b8710 2023-10-06T03:29:41.774 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.766+0000 7f3124202740 1 -- 172.21.15.110:0/1755383629 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f310c053fa0 con 0x22b8710 2023-10-06T03:29:41.774 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.766+0000 7f30f9ffb700 1 -- 172.21.15.110:0/1755383629 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_get_version_reply(handle=1 version=6) v2 ==== 24+0+0 (secure 0 0 0) 0x7f310c051450 con 0x22b8710 2023-10-06T03:29:41.775 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.766+0000 7f3124202740 10 client.4431 fetch_fsmap learned FSMap version 6 2023-10-06T03:29:41.775 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.766+0000 7f3124202740 1 -- 172.21.15.110:0/1755383629 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,fsmap=6,monmap=2+}) v3 -- 0x2414d80 con 0x22b8710 2023-10-06T03:29:41.775 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.766+0000 7f30f9ffb700 1 -- 172.21.15.110:0/1755383629 <== mon.0 v2:172.21.15.110:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f310c053fa0 con 0x22b8710 2023-10-06T03:29:41.775 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.766+0000 7f30f9ffb700 1 -- 172.21.15.110:0/1755383629 <== mon.0 v2:172.21.15.110:3300/0 5 ==== fsmap(e 6) v1 ==== 3108+0+0 (secure 0 0 0) 0x7f310c050200 con 0x22b8710 2023-10-06T03:29:41.775 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.766+0000 7f3124202740 10 client.4431 fetch_fsmap finished waiting for FSMap version 6 2023-10-06T03:29:41.775 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.766+0000 7f3124202740 10 client.4431 resolve_mds: resolved name 'a' to daemon mds.a 2023-10-06T03:29:41.775 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.766+0000 7f3124202740 20 client.4431 populate_metadata read hostname 'smithi110' 2023-10-06T03:29:41.775 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.766+0000 7f3124202740 1 --2- 172.21.15.110:0/1755383629 >> [v2:172.21.15.110:6836/540320464,v1:172.21.15.110:6837/540320464] conn(0x2578ce0 0x25790b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:41.775 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.766+0000 7f3124202740 4 client.4431 mds_command: new command op to 4293 tid=0[{"prefix": "status"}] 2023-10-06T03:29:41.775 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.766+0000 7f3124202740 1 -- 172.21.15.110:0/1755383629 --> [v2:172.21.15.110:6836/540320464,v1:172.21.15.110:6837/540320464] -- command(tid 0: {"prefix": "status"}) v1 -- 0x2434410 con 0x2578ce0 2023-10-06T03:29:41.775 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.766+0000 7f311affd700 1 --2- 172.21.15.110:0/1755383629 >> [v2:172.21.15.110:6836/540320464,v1:172.21.15.110:6837/540320464] conn(0x2578ce0 0x25790b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:41.775 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.766+0000 7f30f9ffb700 10 client.4431 ms_handle_connect on v2:172.21.15.110:6836/540320464 2023-10-06T03:29:41.776 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.766+0000 7f311affd700 1 --2- 172.21.15.110:0/1755383629 >> [v2:172.21.15.110:6836/540320464,v1:172.21.15.110:6837/540320464] conn(0x2578ce0 0x25790b0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ef077ea0edfac4dc server_cookie=65d78a823c304049 in_seq=0 out_seq=0 2023-10-06T03:29:41.777 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.766+0000 7f30f9ffb700 1 -- 172.21.15.110:0/1755383629 <== mds.0 v2:172.21.15.110:6836/540320464 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+324 (crc 0 0 0) 0x2434410 con 0x2578ce0 2023-10-06T03:29:41.777 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.766+0000 7f30f9ffb700 10 client.4431 handle_command_reply: tid=0 2023-10-06T03:29:41.777 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.766+0000 7f3124202740 1 client.4431 shutdown 2023-10-06T03:29:41.777 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.766+0000 7f3124202740 1 -- 172.21.15.110:0/1755383629 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x22b8710 msgr2=0x2593cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:41.777 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.766+0000 7f3124202740 1 --2- 172.21.15.110:0/1755383629 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x22b8710 0x2593cb0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f310c050a10 tx=0x7f310c06a040 comp rx=0 tx=0).stop 2023-10-06T03:29:41.777 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.766+0000 7f3124202740 1 -- 172.21.15.110:0/1755383629 shutdown_connections 2023-10-06T03:29:41.777 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.766+0000 7f3124202740 1 --2- 172.21.15.110:0/1755383629 >> [v2:172.21.15.110:6836/540320464,v1:172.21.15.110:6837/540320464] conn(0x2578ce0 0x25790b0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:41.777 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.766+0000 7f3124202740 1 --2- 172.21.15.110:0/1755383629 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x2303790 0x2578320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:41.777 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.766+0000 7f30f9ffb700 0 client.4431 ms_handle_reset on v2:172.21.15.110:6836/540320464 2023-10-06T03:29:41.777 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.766+0000 7f3124202740 1 --2- 172.21.15.110:0/1755383629 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x22b8ae0 0x25941f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:41.778 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.766+0000 7f3124202740 1 --2- 172.21.15.110:0/1755383629 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x22b8710 0x2593cb0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:41.778 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.766+0000 7f3124202740 1 -- 172.21.15.110:0/1755383629 >> 172.21.15.110:0/1755383629 conn(0x22b8eb0 msgr2=0x24f0240 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:29:41.778 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.766+0000 7f3124202740 1 -- 172.21.15.110:0/1755383629 shutdown_connections 2023-10-06T03:29:41.778 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.766+0000 7f3124202740 1 -- 172.21.15.110:0/1755383629 wait complete. 2023-10-06T03:29:41.778 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.770+0000 7f3124202740 20 client.4431 trim_cache size 0 max 16384 2023-10-06T03:29:41.778 INFO:teuthology.orchestra.run.smithi110.stdout:{ 2023-10-06T03:29:41.778 INFO:teuthology.orchestra.run.smithi110.stdout: "cluster_fsid": "93b8f6ac-4c92-4b45-b8e4-b59617024aa2", 2023-10-06T03:29:41.778 INFO:teuthology.orchestra.run.smithi110.stdout: "whoami": 0, 2023-10-06T03:29:41.779 INFO:teuthology.orchestra.run.smithi110.stdout: "id": 4293, 2023-10-06T03:29:41.779 INFO:teuthology.orchestra.run.smithi110.stdout: "want_state": "up:active", 2023-10-06T03:29:41.779 INFO:teuthology.orchestra.run.smithi110.stdout: "state": "up:active", 2023-10-06T03:29:41.779 INFO:teuthology.orchestra.run.smithi110.stdout: "fs_name": "cephfs", 2023-10-06T03:29:41.779 INFO:teuthology.orchestra.run.smithi110.stdout: "rank_uptime": 7.2564495490000001, 2023-10-06T03:29:41.779 INFO:teuthology.orchestra.run.smithi110.stdout: "mdsmap_epoch": 6, 2023-10-06T03:29:41.779 INFO:teuthology.orchestra.run.smithi110.stdout: "osdmap_epoch": 28, 2023-10-06T03:29:41.779 INFO:teuthology.orchestra.run.smithi110.stdout: "osdmap_epoch_barrier": 27, 2023-10-06T03:29:41.779 INFO:teuthology.orchestra.run.smithi110.stdout: "uptime": 12.288044614 2023-10-06T03:29:41.779 INFO:teuthology.orchestra.run.smithi110.stdout:} 2023-10-06T03:29:41.780 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.770+0000 7f3121f59700 1 -- 172.21.15.110:0/2848864128 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f30fc060550 msgr2=0x7f30fc062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:41.780 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.770+0000 7f3121f59700 1 --2- 172.21.15.110:0/2848864128 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f30fc060550 0x7f30fc062a00 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f311c0c2660 tx=0x7f3104019000 comp rx=0 tx=0).stop 2023-10-06T03:29:41.780 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.770+0000 7f3121f59700 1 -- 172.21.15.110:0/2848864128 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f311c0c98d0 msgr2=0x7f311c0a4c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:41.781 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.770+0000 7f3121f59700 1 --2- 172.21.15.110:0/2848864128 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f311c0c98d0 0x7f311c0a4c50 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f310c00ec50 tx=0x7f310c009250 comp rx=0 tx=0).stop 2023-10-06T03:29:41.781 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.770+0000 7f3121f59700 1 -- 172.21.15.110:0/2848864128 shutdown_connections 2023-10-06T03:29:41.781 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.770+0000 7f3121f59700 1 --2- 172.21.15.110:0/2848864128 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f30fc060550 0x7f30fc062a00 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:41.781 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.770+0000 7f3121f59700 1 --2- 172.21.15.110:0/2848864128 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f311c159d90 0x7f311c0c1f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:41.781 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.770+0000 7f3121f59700 1 --2- 172.21.15.110:0/2848864128 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f311c0cb540 0x7f311c0c1a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:41.781 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.770+0000 7f3121f59700 1 --2- 172.21.15.110:0/2848864128 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f311c0c98d0 0x7f311c0a4c50 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:41.781 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.770+0000 7f3121f59700 1 -- 172.21.15.110:0/2848864128 >> 172.21.15.110:0/2848864128 conn(0x7f311c0cc710 msgr2=0x7f311c0af080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:29:41.781 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.770+0000 7f3121f59700 1 -- 172.21.15.110:0/2848864128 shutdown_connections 2023-10-06T03:29:41.781 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.770+0000 7f3121f59700 1 -- 172.21.15.110:0/2848864128 wait complete. 2023-10-06T03:29:41.793 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-10-06T03:29:41.904 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.898+0000 7fd634ad6700 1 Processor -- start 2023-10-06T03:29:41.905 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.898+0000 7fd634ad6700 1 -- start start 2023-10-06T03:29:41.906 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.898+0000 7fd634ad6700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fd63009b670 0x7fd6300b37d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:41.906 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.898+0000 7fd634ad6700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6300b0810 con 0x7fd6300b1330 2023-10-06T03:29:41.906 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.898+0000 7fd634ad6700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd63009ba40 con 0x7fd6300ce9b0 2023-10-06T03:29:41.906 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.898+0000 7fd634ad6700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd6300af840 con 0x7fd63009b670 2023-10-06T03:29:41.906 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.898+0000 7fd62ed9d700 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7fd6300b1330 0x7fd630097aa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:37224/0 (socket says 172.21.15.110:37224) 2023-10-06T03:29:41.906 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.898+0000 7fd62ed9d700 1 -- 172.21.15.110:0/2631402604 learned_addr learned my addr 172.21.15.110:0/2631402604 (peer_addr_for_me v1:172.21.15.110:0/0) 2023-10-06T03:29:41.906 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.898+0000 7fd62e59c700 1 --2- 172.21.15.110:0/2631402604 >> v2:172.21.15.192:3300/0 conn(0x7fd63009b670 0x7fd6300b37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:41.908 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.898+0000 7fd62d59a700 1 -- 172.21.15.110:0/2631402604 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3519810363 0 0) 0x7fd6300b0810 con 0x7fd6300b1330 2023-10-06T03:29:41.908 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.898+0000 7fd62d59a700 1 -- 172.21.15.110:0/2631402604 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd61c003570 con 0x7fd6300b1330 2023-10-06T03:29:41.908 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.898+0000 7fd62e59c700 1 -- 172.21.15.110:0/2631402604 >> v1:172.21.15.192:6789/0 conn(0x7fd6300ce9b0 legacy=0x7fd6300b0c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:41.908 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.898+0000 7fd62e59c700 1 -- 172.21.15.110:0/2631402604 >> v1:172.21.15.110:6789/0 conn(0x7fd6300b1330 legacy=0x7fd630097aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:41.909 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.898+0000 7fd62e59c700 1 -- 172.21.15.110:0/2631402604 --> v2:172.21.15.192:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6300a8a90 con 0x7fd63009b670 2023-10-06T03:29:41.909 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.898+0000 7fd62e59c700 1 --2- 172.21.15.110:0/2631402604 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd63009b670 0x7fd6300b37d0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fd6180098e0 tx=0x7fd6180091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a960153d20a7158f server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:29:41.909 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.902+0000 7fd62d59a700 1 -- 172.21.15.110:0/2631402604 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd61800b6d0 con 0x7fd63009b670 2023-10-06T03:29:41.909 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.902+0000 7fd62d59a700 1 -- 172.21.15.110:0/2631402604 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd61800b830 con 0x7fd63009b670 2023-10-06T03:29:41.909 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.902+0000 7fd62d59a700 1 -- 172.21.15.110:0/2631402604 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd61800bbd0 con 0x7fd63009b670 2023-10-06T03:29:41.909 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.902+0000 7fd634ad6700 1 -- 172.21.15.110:0/2631402604 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd63009b670 msgr2=0x7fd6300b37d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:41.910 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.902+0000 7fd634ad6700 1 --2- 172.21.15.110:0/2631402604 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd63009b670 0x7fd6300b37d0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fd6180098e0 tx=0x7fd6180091a0 comp rx=0 tx=0).stop 2023-10-06T03:29:41.911 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.902+0000 7fd634ad6700 1 -- 172.21.15.110:0/2631402604 shutdown_connections 2023-10-06T03:29:41.911 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.902+0000 7fd634ad6700 1 --2- 172.21.15.110:0/2631402604 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd63009b670 0x7fd6300b37d0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:41.911 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.902+0000 7fd634ad6700 1 -- 172.21.15.110:0/2631402604 >> 172.21.15.110:0/2631402604 conn(0x7fd63006bfa0 msgr2=0x7fd6300cde60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:29:41.912 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.902+0000 7fd634ad6700 1 -- 172.21.15.110:0/2631402604 shutdown_connections 2023-10-06T03:29:41.912 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.902+0000 7fd634ad6700 1 -- 172.21.15.110:0/2631402604 wait complete. 2023-10-06T03:29:41.916 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.902+0000 7fd634ad6700 1 Processor -- start 2023-10-06T03:29:41.916 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.902+0000 7fd634ad6700 1 -- start start 2023-10-06T03:29:41.917 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.906+0000 7fd634ad6700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd63009b670 0x7fd630153200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:41.917 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.906+0000 7fd634ad6700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd6300b1330 0x7fd6300ad690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:41.917 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.906+0000 7fd634ad6700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd6300ce9b0 0x7fd6300adbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:41.917 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.906+0000 7fd634ad6700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd630169c40 con 0x7fd6300b1330 2023-10-06T03:29:41.917 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.906+0000 7fd634ad6700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6300a8600 con 0x7fd6300ce9b0 2023-10-06T03:29:41.917 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.906+0000 7fd634ad6700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6301690c0 con 0x7fd63009b670 2023-10-06T03:29:41.918 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.906+0000 7fd62dd9b700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd6300b1330 0x7fd6300ad690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:41.918 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.906+0000 7fd62dd9b700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd6300b1330 0x7fd6300ad690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:60348/0 (socket says 172.21.15.110:60348) 2023-10-06T03:29:41.918 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.906+0000 7fd62dd9b700 1 -- 172.21.15.110:0/545389121 learned_addr learned my addr 172.21.15.110:0/545389121 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:29:41.918 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.906+0000 7fd62e59c700 1 --2- 172.21.15.110:0/545389121 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd63009b670 0x7fd630153200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:41.918 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.906+0000 7fd62ed9d700 1 --2- 172.21.15.110:0/545389121 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd6300ce9b0 0x7fd6300adbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:41.918 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.906+0000 7fd62dd9b700 1 -- 172.21.15.110:0/545389121 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd63009b670 msgr2=0x7fd630153200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:41.918 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.906+0000 7fd62dd9b700 1 --2- 172.21.15.110:0/545389121 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd63009b670 0x7fd630153200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:41.919 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.906+0000 7fd62dd9b700 1 -- 172.21.15.110:0/545389121 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd6300ce9b0 msgr2=0x7fd6300adbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:41.919 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.906+0000 7fd62dd9b700 1 --2- 172.21.15.110:0/545389121 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd6300ce9b0 0x7fd6300adbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:41.919 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.906+0000 7fd62dd9b700 1 -- 172.21.15.110:0/545389121 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd618009420 con 0x7fd6300b1330 2023-10-06T03:29:41.919 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.906+0000 7fd62dd9b700 1 --2- 172.21.15.110:0/545389121 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd6300b1330 0x7fd6300ad690 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fd62800cb10 tx=0x7fd6280017b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:29:41.919 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.906+0000 7fd6267fc700 1 -- 172.21.15.110:0/545389121 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6280020c0 con 0x7fd6300b1330 2023-10-06T03:29:41.920 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.906+0000 7fd6267fc700 1 -- 172.21.15.110:0/545389121 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd62800a7e0 con 0x7fd6300b1330 2023-10-06T03:29:41.920 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.906+0000 7fd634ad6700 1 -- 172.21.15.110:0/545389121 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6301693a0 con 0x7fd6300b1330 2023-10-06T03:29:41.920 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.906+0000 7fd6267fc700 1 -- 172.21.15.110:0/545389121 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd62800ab80 con 0x7fd6300b1330 2023-10-06T03:29:41.920 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.906+0000 7fd634ad6700 1 -- 172.21.15.110:0/545389121 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd630169960 con 0x7fd6300b1330 2023-10-06T03:29:41.920 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.906+0000 7fd634ad6700 1 -- 172.21.15.110:0/545389121 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd63006c920 con 0x7fd6300b1330 2023-10-06T03:29:41.921 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.906+0000 7fd6267fc700 1 -- 172.21.15.110:0/545389121 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7fd628003030 con 0x7fd6300b1330 2023-10-06T03:29:41.921 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.910+0000 7fd6267fc700 1 --2- 172.21.15.110:0/545389121 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fd610060550 0x7fd610062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:41.921 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.910+0000 7fd6267fc700 1 -- 172.21.15.110:0/545389121 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6809+0+0 (secure 0 0 0) 0x7fd62807ccb0 con 0x7fd6300b1330 2023-10-06T03:29:41.921 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.910+0000 7fd62e59c700 1 --2- 172.21.15.110:0/545389121 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fd610060550 0x7fd610062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:41.921 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.910+0000 7fd62e59c700 1 --2- 172.21.15.110:0/545389121 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fd610060550 0x7fd610062a00 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fd6180093f0 tx=0x7fd618017000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:29:41.925 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:41.918+0000 7fd6267fc700 1 -- 172.21.15.110:0/545389121 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fd62804ce50 con 0x7fd6300b1330 2023-10-06T03:29:42.119 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:42.114+0000 7fd634ad6700 1 -- 172.21.15.110:0/545389121 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fd63006ff60 con 0x7fd6300b1330 2023-10-06T03:29:42.120 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:42.114+0000 7fd6267fc700 1 -- 172.21.15.110:0/545389121 <== mon.0 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v6) v1 ==== 53+0+88 (secure 0 0 0) 0x7fd628050470 con 0x7fd6300b1330 2023-10-06T03:29:42.120 INFO:teuthology.orchestra.run.smithi110.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2023-10-06T03:29:42.123 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:42.114+0000 7fd634ad6700 1 -- 172.21.15.110:0/545389121 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fd610060550 msgr2=0x7fd610062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:42.123 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:42.114+0000 7fd634ad6700 1 --2- 172.21.15.110:0/545389121 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fd610060550 0x7fd610062a00 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fd6180093f0 tx=0x7fd618017000 comp rx=0 tx=0).stop 2023-10-06T03:29:42.123 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:42.114+0000 7fd634ad6700 1 -- 172.21.15.110:0/545389121 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd6300b1330 msgr2=0x7fd6300ad690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:42.123 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:42.114+0000 7fd634ad6700 1 --2- 172.21.15.110:0/545389121 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd6300b1330 0x7fd6300ad690 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fd62800cb10 tx=0x7fd6280017b0 comp rx=0 tx=0).stop 2023-10-06T03:29:42.123 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:42.114+0000 7fd634ad6700 1 -- 172.21.15.110:0/545389121 shutdown_connections 2023-10-06T03:29:42.123 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:42.114+0000 7fd634ad6700 1 --2- 172.21.15.110:0/545389121 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fd610060550 0x7fd610062a00 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:42.123 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:42.114+0000 7fd634ad6700 1 --2- 172.21.15.110:0/545389121 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd6300ce9b0 0x7fd6300adbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:42.123 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:42.114+0000 7fd634ad6700 1 --2- 172.21.15.110:0/545389121 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd6300b1330 0x7fd6300ad690 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:42.123 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:42.114+0000 7fd634ad6700 1 --2- 172.21.15.110:0/545389121 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd63009b670 0x7fd630153200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:42.124 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:42.114+0000 7fd634ad6700 1 -- 172.21.15.110:0/545389121 >> 172.21.15.110:0/545389121 conn(0x7fd63006bfa0 msgr2=0x7fd63015d180 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:29:42.124 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:42.114+0000 7fd634ad6700 1 -- 172.21.15.110:0/545389121 shutdown_connections 2023-10-06T03:29:42.124 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:42.114+0000 7fd634ad6700 1 -- 172.21.15.110:0/545389121 wait complete. 2023-10-06T03:29:42.134 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2023-10-06T03:29:42.134 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-10-06T03:29:42.135 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi110.front.sepia.ceph.com 2023-10-06T03:29:42.135 INFO:tasks.cephfs.mount:self.client.name = client.admin 2023-10-06T03:29:42.135 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2023-10-06T03:29:42.135 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2023-10-06T03:29:42.135 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2023-10-06T03:29:42.135 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-10-06T03:29:42.135 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.admin' 2023-10-06T03:29:42.135 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-06T03:29:42.135 DEBUG:teuthology.orchestra.run.smithi110:> ip addr 2023-10-06T03:29:42.143 INFO:teuthology.orchestra.run.smithi110.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-10-06T03:29:42.143 INFO:teuthology.orchestra.run.smithi110.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-10-06T03:29:42.143 INFO:teuthology.orchestra.run.smithi110.stdout: inet 127.0.0.1/8 scope host lo 2023-10-06T03:29:42.143 INFO:teuthology.orchestra.run.smithi110.stdout: valid_lft forever preferred_lft forever 2023-10-06T03:29:42.143 INFO:teuthology.orchestra.run.smithi110.stdout: inet6 ::1/128 scope host 2023-10-06T03:29:42.143 INFO:teuthology.orchestra.run.smithi110.stdout: valid_lft forever preferred_lft forever 2023-10-06T03:29:42.144 INFO:teuthology.orchestra.run.smithi110.stdout:2: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2023-10-06T03:29:42.144 INFO:teuthology.orchestra.run.smithi110.stdout: link/ether 0c:c4:7a:d8:db:14 brd ff:ff:ff:ff:ff:ff 2023-10-06T03:29:42.144 INFO:teuthology.orchestra.run.smithi110.stdout:3: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000 2023-10-06T03:29:42.144 INFO:teuthology.orchestra.run.smithi110.stdout: link/ether 0c:c4:7a:d8:db:15 brd ff:ff:ff:ff:ff:ff 2023-10-06T03:29:42.144 INFO:teuthology.orchestra.run.smithi110.stdout:4: enp3s0f0: mtu 1500 qdisc noop state DOWN group default qlen 1000 2023-10-06T03:29:42.144 INFO:teuthology.orchestra.run.smithi110.stdout: link/ether 0c:c4:7a:8f:cc:34 brd ff:ff:ff:ff:ff:ff 2023-10-06T03:29:42.144 INFO:teuthology.orchestra.run.smithi110.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2023-10-06T03:29:42.144 INFO:teuthology.orchestra.run.smithi110.stdout: link/ether 0c:c4:7a:8f:cc:35 brd ff:ff:ff:ff:ff:ff 2023-10-06T03:29:42.144 INFO:teuthology.orchestra.run.smithi110.stdout: inet 172.21.15.110/20 brd 172.21.15.255 scope global dynamic enp3s0f1 2023-10-06T03:29:42.144 INFO:teuthology.orchestra.run.smithi110.stdout: valid_lft 42608sec preferred_lft 42608sec 2023-10-06T03:29:42.144 INFO:teuthology.orchestra.run.smithi110.stdout: inet6 fe80::ec4:7aff:fe8f:cc35/64 scope link 2023-10-06T03:29:42.144 INFO:teuthology.orchestra.run.smithi110.stdout: valid_lft forever preferred_lft forever 2023-10-06T03:29:42.145 INFO:teuthology.orchestra.run.smithi110.stdout:6: docker0: mtu 1500 qdisc noqueue state DOWN group default 2023-10-06T03:29:42.145 INFO:teuthology.orchestra.run.smithi110.stdout: link/ether 02:42:dc:de:aa:fb brd ff:ff:ff:ff:ff:ff 2023-10-06T03:29:42.145 INFO:teuthology.orchestra.run.smithi110.stdout: inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 2023-10-06T03:29:42.145 INFO:teuthology.orchestra.run.smithi110.stdout: valid_lft forever preferred_lft forever 2023-10-06T03:29:42.145 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2023-10-06T03:29:42.145 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-06T03:29:42.146 DEBUG:teuthology.orchestra.run.smithi110:> (cd / && exec bash -c ' 2023-10-06T03:29:42.146 DEBUG:teuthology.orchestra.run.smithi110:> set -e 2023-10-06T03:29:42.146 DEBUG:teuthology.orchestra.run.smithi110:> sudo ip link add name ceph-brx type bridge 2023-10-06T03:29:42.146 DEBUG:teuthology.orchestra.run.smithi110:> sudo ip addr flush dev ceph-brx 2023-10-06T03:29:42.146 DEBUG:teuthology.orchestra.run.smithi110:> sudo ip link set ceph-brx up 2023-10-06T03:29:42.146 DEBUG:teuthology.orchestra.run.smithi110:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2023-10-06T03:29:42.146 DEBUG:teuthology.orchestra.run.smithi110:> ') 2023-10-06T03:29:42.182 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-06T03:29:42.182 DEBUG:teuthology.orchestra.run.smithi110:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-10-06T03:29:42.190 INFO:teuthology.orchestra.run.smithi110.stdout:1 2023-10-06T03:29:42.191 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-06T03:29:42.192 DEBUG:teuthology.orchestra.run.smithi110:> route 2023-10-06T03:29:42.274 INFO:teuthology.orchestra.run.smithi110.stdout:Kernel IP routing table 2023-10-06T03:29:42.274 INFO:teuthology.orchestra.run.smithi110.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-10-06T03:29:42.274 INFO:teuthology.orchestra.run.smithi110.stdout:default _gateway 0.0.0.0 UG 0 0 0 enp3s0f1 2023-10-06T03:29:42.274 INFO:teuthology.orchestra.run.smithi110.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2023-10-06T03:29:42.275 INFO:teuthology.orchestra.run.smithi110.stdout:172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 2023-10-06T03:29:42.275 INFO:teuthology.orchestra.run.smithi110.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 0 0 0 enp3s0f1 2023-10-06T03:29:42.275 INFO:teuthology.orchestra.run.smithi110.stdout:_gateway 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2023-10-06T03:29:42.275 INFO:teuthology.orchestra.run.smithi110.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-10-06T03:29:42.275 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-06T03:29:42.276 DEBUG:teuthology.orchestra.run.smithi110:> (cd / && exec bash -c ' 2023-10-06T03:29:42.276 DEBUG:teuthology.orchestra.run.smithi110:> set -e 2023-10-06T03:29:42.276 DEBUG:teuthology.orchestra.run.smithi110:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2023-10-06T03:29:42.276 DEBUG:teuthology.orchestra.run.smithi110:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2023-10-06T03:29:42.276 DEBUG:teuthology.orchestra.run.smithi110:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2023-10-06T03:29:42.277 DEBUG:teuthology.orchestra.run.smithi110:> ') 2023-10-06T03:29:42.316 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-06T03:29:42.316 DEBUG:teuthology.orchestra.run.smithi110:> ip netns list 2023-10-06T03:29:42.366 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-06T03:29:42.366 DEBUG:teuthology.orchestra.run.smithi110:> ip netns list-id 2023-10-06T03:29:42.372 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-06T03:29:42.373 DEBUG:teuthology.orchestra.run.smithi110:> (cd / && exec bash -c ' 2023-10-06T03:29:42.373 DEBUG:teuthology.orchestra.run.smithi110:> set -e 2023-10-06T03:29:42.373 DEBUG:teuthology.orchestra.run.smithi110:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.admin 2023-10-06T03:29:42.374 DEBUG:teuthology.orchestra.run.smithi110:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.admin 0 2023-10-06T03:29:42.374 DEBUG:teuthology.orchestra.run.smithi110:> ') 2023-10-06T03:29:42.404 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.admin' with 192.168.0.1/16 2023-10-06T03:29:42.404 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-06T03:29:42.404 DEBUG:teuthology.orchestra.run.smithi110:> (cd / && exec bash -c ' 2023-10-06T03:29:42.404 DEBUG:teuthology.orchestra.run.smithi110:> set -e 2023-10-06T03:29:42.405 DEBUG:teuthology.orchestra.run.smithi110:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.admin type veth peer name brx.0 2023-10-06T03:29:42.405 DEBUG:teuthology.orchestra.run.smithi110:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2023-10-06T03:29:42.405 DEBUG:teuthology.orchestra.run.smithi110:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set veth0 up 2023-10-06T03:29:42.405 DEBUG:teuthology.orchestra.run.smithi110:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set lo up 2023-10-06T03:29:42.405 DEBUG:teuthology.orchestra.run.smithi110:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip route add default via 192.168.255.254 2023-10-06T03:29:42.405 DEBUG:teuthology.orchestra.run.smithi110:> ') 2023-10-06T03:29:42.526 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-06T03:29:42.527 DEBUG:teuthology.orchestra.run.smithi110:> (cd / && exec bash -c ' 2023-10-06T03:29:42.527 DEBUG:teuthology.orchestra.run.smithi110:> set -e 2023-10-06T03:29:42.527 DEBUG:teuthology.orchestra.run.smithi110:> sudo ip link set brx.0 up 2023-10-06T03:29:42.527 DEBUG:teuthology.orchestra.run.smithi110:> sudo ip link set dev brx.0 master ceph-brx 2023-10-06T03:29:42.527 DEBUG:teuthology.orchestra.run.smithi110:> ') 2023-10-06T03:29:42.556 INFO:tasks.cephfs.fuse_mount:Client client.admin config is {} 2023-10-06T03:29:42.557 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-06T03:29:42.557 DEBUG:teuthology.orchestra.run.smithi110:> mkdir -p -v /home/ubuntu/cephtest/mnt.admin 2023-10-06T03:29:42.561 INFO:teuthology.orchestra.run.smithi110.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.admin' 2023-10-06T03:29:42.561 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-06T03:29:42.561 DEBUG:teuthology.orchestra.run.smithi110:> chmod 0000 /home/ubuntu/cephtest/mnt.admin 2023-10-06T03:29:42.609 DEBUG:teuthology.orchestra.run.smithi110:> sudo modprobe fuse 2023-10-06T03:29:42.625 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-10-06T03:29:42.625 DEBUG:teuthology.orchestra.run.smithi110:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-10-06T03:29:42.635 INFO:teuthology.orchestra.run.smithi110.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-10-06T03:29:42.636 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-10-06T03:29:42.636 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-06T03:29:42.637 DEBUG:teuthology.orchestra.run.smithi110:> ls /sys/fs/fuse/connections 2023-10-06T03:29:42.642 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2023-10-06T03:29:42.642 DEBUG:teuthology.orchestra.run.smithi110:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.admin --id admin --client_fs=cephfs) 2023-10-06T03:29:42.644 DEBUG:teuthology.orchestra.run.smithi110:> sudo modprobe fuse 2023-10-06T03:29:42.659 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-10-06T03:29:42.659 DEBUG:teuthology.orchestra.run.smithi110:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-10-06T03:29:42.717 INFO:teuthology.orchestra.run.smithi110.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-10-06T03:29:42.718 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-10-06T03:29:42.719 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-06T03:29:42.719 DEBUG:teuthology.orchestra.run.smithi110:> ls /sys/fs/fuse/connections 2023-10-06T03:29:42.729 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi110.stderr:2023-10-06T03:29:42.726+0000 7fb88fb47380 -1 init, newargv = 0x56090c207490 newargc=15 2023-10-06T03:29:42.729 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi110.stderr:ceph-fuse[38312]: starting ceph client 2023-10-06T03:29:42.740 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi110.stderr:ceph-fuse[38312]: starting fuse 2023-10-06T03:29:43.824 DEBUG:teuthology.orchestra.run.smithi110:> sudo modprobe fuse 2023-10-06T03:29:43.843 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-10-06T03:29:43.843 DEBUG:teuthology.orchestra.run.smithi110:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-10-06T03:29:43.901 INFO:teuthology.orchestra.run.smithi110.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-10-06T03:29:43.901 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-10-06T03:29:43.902 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-06T03:29:43.902 DEBUG:teuthology.orchestra.run.smithi110:> ls /sys/fs/fuse/connections 2023-10-06T03:29:43.913 INFO:teuthology.orchestra.run.smithi110.stdout:52 2023-10-06T03:29:43.913 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [52] 2023-10-06T03:29:43.914 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2023-10-06T03:29:43.914 DEBUG:teuthology.orchestra.run.smithi110:> import glob 2023-10-06T03:29:43.914 DEBUG:teuthology.orchestra.run.smithi110:> import re 2023-10-06T03:29:43.914 DEBUG:teuthology.orchestra.run.smithi110:> import os 2023-10-06T03:29:43.915 DEBUG:teuthology.orchestra.run.smithi110:> import subprocess 2023-10-06T03:29:43.915 DEBUG:teuthology.orchestra.run.smithi110:> 2023-10-06T03:29:43.915 DEBUG:teuthology.orchestra.run.smithi110:> def _find_admin_socket(client_name): 2023-10-06T03:29:43.915 DEBUG:teuthology.orchestra.run.smithi110:> asok_path = "/var/run/ceph/ceph-client.admin.*.asok" 2023-10-06T03:29:43.915 DEBUG:teuthology.orchestra.run.smithi110:> files = glob.glob(asok_path) 2023-10-06T03:29:43.915 DEBUG:teuthology.orchestra.run.smithi110:> mountpoint = "/home/ubuntu/cephtest/mnt.admin" 2023-10-06T03:29:43.915 DEBUG:teuthology.orchestra.run.smithi110:> 2023-10-06T03:29:43.915 DEBUG:teuthology.orchestra.run.smithi110:> # Given a non-glob path, it better be there 2023-10-06T03:29:43.915 DEBUG:teuthology.orchestra.run.smithi110:> if "*" not in asok_path: 2023-10-06T03:29:43.915 DEBUG:teuthology.orchestra.run.smithi110:> assert(len(files) == 1) 2023-10-06T03:29:43.915 DEBUG:teuthology.orchestra.run.smithi110:> return files[0] 2023-10-06T03:29:43.915 DEBUG:teuthology.orchestra.run.smithi110:> 2023-10-06T03:29:43.915 DEBUG:teuthology.orchestra.run.smithi110:> for f in files: 2023-10-06T03:29:43.916 DEBUG:teuthology.orchestra.run.smithi110:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2023-10-06T03:29:43.916 DEBUG:teuthology.orchestra.run.smithi110:> if os.path.exists("/proc/{0}".format(pid)): 2023-10-06T03:29:43.916 DEBUG:teuthology.orchestra.run.smithi110:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2023-10-06T03:29:43.916 DEBUG:teuthology.orchestra.run.smithi110:> contents = proc_f.read() 2023-10-06T03:29:43.916 DEBUG:teuthology.orchestra.run.smithi110:> if mountpoint in contents: 2023-10-06T03:29:43.916 DEBUG:teuthology.orchestra.run.smithi110:> return f 2023-10-06T03:29:43.916 DEBUG:teuthology.orchestra.run.smithi110:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2023-10-06T03:29:43.916 DEBUG:teuthology.orchestra.run.smithi110:> 2023-10-06T03:29:43.916 DEBUG:teuthology.orchestra.run.smithi110:> print(_find_admin_socket("client.admin")) 2023-10-06T03:29:43.916 DEBUG:teuthology.orchestra.run.smithi110:> ' 2023-10-06T03:29:43.994 INFO:teuthology.orchestra.run.smithi110.stdout:/var/run/ceph/ceph-client.admin.38312.asok 2023-10-06T03:29:44.181 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.admin.38312.asok 2023-10-06T03:29:44.181 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-06T03:29:44.181 DEBUG:teuthology.orchestra.run.smithi110:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.admin.38312.asok status 2023-10-06T03:29:44.321 INFO:teuthology.orchestra.run.smithi110.stdout:{ 2023-10-06T03:29:44.322 INFO:teuthology.orchestra.run.smithi110.stdout: "metadata": { 2023-10-06T03:29:44.322 INFO:teuthology.orchestra.run.smithi110.stdout: "ceph_sha1": "736ac5ba72cb47af7d6f6666b1e100f7e20a20e2", 2023-10-06T03:29:44.322 INFO:teuthology.orchestra.run.smithi110.stdout: "ceph_version": "ceph version 17.2.6-1296-g736ac5ba (736ac5ba72cb47af7d6f6666b1e100f7e20a20e2) quincy (stable)", 2023-10-06T03:29:44.322 INFO:teuthology.orchestra.run.smithi110.stdout: "entity_id": "admin", 2023-10-06T03:29:44.322 INFO:teuthology.orchestra.run.smithi110.stdout: "hostname": "smithi110", 2023-10-06T03:29:44.322 INFO:teuthology.orchestra.run.smithi110.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.admin", 2023-10-06T03:29:44.322 INFO:teuthology.orchestra.run.smithi110.stdout: "pid": "38312", 2023-10-06T03:29:44.322 INFO:teuthology.orchestra.run.smithi110.stdout: "root": "/" 2023-10-06T03:29:44.322 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2023-10-06T03:29:44.323 INFO:teuthology.orchestra.run.smithi110.stdout: "dentry_count": 0, 2023-10-06T03:29:44.323 INFO:teuthology.orchestra.run.smithi110.stdout: "dentry_pinned_count": 0, 2023-10-06T03:29:44.323 INFO:teuthology.orchestra.run.smithi110.stdout: "id": 4446, 2023-10-06T03:29:44.323 INFO:teuthology.orchestra.run.smithi110.stdout: "inst": { 2023-10-06T03:29:44.323 INFO:teuthology.orchestra.run.smithi110.stdout: "name": { 2023-10-06T03:29:44.323 INFO:teuthology.orchestra.run.smithi110.stdout: "type": "client", 2023-10-06T03:29:44.323 INFO:teuthology.orchestra.run.smithi110.stdout: "num": 4446 2023-10-06T03:29:44.323 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2023-10-06T03:29:44.323 INFO:teuthology.orchestra.run.smithi110.stdout: "addr": { 2023-10-06T03:29:44.323 INFO:teuthology.orchestra.run.smithi110.stdout: "type": "v1", 2023-10-06T03:29:44.324 INFO:teuthology.orchestra.run.smithi110.stdout: "addr": "192.168.0.1:0", 2023-10-06T03:29:44.324 INFO:teuthology.orchestra.run.smithi110.stdout: "nonce": 1516923121 2023-10-06T03:29:44.324 INFO:teuthology.orchestra.run.smithi110.stdout: } 2023-10-06T03:29:44.324 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2023-10-06T03:29:44.324 INFO:teuthology.orchestra.run.smithi110.stdout: "addr": { 2023-10-06T03:29:44.324 INFO:teuthology.orchestra.run.smithi110.stdout: "type": "v1", 2023-10-06T03:29:44.324 INFO:teuthology.orchestra.run.smithi110.stdout: "addr": "192.168.0.1:0", 2023-10-06T03:29:44.324 INFO:teuthology.orchestra.run.smithi110.stdout: "nonce": 1516923121 2023-10-06T03:29:44.324 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2023-10-06T03:29:44.324 INFO:teuthology.orchestra.run.smithi110.stdout: "inst_str": "client.4446 192.168.0.1:0/1516923121", 2023-10-06T03:29:44.325 INFO:teuthology.orchestra.run.smithi110.stdout: "addr_str": "192.168.0.1:0/1516923121", 2023-10-06T03:29:44.325 INFO:teuthology.orchestra.run.smithi110.stdout: "inode_count": 1, 2023-10-06T03:29:44.325 INFO:teuthology.orchestra.run.smithi110.stdout: "mds_epoch": 6, 2023-10-06T03:29:44.325 INFO:teuthology.orchestra.run.smithi110.stdout: "osd_epoch": 28, 2023-10-06T03:29:44.325 INFO:teuthology.orchestra.run.smithi110.stdout: "osd_epoch_barrier": 0, 2023-10-06T03:29:44.325 INFO:teuthology.orchestra.run.smithi110.stdout: "blocklisted": false, 2023-10-06T03:29:44.325 INFO:teuthology.orchestra.run.smithi110.stdout: "fs_name": "cephfs" 2023-10-06T03:29:44.325 INFO:teuthology.orchestra.run.smithi110.stdout:} 2023-10-06T03:29:44.330 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-06T03:29:44.331 DEBUG:teuthology.orchestra.run.smithi110:> stat --file-system '--printf=%T 2023-10-06T03:29:44.331 DEBUG:teuthology.orchestra.run.smithi110:> ' -- /home/ubuntu/cephtest/mnt.admin 2023-10-06T03:29:44.337 INFO:teuthology.orchestra.run.smithi110.stdout:fuseblk 2023-10-06T03:29:44.337 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.admin 2023-10-06T03:29:44.337 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-06T03:29:44.337 DEBUG:teuthology.orchestra.run.smithi110:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.admin 2023-10-06T03:29:44.352 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-06T03:29:44.352 DEBUG:teuthology.orchestra.run.smithi110:> (cd /home/ubuntu/cephtest/mnt.admin && exec bash -c 'setfattr -n ceph.dir.layout.pool -v cephfs_data_ec . && getfattr -n ceph.dir.layout .') 2023-10-06T03:29:44.398 INFO:teuthology.orchestra.run.smithi110.stdout:# file: . 2023-10-06T03:29:44.399 INFO:teuthology.orchestra.run.smithi110.stdout:ceph.dir.layout="stripe_unit=4194304 stripe_count=1 object_size=4194304 pool=cephfs_data_ec" 2023-10-06T03:29:44.399 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-10-06T03:29:44.399 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi110.front.sepia.ceph.com... 2023-10-06T03:29:44.400 INFO:teuthology.orchestra.run:Running command with timeout 1800 2023-10-06T03:29:44.400 DEBUG:teuthology.orchestra.run.smithi110:> sudo fusermount -u /home/ubuntu/cephtest/mnt.admin 2023-10-06T03:29:44.425 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi110.stderr:ceph-fuse[38312]: fuse finished with error 0 and tester_r 0 2023-10-06T03:29:44.427 INFO:teuthology.orchestra.run:waiting for 900 2023-10-06T03:29:44.521 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi110.front.sepia.ceph.com 2023-10-06T03:29:44.521 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-06T03:29:44.521 DEBUG:teuthology.orchestra.run.smithi110:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.admin) 2023-10-06T03:29:44.535 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-06T03:29:44.644 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:44.642+0000 7f08a2120700 1 Processor -- start 2023-10-06T03:29:44.645 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:44.642+0000 7f08a2120700 1 -- start start 2023-10-06T03:29:44.645 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:44.642+0000 7f08a2120700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f089c159d30 0x7f089c0ca670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:44.645 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:44.642+0000 7f08a2120700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f089c0cb390 0x7f089c15a100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:44.645 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:44.642+0000 7f08a2120700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f089c15aaa0 con 0x7f089c15a640 2023-10-06T03:29:44.646 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:44.642+0000 7f08a2120700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f089c06bb90 con 0x7f089c159d30 2023-10-06T03:29:44.646 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:44.642+0000 7f08a2120700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f089c0cb830 con 0x7f089c0cb390 2023-10-06T03:29:44.646 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:44.642+0000 7f089bfff700 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f089c15a640 0x7f089c06aa60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:37228/0 (socket says 172.21.15.110:37228) 2023-10-06T03:29:44.646 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:44.642+0000 7f089bfff700 1 -- 172.21.15.110:0/2359504450 learned_addr learned my addr 172.21.15.110:0/2359504450 (peer_addr_for_me v1:172.21.15.110:0/0) 2023-10-06T03:29:44.648 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:44.642+0000 7f089b7fe700 1 --2- 172.21.15.110:0/2359504450 >> v2:172.21.15.192:3300/0 conn(0x7f089c159d30 0x7f089c0ca670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:44.648 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:44.642+0000 7f089affd700 1 --2- 172.21.15.110:0/2359504450 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f089c0cb390 0x7f089c15a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:44.649 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:44.642+0000 7f089a7fc700 1 -- 172.21.15.110:0/2359504450 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3006721385 0 0) 0x7f089c15aaa0 con 0x7f089c15a640 2023-10-06T03:29:44.649 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:44.642+0000 7f089a7fc700 1 -- 172.21.15.110:0/2359504450 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0888003570 con 0x7f089c15a640 2023-10-06T03:29:44.649 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:44.642+0000 7f089a7fc700 1 -- 172.21.15.110:0/2359504450 <== mon.0 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4070545752 0 0) 0x7f0888003570 con 0x7f089c15a640 2023-10-06T03:29:44.649 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:44.642+0000 7f089a7fc700 1 -- 172.21.15.110:0/2359504450 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f089c0cb390 msgr2=0x7f089c15a100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:29:44.649 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:44.642+0000 7f089a7fc700 1 --2- 172.21.15.110:0/2359504450 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f089c0cb390 0x7f089c15a100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:44.649 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:44.642+0000 7f089a7fc700 1 -- 172.21.15.110:0/2359504450 >> v2:172.21.15.192:3300/0 conn(0x7f089c159d30 msgr2=0x7f089c0ca670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:29:44.649 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:44.642+0000 7f089a7fc700 1 --2- 172.21.15.110:0/2359504450 >> v2:172.21.15.192:3300/0 conn(0x7f089c159d30 0x7f089c0ca670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:44.649 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:44.642+0000 7f089a7fc700 1 -- 172.21.15.110:0/2359504450 --> v1:172.21.15.110:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f089c0cba20 con 0x7f089c15a640 2023-10-06T03:29:44.649 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:44.642+0000 7f089a7fc700 1 -- 172.21.15.110:0/2359504450 <== mon.0 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 118603703 0 0) 0x7f0894001470 con 0x7f089c15a640 2023-10-06T03:29:44.649 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:44.642+0000 7f089a7fc700 1 -- 172.21.15.110:0/2359504450 >> v1:172.21.15.110:6789/0 conn(0x7f089c15a640 legacy=0x7f089c06aa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:44.649 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:44.642+0000 7f089a7fc700 1 --2- 172.21.15.110:0/2359504450 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0888003a60 0x7f0888003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:44.650 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:44.642+0000 7f089a7fc700 1 --2- 172.21.15.110:0/2359504450 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f089c0cb390 0x7f0888004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:44.650 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:44.642+0000 7f089a7fc700 1 --2- 172.21.15.110:0/2359504450 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0888004d50 0x7f08880051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:44.650 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:44.642+0000 7f089a7fc700 1 -- 172.21.15.110:0/2359504450 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f089c06bb90 con 0x7f0888003a60 2023-10-06T03:29:44.650 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:44.642+0000 7f089a7fc700 1 -- 172.21.15.110:0/2359504450 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f088800ba60 con 0x7f0888004d50 2023-10-06T03:29:44.650 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:44.642+0000 7f089a7fc700 1 -- 172.21.15.110:0/2359504450 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f088800bba0 con 0x7f089c0cb390 2023-10-06T03:29:44.650 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:44.642+0000 7f089b7fe700 1 --2- 172.21.15.110:0/2359504450 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0888003a60 0x7f0888003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:44.650 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:44.642+0000 7f089bfff700 1 --2- 172.21.15.110:0/2359504450 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0888004d50 0x7f08880051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:44.650 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:44.642+0000 7f089affd700 1 --2- 172.21.15.110:0/2359504450 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f089c0cb390 0x7f0888004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:44.650 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:44.642+0000 7f089b7fe700 1 -- 172.21.15.110:0/2359504450 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f089c0cb390 msgr2=0x7f0888004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:29:44.650 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:44.642+0000 7f089b7fe700 1 --2- 172.21.15.110:0/2359504450 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f089c0cb390 0x7f0888004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:44.650 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:44.642+0000 7f089b7fe700 1 -- 172.21.15.110:0/2359504450 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0888004d50 msgr2=0x7f08880051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:29:44.651 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:44.642+0000 7f089b7fe700 1 --2- 172.21.15.110:0/2359504450 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0888004d50 0x7f08880051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:44.651 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:44.642+0000 7f089b7fe700 1 -- 172.21.15.110:0/2359504450 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0888003570 con 0x7f0888003a60 2023-10-06T03:29:44.651 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:44.642+0000 7f089affd700 1 --2- 172.21.15.110:0/2359504450 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f089c0cb390 0x7f0888004810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-06T03:29:44.651 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:44.646+0000 7f089b7fe700 1 --2- 172.21.15.110:0/2359504450 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0888003a60 0x7f0888003e70 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f088c009c00 tx=0x7f088c0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe4b814724fecc26 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:29:44.651 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:44.646+0000 7f089a7fc700 1 -- 172.21.15.110:0/2359504450 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f088c00e6b0 con 0x7f0888003a60 2023-10-06T03:29:44.651 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:44.646+0000 7f089a7fc700 1 -- 172.21.15.110:0/2359504450 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f088c00e810 con 0x7f0888003a60 2023-10-06T03:29:44.651 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:44.646+0000 7f089a7fc700 1 -- 172.21.15.110:0/2359504450 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f088c00ebd0 con 0x7f0888003a60 2023-10-06T03:29:44.651 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:44.646+0000 7f08a2120700 1 -- 172.21.15.110:0/2359504450 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0888003a60 msgr2=0x7f0888003e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:44.651 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:44.646+0000 7f08a2120700 1 --2- 172.21.15.110:0/2359504450 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0888003a60 0x7f0888003e70 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f088c009c00 tx=0x7f088c0091a0 comp rx=0 tx=0).stop 2023-10-06T03:29:44.651 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:44.646+0000 7f089bfff700 1 -- 172.21.15.110:0/2359504450 reap_dead start 2023-10-06T03:29:44.651 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:44.646+0000 7f08a2120700 1 -- 172.21.15.110:0/2359504450 shutdown_connections 2023-10-06T03:29:44.652 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:44.646+0000 7f08a2120700 1 -- 172.21.15.110:0/2359504450 >> 172.21.15.110:0/2359504450 conn(0x7f089c0cbd00 msgr2=0x7f089c0cc100 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:29:44.652 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:44.646+0000 7f08a2120700 1 -- 172.21.15.110:0/2359504450 shutdown_connections 2023-10-06T03:29:44.652 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:44.646+0000 7f08a2120700 1 -- 172.21.15.110:0/2359504450 wait complete. 2023-10-06T03:29:44.652 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:44.650+0000 7f08a2120700 1 Processor -- start 2023-10-06T03:29:44.652 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:44.650+0000 7f08a2120700 1 -- start start 2023-10-06T03:29:44.655 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:44.650+0000 7f08a2120700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0888003a60 0x7f089c1406b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:44.655 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:44.650+0000 7f08a2120700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f089c140bf0 0x7f089c147e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:44.655 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:44.650+0000 7f08a2120700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f089c1483a0 0x7f089c0c3760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:44.656 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:44.650+0000 7f08a2120700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f089c148950 con 0x7f089c140bf0 2023-10-06T03:29:44.656 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:44.650+0000 7f08a2120700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f089c148c30 con 0x7f0888003a60 2023-10-06T03:29:44.656 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:44.650+0000 7f08a2120700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f089c148f10 con 0x7f089c1483a0 2023-10-06T03:29:44.656 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:44.650+0000 7f089affd700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f089c140bf0 0x7f089c147e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:44.656 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:44.650+0000 7f089affd700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f089c140bf0 0x7f089c147e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:60356/0 (socket says 172.21.15.110:60356) 2023-10-06T03:29:44.656 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:44.650+0000 7f089affd700 1 -- 172.21.15.110:0/253450518 learned_addr learned my addr 172.21.15.110:0/253450518 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:29:44.656 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:44.650+0000 7f089b7fe700 1 --2- 172.21.15.110:0/253450518 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0888003a60 0x7f089c1406b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:44.656 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:44.650+0000 7f089bfff700 1 --2- 172.21.15.110:0/253450518 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f089c1483a0 0x7f089c0c3760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:44.656 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:44.650+0000 7f089affd700 1 -- 172.21.15.110:0/253450518 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f089c1483a0 msgr2=0x7f089c0c3760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:44.656 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:44.650+0000 7f089affd700 1 --2- 172.21.15.110:0/253450518 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f089c1483a0 0x7f089c0c3760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:44.656 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:44.650+0000 7f089affd700 1 -- 172.21.15.110:0/253450518 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0888003a60 msgr2=0x7f089c1406b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:44.657 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:44.650+0000 7f089affd700 1 --2- 172.21.15.110:0/253450518 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0888003a60 0x7f089c1406b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:44.657 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:44.650+0000 7f089affd700 1 -- 172.21.15.110:0/253450518 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f089c0c3240 con 0x7f089c140bf0 2023-10-06T03:29:44.657 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:44.650+0000 7f089affd700 1 --2- 172.21.15.110:0/253450518 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f089c140bf0 0x7f089c147e60 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f088400f4d0 tx=0x7f088400b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:29:44.657 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:44.650+0000 7f08937fe700 1 -- 172.21.15.110:0/253450518 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f088401a070 con 0x7f089c140bf0 2023-10-06T03:29:44.657 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:44.650+0000 7f08937fe700 1 -- 172.21.15.110:0/253450518 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0884016420 con 0x7f089c140bf0 2023-10-06T03:29:44.657 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:44.654+0000 7f08a2120700 1 -- 172.21.15.110:0/253450518 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f089c145c60 con 0x7f089c140bf0 2023-10-06T03:29:44.657 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:44.654+0000 7f08937fe700 1 -- 172.21.15.110:0/253450518 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08840167e0 con 0x7f089c140bf0 2023-10-06T03:29:44.657 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:44.654+0000 7f08a2120700 1 -- 172.21.15.110:0/253450518 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f089c0c3ec0 con 0x7f089c140bf0 2023-10-06T03:29:44.658 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:44.654+0000 7f08a2120700 1 -- 172.21.15.110:0/253450518 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f089c149150 con 0x7f089c140bf0 2023-10-06T03:29:44.658 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:44.654+0000 7f08937fe700 1 -- 172.21.15.110:0/253450518 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7f0884003020 con 0x7f089c140bf0 2023-10-06T03:29:44.658 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:44.654+0000 7f08937fe700 1 --2- 172.21.15.110:0/253450518 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f087c060550 0x7f087c062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:44.658 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:44.654+0000 7f08937fe700 1 -- 172.21.15.110:0/253450518 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6809+0+0 (secure 0 0 0) 0x7f088407d870 con 0x7f089c140bf0 2023-10-06T03:29:44.659 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:44.654+0000 7f089b7fe700 1 --2- 172.21.15.110:0/253450518 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f087c060550 0x7f087c062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:44.659 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:44.654+0000 7f089b7fe700 1 --2- 172.21.15.110:0/253450518 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f087c060550 0x7f087c062a00 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f088c0091e0 tx=0x7f088c01a000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:29:44.665 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:44.662+0000 7f08937fe700 1 -- 172.21.15.110:0/253450518 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f089c149150 con 0x7f089c140bf0 2023-10-06T03:29:44.832 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:44.830+0000 7f08a2120700 1 -- 172.21.15.110:0/253450518 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f089c149150 con 0x7f089c140bf0 2023-10-06T03:29:44.834 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:44.830+0000 7f08937fe700 1 -- 172.21.15.110:0/253450518 <== mon.0 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v28) v1 ==== 74+0+18755 (secure 0 0 0) 0x7f089c149150 con 0x7f089c140bf0 2023-10-06T03:29:44.835 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-10-06T03:29:44.835 INFO:teuthology.orchestra.run.smithi110.stdout:{"epoch":28,"fsid":"93b8f6ac-4c92-4b45-b8e4-b59617024aa2","created":"2023-10-06T03:29:10.519583+0000","modified":"2023-10-06T03:29:38.951740+0000","last_up_change":"2023-10-06T03:29:21.960511+0000","last_in_change":"2023-10-06T03:29:12.560436+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-06T03:29:20.703743+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2023-10-06T03:29:28.307281+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2023-10-06T03:29:32.247251+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data","create_time":"2023-10-06T03:29:32.880086+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}}},{"pool":5,"pool_name":"cephfs_data_ec","create_time":"2023-10-06T03:29:36.613779+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"28","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"5866eb6c-2adc-46e4-be7a-e5bcda0f4065","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6824","nonce":33394},{"type":"v1","addr":"172.21.15.110:6825","nonce":33394}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6826","nonce":33394},{"type":"v1","addr":"172.21.15.110:6827","nonce":33394}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6830","nonce":33394},{"type":"v1","addr":"172.21.15.110:6831","nonce":33394}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6828","nonce":33394},{"type":"v1","addr":"172.21.15.110:6829","nonce":33394}]},"public_addr":"172.21.15.110:6825/33394","cluster_addr":"172.21.15.110:6827/33394","heartbeat_back_addr":"172.21.15.110:6831/33394","heartbeat_front_addr":"172.21.15.110:6829/33394","state":["exists","up"]},{"osd":1,"uuid":"a38de118-062b-4a05-9782-2ceddf6ff4df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6800","nonce":33396},{"type":"v1","addr":"172.21.15.110:6801","nonce":33396}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6802","nonce":33396},{"type":"v1","addr":"172.21.15.110:6803","nonce":33396}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6806","nonce":33396},{"type":"v1","addr":"172.21.15.110:6807","nonce":33396}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6804","nonce":33396},{"type":"v1","addr":"172.21.15.110:6805","nonce":33396}]},"public_addr":"172.21.15.110:6801/33396","cluster_addr":"172.21.15.110:6803/33396","heartbeat_back_addr":"172.21.15.110:6807/33396","heartbeat_front_addr":"172.21.15.110:6805/33396","state":["exists","up"]},{"osd":2,"uuid":"56e39be2-3407-4b69-877e-cf16617b06d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6816","nonce":33397},{"type":"v1","addr":"172.21.15.110:6817","nonce":33397}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6818","nonce":33397},{"type":"v1","addr":"172.21.15.110:6819","nonce":33397}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6822","nonce":33397},{"type":"v1","addr":"172.21.15.110:6823","nonce":33397}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6820","nonce":33397},{"type":"v1","addr":"172.21.15.110:6821","nonce":33397}]},"public_addr":"172.21.15.110:6817/33397","cluster_addr":"172.21.15.110:6819/33397","heartbeat_back_addr":"172.21.15.110:6823/33397","heartbeat_front_addr":"172.21.15.110:6821/33397","state":["exists","up"]},{"osd":3,"uuid":"01c82d13-c1d4-4bde-b55e-3e0089b40115","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6808","nonce":33395},{"type":"v1","addr":"172.21.15.110:6809","nonce":33395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6810","nonce":33395},{"type":"v1","addr":"172.21.15.110:6811","nonce":33395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6814","nonce":33395},{"type":"v1","addr":"172.21.15.110:6815","nonce":33395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6812","nonce":33395},{"type":"v1","addr":"172.21.15.110:6813","nonce":33395}]},"public_addr":"172.21.15.110:6809/33395","cluster_addr":"172.21.15.110:6811/33395","heartbeat_back_addr":"172.21.15.110:6815/33395","heartbeat_front_addr":"172.21.15.110:6813/33395","state":["exists","up"]},{"osd":4,"uuid":"86983430-5fa3-4d17-998f-03e2cd843237","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":33683},{"type":"v1","addr":"172.21.15.192:6817","nonce":33683}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":33683},{"type":"v1","addr":"172.21.15.192:6819","nonce":33683}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":33683},{"type":"v1","addr":"172.21.15.192:6823","nonce":33683}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6820","nonce":33683},{"type":"v1","addr":"172.21.15.192:6821","nonce":33683}]},"public_addr":"172.21.15.192:6817/33683","cluster_addr":"172.21.15.192:6819/33683","heartbeat_back_addr":"172.21.15.192:6823/33683","heartbeat_front_addr":"172.21.15.192:6821/33683","state":["exists","up"]},{"osd":5,"uuid":"6945257f-821d-4619-a63f-ebe857eba6f4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":33684},{"type":"v1","addr":"172.21.15.192:6801","nonce":33684}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":33684},{"type":"v1","addr":"172.21.15.192:6803","nonce":33684}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":33684},{"type":"v1","addr":"172.21.15.192:6807","nonce":33684}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6804","nonce":33684},{"type":"v1","addr":"172.21.15.192:6805","nonce":33684}]},"public_addr":"172.21.15.192:6801/33684","cluster_addr":"172.21.15.192:6803/33684","heartbeat_back_addr":"172.21.15.192:6807/33684","heartbeat_front_addr":"172.21.15.192:6805/33684","state":["exists","up"]},{"osd":6,"uuid":"7854d1ae-7060-4780-8f97-73cea5e7a12f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6824","nonce":33686},{"type":"v1","addr":"172.21.15.192:6825","nonce":33686}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":33686},{"type":"v1","addr":"172.21.15.192:6827","nonce":33686}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":33686},{"type":"v1","addr":"172.21.15.192:6831","nonce":33686}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6828","nonce":33686},{"type":"v1","addr":"172.21.15.192:6829","nonce":33686}]},"public_addr":"172.21.15.192:6825/33686","cluster_addr":"172.21.15.192:6827/33686","heartbeat_back_addr":"172.21.15.192:6831/33686","heartbeat_front_addr":"172.21.15.192:6829/33686","state":["exists","up"]},{"osd":7,"uuid":"f1ebdfd5-b859-482b-9f97-c18c4a26f445","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6808","nonce":33685},{"type":"v1","addr":"172.21.15.192:6809","nonce":33685}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":33685},{"type":"v1","addr":"172.21.15.192:6811","nonce":33685}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":33685},{"type":"v1","addr":"172.21.15.192:6815","nonce":33685}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6812","nonce":33685},{"type":"v1","addr":"172.21.15.192:6813","nonce":33685}]},"public_addr":"172.21.15.192:6809/33685","cluster_addr":"172.21.15.192:6811/33685","heartbeat_back_addr":"172.21.15.192:6815/33685","heartbeat_front_addr":"172.21.15.192:6813/33685","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T03:29:18.002048+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T03:29:17.478576+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T03:29:18.034944+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T03:29:17.628513+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T03:29:18.047338+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T03:29:17.772349+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T03:29:17.883958+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T03:29:18.151838+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-06T03:29:44.836 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:44.834+0000 7f08a2120700 1 -- 172.21.15.110:0/253450518 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f087c060550 msgr2=0x7f087c062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:44.837 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:44.834+0000 7f08a2120700 1 --2- 172.21.15.110:0/253450518 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f087c060550 0x7f087c062a00 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f088c0091e0 tx=0x7f088c01a000 comp rx=0 tx=0).stop 2023-10-06T03:29:44.837 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:44.834+0000 7f08a2120700 1 -- 172.21.15.110:0/253450518 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f089c140bf0 msgr2=0x7f089c147e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:44.837 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:44.834+0000 7f08a2120700 1 --2- 172.21.15.110:0/253450518 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f089c140bf0 0x7f089c147e60 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f088400f4d0 tx=0x7f088400b040 comp rx=0 tx=0).stop 2023-10-06T03:29:44.837 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:44.834+0000 7f08a2120700 1 -- 172.21.15.110:0/253450518 shutdown_connections 2023-10-06T03:29:44.837 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:44.834+0000 7f08a2120700 1 --2- 172.21.15.110:0/253450518 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f087c060550 0x7f087c062a00 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:44.837 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:44.834+0000 7f08a2120700 1 --2- 172.21.15.110:0/253450518 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f089c1483a0 0x7f089c0c3760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:44.837 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:44.834+0000 7f08a2120700 1 --2- 172.21.15.110:0/253450518 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f089c140bf0 0x7f089c147e60 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:44.837 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:44.834+0000 7f08a2120700 1 --2- 172.21.15.110:0/253450518 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0888003a60 0x7f089c1406b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:44.837 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:44.834+0000 7f08a2120700 1 -- 172.21.15.110:0/253450518 >> 172.21.15.110:0/253450518 conn(0x7f089c0cbd00 msgr2=0x7f089c0c8410 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:29:44.838 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:44.834+0000 7f08a2120700 1 -- 172.21.15.110:0/253450518 shutdown_connections 2023-10-06T03:29:44.838 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:44.834+0000 7f08a2120700 1 -- 172.21.15.110:0/253450518 wait complete. 2023-10-06T03:29:44.849 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-06T03:29:44.956 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:44.954+0000 7fce8d495700 1 Processor -- start 2023-10-06T03:29:44.956 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:44.954+0000 7fce8d495700 1 -- start start 2023-10-06T03:29:44.958 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:44.954+0000 7fce8d495700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fce880adac0 0x7fce880ade90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:44.958 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:44.954+0000 7fce8d495700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fce880c9bb0 0x7fce880caa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:44.958 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:44.954+0000 7fce8d495700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fce880aec00 0x7fce880c6540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:44.958 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:44.954+0000 7fce8d495700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fce8806c190 con 0x7fce880c9bb0 2023-10-06T03:29:44.958 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:44.954+0000 7fce8d495700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fce880cb2b0 con 0x7fce880adac0 2023-10-06T03:29:44.958 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:44.954+0000 7fce8d495700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce8806b860 con 0x7fce880aec00 2023-10-06T03:29:44.958 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:44.954+0000 7fce877fe700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fce880c9bb0 0x7fce880caa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:44.958 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:44.954+0000 7fce877fe700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fce880c9bb0 0x7fce880caa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:60366/0 (socket says 172.21.15.110:60366) 2023-10-06T03:29:44.959 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:44.954+0000 7fce877fe700 1 -- 172.21.15.110:0/1044238260 learned_addr learned my addr 172.21.15.110:0/1044238260 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:29:44.959 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:44.954+0000 7fce877fe700 1 -- 172.21.15.110:0/1044238260 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fce880aec00 msgr2=0x7fce880c6540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:29:44.959 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:44.954+0000 7fce877fe700 1 --2- 172.21.15.110:0/1044238260 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fce880aec00 0x7fce880c6540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:44.959 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:44.954+0000 7fce877fe700 1 -- 172.21.15.110:0/1044238260 >> v2:172.21.15.192:3300/0 conn(0x7fce880adac0 msgr2=0x7fce880ade90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:29:44.959 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:44.954+0000 7fce877fe700 1 --2- 172.21.15.110:0/1044238260 >> v2:172.21.15.192:3300/0 conn(0x7fce880adac0 0x7fce880ade90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:44.959 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:44.954+0000 7fce877fe700 1 -- 172.21.15.110:0/1044238260 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce8806bb40 con 0x7fce880c9bb0 2023-10-06T03:29:44.959 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:44.954+0000 7fce877fe700 1 --2- 172.21.15.110:0/1044238260 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fce880c9bb0 0x7fce880caa10 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fce880c9650 tx=0x7fce80009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1832ddcd06345ebb server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:29:44.959 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:44.954+0000 7fce86ffd700 1 -- 172.21.15.110:0/1044238260 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce8000e070 con 0x7fce880c9bb0 2023-10-06T03:29:44.959 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:44.954+0000 7fce86ffd700 1 -- 172.21.15.110:0/1044238260 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce8000ba50 con 0x7fce880c9bb0 2023-10-06T03:29:44.959 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:44.954+0000 7fce86ffd700 1 -- 172.21.15.110:0/1044238260 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce800092e0 con 0x7fce880c9bb0 2023-10-06T03:29:44.959 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:44.954+0000 7fce8d495700 1 -- 172.21.15.110:0/1044238260 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fce880c9bb0 msgr2=0x7fce880caa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:44.960 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:44.954+0000 7fce8d495700 1 --2- 172.21.15.110:0/1044238260 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fce880c9bb0 0x7fce880caa10 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fce880c9650 tx=0x7fce80009840 comp rx=0 tx=0).stop 2023-10-06T03:29:44.960 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:44.954+0000 7fce8d495700 1 -- 172.21.15.110:0/1044238260 shutdown_connections 2023-10-06T03:29:44.960 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:44.954+0000 7fce8d495700 1 --2- 172.21.15.110:0/1044238260 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fce880aec00 0x7fce880c6540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:44.960 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:44.954+0000 7fce8d495700 1 --2- 172.21.15.110:0/1044238260 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fce880c9bb0 0x7fce880caa10 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:44.960 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:44.954+0000 7fce8d495700 1 --2- 172.21.15.110:0/1044238260 >> v2:172.21.15.192:3300/0 conn(0x7fce880adac0 0x7fce880ade90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:44.960 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:44.954+0000 7fce8d495700 1 -- 172.21.15.110:0/1044238260 >> 172.21.15.110:0/1044238260 conn(0x7fce880cc470 msgr2=0x7fce880c8410 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:29:44.960 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:44.954+0000 7fce8d495700 1 -- 172.21.15.110:0/1044238260 shutdown_connections 2023-10-06T03:29:44.960 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:44.954+0000 7fce8d495700 1 -- 172.21.15.110:0/1044238260 wait complete. 2023-10-06T03:29:44.962 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:44.954+0000 7fce8d495700 1 Processor -- start 2023-10-06T03:29:44.962 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:44.958+0000 7fce8d495700 1 -- start start 2023-10-06T03:29:44.962 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:44.958+0000 7fce8d495700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fce880adac0 0x7fce88157430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:44.962 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:44.958+0000 7fce8d495700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fce880aec00 0x7fce880af810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:44.962 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:44.958+0000 7fce8d495700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fce880c9bb0 0x7fce880afd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:44.962 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:44.958+0000 7fce8d495700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce88157970 con 0x7fce880adac0 2023-10-06T03:29:44.962 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:44.958+0000 7fce8d495700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce88170db0 con 0x7fce880aec00 2023-10-06T03:29:44.962 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:44.958+0000 7fce8d495700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce88171000 con 0x7fce880c9bb0 2023-10-06T03:29:44.963 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:44.958+0000 7fce87fff700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fce880adac0 0x7fce88157430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:44.963 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:44.958+0000 7fce87fff700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fce880adac0 0x7fce88157430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:60368/0 (socket says 172.21.15.110:60368) 2023-10-06T03:29:44.963 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:44.958+0000 7fce87fff700 1 -- 172.21.15.110:0/1254178413 learned_addr learned my addr 172.21.15.110:0/1254178413 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:29:44.963 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:44.958+0000 7fce877fe700 1 --2- 172.21.15.110:0/1254178413 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fce880aec00 0x7fce880af810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:44.963 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:44.958+0000 7fce8cc94700 1 --2- 172.21.15.110:0/1254178413 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fce880c9bb0 0x7fce880afd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:44.963 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:44.958+0000 7fce87fff700 1 -- 172.21.15.110:0/1254178413 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fce880c9bb0 msgr2=0x7fce880afd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:44.963 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:44.958+0000 7fce87fff700 1 --2- 172.21.15.110:0/1254178413 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fce880c9bb0 0x7fce880afd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:44.963 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:44.958+0000 7fce87fff700 1 -- 172.21.15.110:0/1254178413 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fce880aec00 msgr2=0x7fce880af810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:44.963 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:44.958+0000 7fce87fff700 1 --2- 172.21.15.110:0/1254178413 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fce880aec00 0x7fce880af810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:44.963 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:44.958+0000 7fce87fff700 1 -- 172.21.15.110:0/1254178413 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce80009ae0 con 0x7fce880adac0 2023-10-06T03:29:44.963 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:44.958+0000 7fce87fff700 1 --2- 172.21.15.110:0/1254178413 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fce880adac0 0x7fce88157430 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fce7800b810 tx=0x7fce78009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:29:44.964 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:44.958+0000 7fce6ffff700 1 -- 172.21.15.110:0/1254178413 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce7800f510 con 0x7fce880adac0 2023-10-06T03:29:44.964 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:44.958+0000 7fce6ffff700 1 -- 172.21.15.110:0/1254178413 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce7800f670 con 0x7fce880adac0 2023-10-06T03:29:44.964 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:44.958+0000 7fce6ffff700 1 -- 172.21.15.110:0/1254178413 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce7800fa30 con 0x7fce880adac0 2023-10-06T03:29:44.964 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:44.958+0000 7fce8d495700 1 -- 172.21.15.110:0/1254178413 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce881712e0 con 0x7fce880adac0 2023-10-06T03:29:44.964 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:44.958+0000 7fce8d495700 1 -- 172.21.15.110:0/1254178413 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce881718a0 con 0x7fce880adac0 2023-10-06T03:29:44.965 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:44.958+0000 7fce8d495700 1 -- 172.21.15.110:0/1254178413 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce88171a90 con 0x7fce880adac0 2023-10-06T03:29:44.965 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:44.958+0000 7fce6ffff700 1 -- 172.21.15.110:0/1254178413 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7fce7801a400 con 0x7fce880adac0 2023-10-06T03:29:44.965 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:44.962+0000 7fce6ffff700 1 --2- 172.21.15.110:0/1254178413 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fce74060550 0x7fce74062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:44.965 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:44.962+0000 7fce6ffff700 1 -- 172.21.15.110:0/1254178413 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6809+0+0 (secure 0 0 0) 0x7fce7807c6f0 con 0x7fce880adac0 2023-10-06T03:29:44.965 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:44.962+0000 7fce877fe700 1 --2- 172.21.15.110:0/1254178413 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fce74060550 0x7fce74062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:44.965 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:44.962+0000 7fce877fe700 1 --2- 172.21.15.110:0/1254178413 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fce74060550 0x7fce74062a00 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fce80009880 tx=0x7fce80018000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:29:44.969 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:44.966+0000 7fce6ffff700 1 -- 172.21.15.110:0/1254178413 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fce7804c910 con 0x7fce880adac0 2023-10-06T03:29:45.129 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:45.126+0000 7fce8d495700 1 -- 172.21.15.110:0/1254178413 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fce88070ff0 con 0x7fce880adac0 2023-10-06T03:29:45.130 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:45.126+0000 7fce6ffff700 1 -- 172.21.15.110:0/1254178413 <== mon.0 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v28) v1 ==== 74+0+18755 (secure 0 0 0) 0x7fce7804ff30 con 0x7fce880adac0 2023-10-06T03:29:45.130 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-10-06T03:29:45.130 INFO:teuthology.orchestra.run.smithi110.stdout:{"epoch":28,"fsid":"93b8f6ac-4c92-4b45-b8e4-b59617024aa2","created":"2023-10-06T03:29:10.519583+0000","modified":"2023-10-06T03:29:38.951740+0000","last_up_change":"2023-10-06T03:29:21.960511+0000","last_in_change":"2023-10-06T03:29:12.560436+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-06T03:29:20.703743+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2023-10-06T03:29:28.307281+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2023-10-06T03:29:32.247251+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data","create_time":"2023-10-06T03:29:32.880086+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}}},{"pool":5,"pool_name":"cephfs_data_ec","create_time":"2023-10-06T03:29:36.613779+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"28","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"5866eb6c-2adc-46e4-be7a-e5bcda0f4065","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6824","nonce":33394},{"type":"v1","addr":"172.21.15.110:6825","nonce":33394}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6826","nonce":33394},{"type":"v1","addr":"172.21.15.110:6827","nonce":33394}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6830","nonce":33394},{"type":"v1","addr":"172.21.15.110:6831","nonce":33394}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6828","nonce":33394},{"type":"v1","addr":"172.21.15.110:6829","nonce":33394}]},"public_addr":"172.21.15.110:6825/33394","cluster_addr":"172.21.15.110:6827/33394","heartbeat_back_addr":"172.21.15.110:6831/33394","heartbeat_front_addr":"172.21.15.110:6829/33394","state":["exists","up"]},{"osd":1,"uuid":"a38de118-062b-4a05-9782-2ceddf6ff4df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6800","nonce":33396},{"type":"v1","addr":"172.21.15.110:6801","nonce":33396}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6802","nonce":33396},{"type":"v1","addr":"172.21.15.110:6803","nonce":33396}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6806","nonce":33396},{"type":"v1","addr":"172.21.15.110:6807","nonce":33396}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6804","nonce":33396},{"type":"v1","addr":"172.21.15.110:6805","nonce":33396}]},"public_addr":"172.21.15.110:6801/33396","cluster_addr":"172.21.15.110:6803/33396","heartbeat_back_addr":"172.21.15.110:6807/33396","heartbeat_front_addr":"172.21.15.110:6805/33396","state":["exists","up"]},{"osd":2,"uuid":"56e39be2-3407-4b69-877e-cf16617b06d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6816","nonce":33397},{"type":"v1","addr":"172.21.15.110:6817","nonce":33397}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6818","nonce":33397},{"type":"v1","addr":"172.21.15.110:6819","nonce":33397}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6822","nonce":33397},{"type":"v1","addr":"172.21.15.110:6823","nonce":33397}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6820","nonce":33397},{"type":"v1","addr":"172.21.15.110:6821","nonce":33397}]},"public_addr":"172.21.15.110:6817/33397","cluster_addr":"172.21.15.110:6819/33397","heartbeat_back_addr":"172.21.15.110:6823/33397","heartbeat_front_addr":"172.21.15.110:6821/33397","state":["exists","up"]},{"osd":3,"uuid":"01c82d13-c1d4-4bde-b55e-3e0089b40115","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6808","nonce":33395},{"type":"v1","addr":"172.21.15.110:6809","nonce":33395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6810","nonce":33395},{"type":"v1","addr":"172.21.15.110:6811","nonce":33395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6814","nonce":33395},{"type":"v1","addr":"172.21.15.110:6815","nonce":33395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6812","nonce":33395},{"type":"v1","addr":"172.21.15.110:6813","nonce":33395}]},"public_addr":"172.21.15.110:6809/33395","cluster_addr":"172.21.15.110:6811/33395","heartbeat_back_addr":"172.21.15.110:6815/33395","heartbeat_front_addr":"172.21.15.110:6813/33395","state":["exists","up"]},{"osd":4,"uuid":"86983430-5fa3-4d17-998f-03e2cd843237","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":33683},{"type":"v1","addr":"172.21.15.192:6817","nonce":33683}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":33683},{"type":"v1","addr":"172.21.15.192:6819","nonce":33683}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":33683},{"type":"v1","addr":"172.21.15.192:6823","nonce":33683}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6820","nonce":33683},{"type":"v1","addr":"172.21.15.192:6821","nonce":33683}]},"public_addr":"172.21.15.192:6817/33683","cluster_addr":"172.21.15.192:6819/33683","heartbeat_back_addr":"172.21.15.192:6823/33683","heartbeat_front_addr":"172.21.15.192:6821/33683","state":["exists","up"]},{"osd":5,"uuid":"6945257f-821d-4619-a63f-ebe857eba6f4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":33684},{"type":"v1","addr":"172.21.15.192:6801","nonce":33684}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":33684},{"type":"v1","addr":"172.21.15.192:6803","nonce":33684}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":33684},{"type":"v1","addr":"172.21.15.192:6807","nonce":33684}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6804","nonce":33684},{"type":"v1","addr":"172.21.15.192:6805","nonce":33684}]},"public_addr":"172.21.15.192:6801/33684","cluster_addr":"172.21.15.192:6803/33684","heartbeat_back_addr":"172.21.15.192:6807/33684","heartbeat_front_addr":"172.21.15.192:6805/33684","state":["exists","up"]},{"osd":6,"uuid":"7854d1ae-7060-4780-8f97-73cea5e7a12f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6824","nonce":33686},{"type":"v1","addr":"172.21.15.192:6825","nonce":33686}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":33686},{"type":"v1","addr":"172.21.15.192:6827","nonce":33686}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":33686},{"type":"v1","addr":"172.21.15.192:6831","nonce":33686}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6828","nonce":33686},{"type":"v1","addr":"172.21.15.192:6829","nonce":33686}]},"public_addr":"172.21.15.192:6825/33686","cluster_addr":"172.21.15.192:6827/33686","heartbeat_back_addr":"172.21.15.192:6831/33686","heartbeat_front_addr":"172.21.15.192:6829/33686","state":["exists","up"]},{"osd":7,"uuid":"f1ebdfd5-b859-482b-9f97-c18c4a26f445","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6808","nonce":33685},{"type":"v1","addr":"172.21.15.192:6809","nonce":33685}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":33685},{"type":"v1","addr":"172.21.15.192:6811","nonce":33685}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":33685},{"type":"v1","addr":"172.21.15.192:6815","nonce":33685}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6812","nonce":33685},{"type":"v1","addr":"172.21.15.192:6813","nonce":33685}]},"public_addr":"172.21.15.192:6809/33685","cluster_addr":"172.21.15.192:6811/33685","heartbeat_back_addr":"172.21.15.192:6815/33685","heartbeat_front_addr":"172.21.15.192:6813/33685","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T03:29:18.002048+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T03:29:17.478576+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T03:29:18.034944+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T03:29:17.628513+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T03:29:18.047338+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T03:29:17.772349+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T03:29:17.883958+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T03:29:18.151838+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-06T03:29:45.132 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:45.130+0000 7fce8d495700 1 -- 172.21.15.110:0/1254178413 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fce74060550 msgr2=0x7fce74062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:45.132 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:45.130+0000 7fce8d495700 1 --2- 172.21.15.110:0/1254178413 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fce74060550 0x7fce74062a00 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fce80009880 tx=0x7fce80018000 comp rx=0 tx=0).stop 2023-10-06T03:29:45.132 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:45.130+0000 7fce8d495700 1 -- 172.21.15.110:0/1254178413 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fce880adac0 msgr2=0x7fce88157430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:45.133 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:45.130+0000 7fce8d495700 1 --2- 172.21.15.110:0/1254178413 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fce880adac0 0x7fce88157430 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fce7800b810 tx=0x7fce78009250 comp rx=0 tx=0).stop 2023-10-06T03:29:45.133 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:45.130+0000 7fce8d495700 1 -- 172.21.15.110:0/1254178413 shutdown_connections 2023-10-06T03:29:45.133 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:45.130+0000 7fce8d495700 1 --2- 172.21.15.110:0/1254178413 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fce74060550 0x7fce74062a00 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:45.133 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:45.130+0000 7fce8d495700 1 --2- 172.21.15.110:0/1254178413 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fce880c9bb0 0x7fce880afd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:45.133 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:45.130+0000 7fce8d495700 1 --2- 172.21.15.110:0/1254178413 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fce880aec00 0x7fce880af810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:45.133 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:45.130+0000 7fce8d495700 1 --2- 172.21.15.110:0/1254178413 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fce880adac0 0x7fce88157430 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:45.133 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:45.130+0000 7fce8d495700 1 -- 172.21.15.110:0/1254178413 >> 172.21.15.110:0/1254178413 conn(0x7fce880cc470 msgr2=0x7fce8815b6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:29:45.133 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:45.130+0000 7fce8d495700 1 -- 172.21.15.110:0/1254178413 shutdown_connections 2023-10-06T03:29:45.133 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:45.130+0000 7fce8d495700 1 -- 172.21.15.110:0/1254178413 wait complete. 2023-10-06T03:29:45.144 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2023-10-06T03:29:45.260 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:45.258+0000 7fd7a5de8700 1 Processor -- start 2023-10-06T03:29:45.260 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:45.258+0000 7fd7a5de8700 1 -- start start 2023-10-06T03:29:45.261 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:45.258+0000 7fd7a5de8700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fd7a006c970 0x7fd7a006cd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:45.261 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:45.258+0000 7fd7a5de8700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7a01591d0 con 0x7fd7a0159c20 2023-10-06T03:29:45.262 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:45.258+0000 7fd7a5de8700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7a0156700 con 0x7fd7a0152ca0 2023-10-06T03:29:45.262 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:45.258+0000 7fd7a5de8700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd7a01569e0 con 0x7fd7a006c970 2023-10-06T03:29:45.262 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:45.258+0000 7fd7a4de6700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fd7a006c970 0x7fd7a006cd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:45.262 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:45.258+0000 7fd79ffff700 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7fd7a0159c20 0x7fd7a0155b90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:37244/0 (socket says 172.21.15.110:37244) 2023-10-06T03:29:45.262 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:45.258+0000 7fd7a4de6700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fd7a006c970 0x7fd7a006cd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:60382/0 (socket says 172.21.15.110:60382) 2023-10-06T03:29:45.262 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:45.258+0000 7fd79ffff700 1 -- 172.21.15.110:0/2718768689 learned_addr learned my addr 172.21.15.110:0/2718768689 (peer_addr_for_me v1:172.21.15.110:0/0) 2023-10-06T03:29:45.264 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:45.258+0000 7fd7a4de6700 1 -- 172.21.15.110:0/2718768689 >> v1:172.21.15.192:6789/0 conn(0x7fd7a0152ca0 legacy=0x7fd7a00b06e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:45.264 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:45.258+0000 7fd7a4de6700 1 -- 172.21.15.110:0/2718768689 >> v1:172.21.15.110:6789/0 conn(0x7fd7a0159c20 legacy=0x7fd7a0155b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:45.264 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:45.258+0000 7fd79f7fe700 1 -- 172.21.15.110:0/2718768689 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1285997035 0 0) 0x7fd7a01591d0 con 0x7fd7a0159c20 2023-10-06T03:29:45.264 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:45.258+0000 7fd7a4de6700 1 -- 172.21.15.110:0/2718768689 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7a0159420 con 0x7fd7a006c970 2023-10-06T03:29:45.264 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:45.258+0000 7fd7a4de6700 1 --2- 172.21.15.110:0/2718768689 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd7a006c970 0x7fd7a006cd40 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fd790009e90 tx=0x7fd790009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=688ef24c5f50e31c server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:29:45.264 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:45.258+0000 7fd79f7fe700 1 -- 172.21.15.110:0/2718768689 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd79000e070 con 0x7fd7a006c970 2023-10-06T03:29:45.265 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:45.258+0000 7fd79f7fe700 1 -- 172.21.15.110:0/2718768689 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd79000bbf0 con 0x7fd7a006c970 2023-10-06T03:29:45.265 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:45.258+0000 7fd79f7fe700 1 -- 172.21.15.110:0/2718768689 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd790012430 con 0x7fd7a006c970 2023-10-06T03:29:45.265 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:45.262+0000 7fd7a5de8700 1 -- 172.21.15.110:0/2718768689 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd7a006c970 msgr2=0x7fd7a006cd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:45.265 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:45.262+0000 7fd7a5de8700 1 --2- 172.21.15.110:0/2718768689 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd7a006c970 0x7fd7a006cd40 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fd790009e90 tx=0x7fd790009780 comp rx=0 tx=0).stop 2023-10-06T03:29:45.265 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:45.262+0000 7fd7a5de8700 1 -- 172.21.15.110:0/2718768689 shutdown_connections 2023-10-06T03:29:45.265 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:45.262+0000 7fd7a5de8700 1 --2- 172.21.15.110:0/2718768689 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd7a006c970 0x7fd7a006cd40 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:45.265 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:45.262+0000 7fd7a5de8700 1 -- 172.21.15.110:0/2718768689 >> 172.21.15.110:0/2718768689 conn(0x7fd7a006ab30 msgr2=0x7fd7a00adff0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:29:45.265 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:45.262+0000 7fd7a5de8700 1 -- 172.21.15.110:0/2718768689 shutdown_connections 2023-10-06T03:29:45.265 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:45.262+0000 7fd7a5de8700 1 -- 172.21.15.110:0/2718768689 wait complete. 2023-10-06T03:29:45.266 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:45.262+0000 7fd7a5de8700 1 Processor -- start 2023-10-06T03:29:45.266 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:45.262+0000 7fd7a5de8700 1 -- start start 2023-10-06T03:29:45.269 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:45.266+0000 7fd7a5de8700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd7a006c970 0x7fd7a00c7620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:45.269 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:45.266+0000 7fd7a5de8700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd7a0152ca0 0x7fd7a016aed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:45.269 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:45.266+0000 7fd7a5de8700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd7a0159c20 0x7fd7a016b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:45.269 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:45.266+0000 7fd7a5de8700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7a00c1d60 con 0x7fd7a0152ca0 2023-10-06T03:29:45.270 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:45.266+0000 7fd7a5de8700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7a0153610 con 0x7fd7a006c970 2023-10-06T03:29:45.270 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:45.266+0000 7fd7a5de8700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7a0153eb0 con 0x7fd7a0159c20 2023-10-06T03:29:45.270 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:45.266+0000 7fd79ffff700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd7a0152ca0 0x7fd7a016aed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:45.270 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:45.266+0000 7fd79ffff700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd7a0152ca0 0x7fd7a016aed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:60394/0 (socket says 172.21.15.110:60394) 2023-10-06T03:29:45.270 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:45.266+0000 7fd79ffff700 1 -- 172.21.15.110:0/2487536017 learned_addr learned my addr 172.21.15.110:0/2487536017 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:29:45.270 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:45.266+0000 7fd7a4de6700 1 --2- 172.21.15.110:0/2487536017 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd7a006c970 0x7fd7a00c7620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:45.270 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:45.266+0000 7fd7a55e7700 1 --2- 172.21.15.110:0/2487536017 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd7a0159c20 0x7fd7a016b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:45.270 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:45.266+0000 7fd79ffff700 1 -- 172.21.15.110:0/2487536017 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd7a0159c20 msgr2=0x7fd7a016b410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:45.270 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:45.266+0000 7fd79ffff700 1 --2- 172.21.15.110:0/2487536017 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd7a0159c20 0x7fd7a016b410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:45.270 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:45.266+0000 7fd79ffff700 1 -- 172.21.15.110:0/2487536017 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd7a006c970 msgr2=0x7fd7a00c7620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:45.270 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:45.266+0000 7fd79ffff700 1 --2- 172.21.15.110:0/2487536017 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd7a006c970 0x7fd7a00c7620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:45.270 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:45.266+0000 7fd79ffff700 1 -- 172.21.15.110:0/2487536017 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd790009a20 con 0x7fd7a0152ca0 2023-10-06T03:29:45.271 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:45.266+0000 7fd7a55e7700 1 --2- 172.21.15.110:0/2487536017 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd7a0159c20 0x7fd7a016b410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-06T03:29:45.271 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:45.266+0000 7fd7a4de6700 1 --2- 172.21.15.110:0/2487536017 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd7a006c970 0x7fd7a00c7620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-06T03:29:45.271 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:45.266+0000 7fd79ffff700 1 --2- 172.21.15.110:0/2487536017 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd7a0152ca0 0x7fd7a016aed0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fd78800c5b0 tx=0x7fd788007f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:29:45.271 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:45.266+0000 7fd79cff9700 1 -- 172.21.15.110:0/2487536017 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7880062e0 con 0x7fd7a0152ca0 2023-10-06T03:29:45.271 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:45.266+0000 7fd79cff9700 1 -- 172.21.15.110:0/2487536017 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd788006440 con 0x7fd7a0152ca0 2023-10-06T03:29:45.271 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:45.266+0000 7fd7a5de8700 1 -- 172.21.15.110:0/2487536017 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd7a00ad790 con 0x7fd7a0152ca0 2023-10-06T03:29:45.271 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:45.266+0000 7fd79cff9700 1 -- 172.21.15.110:0/2487536017 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd788004830 con 0x7fd7a0152ca0 2023-10-06T03:29:45.271 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:45.266+0000 7fd7a5de8700 1 -- 172.21.15.110:0/2487536017 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7a0153b40 con 0x7fd7a0152ca0 2023-10-06T03:29:45.272 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:45.270+0000 7fd79cff9700 1 -- 172.21.15.110:0/2487536017 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7fd788004be0 con 0x7fd7a0152ca0 2023-10-06T03:29:45.272 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:45.270+0000 7fd7a5de8700 1 -- 172.21.15.110:0/2487536017 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd7a0154060 con 0x7fd7a0152ca0 2023-10-06T03:29:45.272 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:45.270+0000 7fd79cff9700 1 --2- 172.21.15.110:0/2487536017 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fd780060580 0x7fd780062a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:45.273 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:45.270+0000 7fd7a4de6700 1 --2- 172.21.15.110:0/2487536017 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fd780060580 0x7fd780062a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:45.273 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:45.270+0000 7fd79cff9700 1 -- 172.21.15.110:0/2487536017 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6809+0+0 (secure 0 0 0) 0x7fd78807c070 con 0x7fd7a0152ca0 2023-10-06T03:29:45.273 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:45.270+0000 7fd7a4de6700 1 --2- 172.21.15.110:0/2487536017 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fd780060580 0x7fd780062a30 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fd790009e90 tx=0x7fd790019000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:29:45.279 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:45.278+0000 7fd79cff9700 1 -- 172.21.15.110:0/2487536017 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fd7a0154060 con 0x7fd7a0152ca0 2023-10-06T03:29:45.475 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:45.474+0000 7fd7a5de8700 1 -- 172.21.15.110:0/2487536017 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7fd7a0154020 con 0x7fd7a0152ca0 2023-10-06T03:29:46.312 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:46.310+0000 7fd79cff9700 1 -- 172.21.15.110:0/2487536017 <== mon.0 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v7) v1 ==== 118+0+0 (secure 0 0 0) 0x7fd78804c210 con 0x7fd7a0152ca0 2023-10-06T03:29:46.316 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:46.314+0000 7fd7a5de8700 1 -- 172.21.15.110:0/2487536017 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fd780060580 msgr2=0x7fd780062a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:46.316 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:46.314+0000 7fd7a5de8700 1 --2- 172.21.15.110:0/2487536017 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fd780060580 0x7fd780062a30 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fd790009e90 tx=0x7fd790019000 comp rx=0 tx=0).stop 2023-10-06T03:29:46.317 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:46.314+0000 7fd7a5de8700 1 -- 172.21.15.110:0/2487536017 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd7a0152ca0 msgr2=0x7fd7a016aed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:46.317 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:46.314+0000 7fd7a5de8700 1 --2- 172.21.15.110:0/2487536017 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd7a0152ca0 0x7fd7a016aed0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fd78800c5b0 tx=0x7fd788007f90 comp rx=0 tx=0).stop 2023-10-06T03:29:46.317 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:46.314+0000 7fd7a5de8700 1 -- 172.21.15.110:0/2487536017 shutdown_connections 2023-10-06T03:29:46.317 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:46.314+0000 7fd7a5de8700 1 --2- 172.21.15.110:0/2487536017 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fd780060580 0x7fd780062a30 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:46.317 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:46.314+0000 7fd7a5de8700 1 --2- 172.21.15.110:0/2487536017 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd7a0159c20 0x7fd7a016b410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:46.317 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:46.314+0000 7fd7a5de8700 1 --2- 172.21.15.110:0/2487536017 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd7a0152ca0 0x7fd7a016aed0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:46.317 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:46.314+0000 7fd7a5de8700 1 --2- 172.21.15.110:0/2487536017 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd7a006c970 0x7fd7a00c7620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:46.317 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:46.314+0000 7fd7a5de8700 1 -- 172.21.15.110:0/2487536017 >> 172.21.15.110:0/2487536017 conn(0x7fd7a006ab30 msgr2=0x7fd7a00670a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:29:46.318 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:46.314+0000 7fd7a5de8700 1 -- 172.21.15.110:0/2487536017 shutdown_connections 2023-10-06T03:29:46.318 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:46.314+0000 7fd7a5de8700 1 -- 172.21.15.110:0/2487536017 wait complete. 2023-10-06T03:29:46.333 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2023-10-06T03:29:46.457 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:46.454+0000 7fa0e9b38700 1 Processor -- start 2023-10-06T03:29:46.457 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:46.454+0000 7fa0e9b38700 1 -- start start 2023-10-06T03:29:46.459 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:46.454+0000 7fa0e9b38700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fa0e406b7c0 0x7fa0e40ce170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:46.459 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:46.454+0000 7fa0e9b38700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa0e40ce6b0 0x7fa0e406bb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:46.459 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:46.454+0000 7fa0e9b38700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0e40cfa00 con 0x7fa0e406c0d0 2023-10-06T03:29:46.459 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:46.454+0000 7fa0e9b38700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa0e40cb1c0 con 0x7fa0e406b7c0 2023-10-06T03:29:46.459 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:46.454+0000 7fa0e9b38700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0e40cedf0 con 0x7fa0e40ce6b0 2023-10-06T03:29:46.459 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:46.454+0000 7fa0e8b36700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fa0e406b7c0 0x7fa0e40ce170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:46.459 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:46.454+0000 7fa0e8b36700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fa0e406b7c0 0x7fa0e40ce170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:60402/0 (socket says 172.21.15.110:60402) 2023-10-06T03:29:46.460 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:46.458+0000 7fa0e8b36700 1 -- 172.21.15.110:0/503974596 learned_addr learned my addr 172.21.15.110:0/503974596 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:29:46.460 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:46.458+0000 7fa0e3fff700 1 --2- 172.21.15.110:0/503974596 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa0e40ce6b0 0x7fa0e406bb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:46.461 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:46.458+0000 7fa0e8b36700 1 -- 172.21.15.110:0/503974596 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa0e40ce6b0 msgr2=0x7fa0e406bb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:29:46.461 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:46.458+0000 7fa0e8b36700 1 --2- 172.21.15.110:0/503974596 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa0e40ce6b0 0x7fa0e406bb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:46.461 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:46.458+0000 7fa0e37fe700 1 -- 172.21.15.110:0/503974596 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1841489844 0 0) 0x7fa0e40cfa00 con 0x7fa0e406c0d0 2023-10-06T03:29:46.461 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:46.458+0000 7fa0e8b36700 1 -- 172.21.15.110:0/503974596 >> v1:172.21.15.192:6789/0 conn(0x7fa0e406c0d0 legacy=0x7fa0e40afa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:46.461 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:46.458+0000 7fa0e8b36700 1 -- 172.21.15.110:0/503974596 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0e40cf0d0 con 0x7fa0e406b7c0 2023-10-06T03:29:46.461 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:46.458+0000 7fa0e8b36700 1 --2- 172.21.15.110:0/503974596 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa0e406b7c0 0x7fa0e40ce170 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fa0d80099b0 tx=0x7fa0d8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=25b66f1fc199e796 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:29:46.461 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:46.458+0000 7fa0e37fe700 1 -- 172.21.15.110:0/503974596 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0d8009ea0 con 0x7fa0e406b7c0 2023-10-06T03:29:46.461 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:46.458+0000 7fa0e37fe700 1 -- 172.21.15.110:0/503974596 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0d800b860 con 0x7fa0e406b7c0 2023-10-06T03:29:46.461 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:46.458+0000 7fa0e37fe700 1 -- 172.21.15.110:0/503974596 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0d800bc00 con 0x7fa0e406b7c0 2023-10-06T03:29:46.462 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:46.458+0000 7fa0e9b38700 1 -- 172.21.15.110:0/503974596 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa0e406b7c0 msgr2=0x7fa0e40ce170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:46.462 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:46.458+0000 7fa0e9b38700 1 --2- 172.21.15.110:0/503974596 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa0e406b7c0 0x7fa0e40ce170 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fa0d80099b0 tx=0x7fa0d8009250 comp rx=0 tx=0).stop 2023-10-06T03:29:46.463 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:46.458+0000 7fa0e9b38700 1 -- 172.21.15.110:0/503974596 shutdown_connections 2023-10-06T03:29:46.463 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:46.458+0000 7fa0e9b38700 1 --2- 172.21.15.110:0/503974596 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa0e40ce6b0 0x7fa0e406bb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:46.463 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:46.458+0000 7fa0e9b38700 1 --2- 172.21.15.110:0/503974596 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa0e406b7c0 0x7fa0e40ce170 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:46.463 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:46.458+0000 7fa0e9b38700 1 -- 172.21.15.110:0/503974596 >> 172.21.15.110:0/503974596 conn(0x7fa0e40cfea0 msgr2=0x7fa0e40d02a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:29:46.463 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:46.458+0000 7fa0e9b38700 1 -- 172.21.15.110:0/503974596 shutdown_connections 2023-10-06T03:29:46.463 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:46.458+0000 7fa0e9b38700 1 -- 172.21.15.110:0/503974596 wait complete. 2023-10-06T03:29:46.464 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:46.462+0000 7fa0e9b38700 1 Processor -- start 2023-10-06T03:29:46.464 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:46.462+0000 7fa0e9b38700 1 -- start start 2023-10-06T03:29:46.466 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:46.462+0000 7fa0e9b38700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa0e406b7c0 0x7fa0e416d290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:46.466 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:46.462+0000 7fa0e9b38700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa0e406c0d0 0x7fa0e416a390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:46.467 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:46.462+0000 7fa0e9b38700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa0e40ce6b0 0x7fa0e40c97b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:46.467 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:46.462+0000 7fa0e9b38700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0e416d7d0 con 0x7fa0e406c0d0 2023-10-06T03:29:46.467 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:46.462+0000 7fa0e9b38700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0e4152160 con 0x7fa0e406b7c0 2023-10-06T03:29:46.467 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:46.462+0000 7fa0e9b38700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0e416a8d0 con 0x7fa0e40ce6b0 2023-10-06T03:29:46.467 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:46.462+0000 7fa0e3fff700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa0e406c0d0 0x7fa0e416a390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:46.467 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:46.462+0000 7fa0e3fff700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa0e406c0d0 0x7fa0e416a390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:60412/0 (socket says 172.21.15.110:60412) 2023-10-06T03:29:46.467 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:46.462+0000 7fa0e3fff700 1 -- 172.21.15.110:0/2503584794 learned_addr learned my addr 172.21.15.110:0/2503584794 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:29:46.467 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:46.462+0000 7fa0e8b36700 1 --2- 172.21.15.110:0/2503584794 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa0e406b7c0 0x7fa0e416d290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:46.467 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:46.462+0000 7fa0e9337700 1 --2- 172.21.15.110:0/2503584794 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa0e40ce6b0 0x7fa0e40c97b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:46.467 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:46.462+0000 7fa0e3fff700 1 -- 172.21.15.110:0/2503584794 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa0e40ce6b0 msgr2=0x7fa0e40c97b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:46.468 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:46.462+0000 7fa0e3fff700 1 --2- 172.21.15.110:0/2503584794 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa0e40ce6b0 0x7fa0e40c97b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:46.468 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:46.462+0000 7fa0e3fff700 1 -- 172.21.15.110:0/2503584794 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa0e406b7c0 msgr2=0x7fa0e416d290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:46.468 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:46.462+0000 7fa0e3fff700 1 --2- 172.21.15.110:0/2503584794 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa0e406b7c0 0x7fa0e416d290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:46.468 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:46.462+0000 7fa0e3fff700 1 -- 172.21.15.110:0/2503584794 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0d80094f0 con 0x7fa0e406c0d0 2023-10-06T03:29:46.468 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:46.462+0000 7fa0e8b36700 1 --2- 172.21.15.110:0/2503584794 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa0e406b7c0 0x7fa0e416d290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-06T03:29:46.468 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:46.462+0000 7fa0e9337700 1 --2- 172.21.15.110:0/2503584794 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa0e40ce6b0 0x7fa0e40c97b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-06T03:29:46.469 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:46.462+0000 7fa0e3fff700 1 --2- 172.21.15.110:0/2503584794 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa0e406c0d0 0x7fa0e416a390 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fa0d400e930 tx=0x7fa0d40091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:29:46.469 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:46.462+0000 7fa0e0ff9700 1 -- 172.21.15.110:0/2503584794 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0d400eed0 con 0x7fa0e406c0d0 2023-10-06T03:29:46.469 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:46.466+0000 7fa0e0ff9700 1 -- 172.21.15.110:0/2503584794 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0d400c850 con 0x7fa0e406c0d0 2023-10-06T03:29:46.469 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:46.466+0000 7fa0e9b38700 1 -- 172.21.15.110:0/2503584794 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0e40b3cb0 con 0x7fa0e406c0d0 2023-10-06T03:29:46.469 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:46.466+0000 7fa0e0ff9700 1 -- 172.21.15.110:0/2503584794 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0d400cbf0 con 0x7fa0e406c0d0 2023-10-06T03:29:46.470 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:46.466+0000 7fa0e9b38700 1 -- 172.21.15.110:0/2503584794 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0e40b4270 con 0x7fa0e406c0d0 2023-10-06T03:29:46.470 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:46.466+0000 7fa0e9b38700 1 -- 172.21.15.110:0/2503584794 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa0e40b4750 con 0x7fa0e406c0d0 2023-10-06T03:29:46.470 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:46.466+0000 7fa0e0ff9700 1 -- 172.21.15.110:0/2503584794 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7fa0d4013470 con 0x7fa0e406c0d0 2023-10-06T03:29:46.471 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:46.466+0000 7fa0e0ff9700 1 --2- 172.21.15.110:0/2503584794 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fa0d0060550 0x7fa0d0062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:46.471 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:46.466+0000 7fa0e0ff9700 1 -- 172.21.15.110:0/2503584794 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6809+0+0 (secure 0 0 0) 0x7fa0d407d5e0 con 0x7fa0e406c0d0 2023-10-06T03:29:46.471 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:46.466+0000 7fa0e8b36700 1 --2- 172.21.15.110:0/2503584794 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fa0d0060550 0x7fa0d0062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:46.471 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:46.470+0000 7fa0e8b36700 1 --2- 172.21.15.110:0/2503584794 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fa0d0060550 0x7fa0d0062a00 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fa0d80099b0 tx=0x7fa0d8017000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:29:46.476 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:46.474+0000 7fa0e0ff9700 1 -- 172.21.15.110:0/2503584794 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fa0d404d800 con 0x7fa0e406c0d0 2023-10-06T03:29:46.682 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:46.678+0000 7fa0e9b38700 1 -- 172.21.15.110:0/2503584794 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7fa0e4070ff0 con 0x7fa0e406c0d0 2023-10-06T03:29:47.322 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:47.317+0000 7fa0e0ff9700 1 -- 172.21.15.110:0/2503584794 <== mon.0 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v8) v1 ==== 122+0+0 (secure 0 0 0) 0x7fa0d4050e20 con 0x7fa0e406c0d0 2023-10-06T03:29:47.327 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:47.321+0000 7fa0e9b38700 1 -- 172.21.15.110:0/2503584794 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fa0d0060550 msgr2=0x7fa0d0062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:47.327 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:47.321+0000 7fa0e9b38700 1 --2- 172.21.15.110:0/2503584794 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fa0d0060550 0x7fa0d0062a00 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fa0d80099b0 tx=0x7fa0d8017000 comp rx=0 tx=0).stop 2023-10-06T03:29:47.327 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:47.321+0000 7fa0e9b38700 1 -- 172.21.15.110:0/2503584794 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa0e406c0d0 msgr2=0x7fa0e416a390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:47.327 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:47.321+0000 7fa0e9b38700 1 --2- 172.21.15.110:0/2503584794 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa0e406c0d0 0x7fa0e416a390 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fa0d400e930 tx=0x7fa0d40091a0 comp rx=0 tx=0).stop 2023-10-06T03:29:47.327 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:47.321+0000 7fa0e9b38700 1 -- 172.21.15.110:0/2503584794 shutdown_connections 2023-10-06T03:29:47.327 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:47.321+0000 7fa0e9b38700 1 --2- 172.21.15.110:0/2503584794 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fa0d0060550 0x7fa0d0062a00 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:47.327 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:47.321+0000 7fa0e9b38700 1 --2- 172.21.15.110:0/2503584794 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa0e40ce6b0 0x7fa0e40c97b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:47.327 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:47.321+0000 7fa0e9b38700 1 --2- 172.21.15.110:0/2503584794 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa0e406c0d0 0x7fa0e416a390 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:47.327 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:47.321+0000 7fa0e9b38700 1 --2- 172.21.15.110:0/2503584794 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa0e406b7c0 0x7fa0e416d290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:47.328 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:47.321+0000 7fa0e9b38700 1 -- 172.21.15.110:0/2503584794 >> 172.21.15.110:0/2503584794 conn(0x7fa0e40cfea0 msgr2=0x7fa0e4153ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:29:47.328 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:47.321+0000 7fa0e9b38700 1 -- 172.21.15.110:0/2503584794 shutdown_connections 2023-10-06T03:29:47.328 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:47.321+0000 7fa0e9b38700 1 -- 172.21.15.110:0/2503584794 wait complete. 2023-10-06T03:29:47.341 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-06T03:29:47.453 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:47.449+0000 7f144674b700 1 Processor -- start 2023-10-06T03:29:47.453 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:47.449+0000 7f144674b700 1 -- start start 2023-10-06T03:29:47.454 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:47.449+0000 7f144674b700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1440156f50 0x7f14400b0e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:47.454 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:47.449+0000 7f144674b700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f14400b1380 0x7f14400ad560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:47.454 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:47.449+0000 7f144674b700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14400c7f30 con 0x7f1440156b80 2023-10-06T03:29:47.455 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:47.449+0000 7f144674b700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f14400c7320 con 0x7f14400b1380 2023-10-06T03:29:47.455 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:47.449+0000 7f144674b700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14400c7600 con 0x7f1440156f50 2023-10-06T03:29:47.455 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:47.449+0000 7f1445749700 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f1440156b80 0x7f14400c68d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:37256/0 (socket says 172.21.15.110:37256) 2023-10-06T03:29:47.455 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:47.449+0000 7f1445749700 1 -- 172.21.15.110:0/611989552 learned_addr learned my addr 172.21.15.110:0/611989552 (peer_addr_for_me v1:172.21.15.110:0/0) 2023-10-06T03:29:47.455 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:47.449+0000 7f1444f48700 1 --2- 172.21.15.110:0/611989552 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1440156f50 0x7f14400b0e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:47.455 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:47.449+0000 7f1445f4a700 1 --2- 172.21.15.110:0/611989552 >> v2:172.21.15.192:3300/0 conn(0x7f14400b1380 0x7f14400ad560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:47.456 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:47.449+0000 7f1437fff700 1 -- 172.21.15.110:0/611989552 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2400546288 0 0) 0x7f14400c7f30 con 0x7f1440156b80 2023-10-06T03:29:47.456 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:47.449+0000 7f1437fff700 1 -- 172.21.15.110:0/611989552 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f142c003570 con 0x7f1440156b80 2023-10-06T03:29:47.456 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:47.453+0000 7f1444f48700 1 -- 172.21.15.110:0/611989552 >> v2:172.21.15.192:3300/0 conn(0x7f14400b1380 msgr2=0x7f14400ad560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:29:47.456 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:47.453+0000 7f1444f48700 1 --2- 172.21.15.110:0/611989552 >> v2:172.21.15.192:3300/0 conn(0x7f14400b1380 0x7f14400ad560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:47.456 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:47.453+0000 7f1444f48700 1 -- 172.21.15.110:0/611989552 >> v1:172.21.15.110:6789/0 conn(0x7f1440156b80 legacy=0x7f14400c68d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:47.457 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:47.453+0000 7f1444f48700 1 -- 172.21.15.110:0/611989552 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14400c78e0 con 0x7f1440156f50 2023-10-06T03:29:47.457 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:47.453+0000 7f1444f48700 1 --2- 172.21.15.110:0/611989552 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1440156f50 0x7f14400b0e40 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f143c009e60 tx=0x7f143c009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4f8f27eec0d1eea5 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:29:47.457 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:47.453+0000 7f1437fff700 1 -- 172.21.15.110:0/611989552 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f143c00e070 con 0x7f1440156f50 2023-10-06T03:29:47.457 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:47.453+0000 7f1437fff700 1 -- 172.21.15.110:0/611989552 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f143c00bab0 con 0x7f1440156f50 2023-10-06T03:29:47.457 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:47.453+0000 7f1437fff700 1 -- 172.21.15.110:0/611989552 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f143c00be70 con 0x7f1440156f50 2023-10-06T03:29:47.458 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:47.453+0000 7f144674b700 1 -- 172.21.15.110:0/611989552 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1440156f50 msgr2=0x7f14400b0e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:47.458 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:47.453+0000 7f144674b700 1 --2- 172.21.15.110:0/611989552 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1440156f50 0x7f14400b0e40 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f143c009e60 tx=0x7f143c009780 comp rx=0 tx=0).stop 2023-10-06T03:29:47.458 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:47.453+0000 7f144674b700 1 -- 172.21.15.110:0/611989552 shutdown_connections 2023-10-06T03:29:47.458 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:47.453+0000 7f144674b700 1 --2- 172.21.15.110:0/611989552 >> v2:172.21.15.192:3300/0 conn(0x7f14400b1380 0x7f14400ad560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:47.458 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:47.453+0000 7f144674b700 1 --2- 172.21.15.110:0/611989552 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1440156f50 0x7f14400b0e40 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:47.458 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:47.453+0000 7f144674b700 1 -- 172.21.15.110:0/611989552 >> 172.21.15.110:0/611989552 conn(0x7f14400cc470 msgr2=0x7f14400c8410 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:29:47.458 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:47.453+0000 7f144674b700 1 -- 172.21.15.110:0/611989552 shutdown_connections 2023-10-06T03:29:47.458 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:47.453+0000 7f144674b700 1 -- 172.21.15.110:0/611989552 wait complete. 2023-10-06T03:29:47.459 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:47.453+0000 7f144674b700 1 Processor -- start 2023-10-06T03:29:47.459 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:47.453+0000 7f144674b700 1 -- start start 2023-10-06T03:29:47.461 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:47.457+0000 7f144674b700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f14400b1380 0x7f14401592c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:47.461 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:47.457+0000 7f144674b700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1440156b80 0x7f1440149c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:47.461 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:47.457+0000 7f144674b700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1440156f50 0x7f144006c790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:47.461 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:47.457+0000 7f144674b700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1440159b70 con 0x7f14400b1380 2023-10-06T03:29:47.461 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:47.457+0000 7f144674b700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f144015a990 con 0x7f1440156b80 2023-10-06T03:29:47.461 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:47.457+0000 7f144674b700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1440159d80 con 0x7f1440156f50 2023-10-06T03:29:47.461 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:47.457+0000 7f1445749700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f14400b1380 0x7f14401592c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:47.461 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:47.457+0000 7f1445749700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f14400b1380 0x7f14401592c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:60424/0 (socket says 172.21.15.110:60424) 2023-10-06T03:29:47.462 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:47.457+0000 7f1445749700 1 -- 172.21.15.110:0/1709579210 learned_addr learned my addr 172.21.15.110:0/1709579210 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:29:47.462 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:47.457+0000 7f1444f48700 1 --2- 172.21.15.110:0/1709579210 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1440156b80 0x7f1440149c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:47.462 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:47.457+0000 7f1445f4a700 1 --2- 172.21.15.110:0/1709579210 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1440156f50 0x7f144006c790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:47.462 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:47.457+0000 7f1445749700 1 -- 172.21.15.110:0/1709579210 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1440156f50 msgr2=0x7f144006c790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:47.462 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:47.457+0000 7f1445749700 1 --2- 172.21.15.110:0/1709579210 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1440156f50 0x7f144006c790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:47.462 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:47.457+0000 7f1445749700 1 -- 172.21.15.110:0/1709579210 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1440156b80 msgr2=0x7f1440149c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:47.462 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:47.457+0000 7f1445749700 1 --2- 172.21.15.110:0/1709579210 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1440156b80 0x7f1440149c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:47.462 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:47.457+0000 7f1445749700 1 -- 172.21.15.110:0/1709579210 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f143c009a20 con 0x7f14400b1380 2023-10-06T03:29:47.462 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:47.457+0000 7f1445749700 1 --2- 172.21.15.110:0/1709579210 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f14400b1380 0x7f14401592c0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f143000c7c0 tx=0x7f1430007e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:29:47.463 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:47.457+0000 7f14357fa700 1 -- 172.21.15.110:0/1709579210 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1430005cd0 con 0x7f14400b1380 2023-10-06T03:29:47.463 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:47.457+0000 7f14357fa700 1 -- 172.21.15.110:0/1709579210 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1430005e30 con 0x7f14400b1380 2023-10-06T03:29:47.463 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:47.457+0000 7f144674b700 1 -- 172.21.15.110:0/1709579210 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f144015a060 con 0x7f14400b1380 2023-10-06T03:29:47.463 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:47.457+0000 7f14357fa700 1 -- 172.21.15.110:0/1709579210 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1430007320 con 0x7f14400b1380 2023-10-06T03:29:47.463 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:47.457+0000 7f144674b700 1 -- 172.21.15.110:0/1709579210 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f144015a620 con 0x7f14400b1380 2023-10-06T03:29:47.464 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:47.461+0000 7f14357fa700 1 -- 172.21.15.110:0/1709579210 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7f143001b3b0 con 0x7f14400b1380 2023-10-06T03:29:47.464 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:47.461+0000 7f144674b700 1 -- 172.21.15.110:0/1709579210 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1440070ff0 con 0x7f14400b1380 2023-10-06T03:29:47.465 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:47.461+0000 7f14357fa700 1 --2- 172.21.15.110:0/1709579210 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f1424060550 0x7f1424062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:47.465 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:47.461+0000 7f1444f48700 1 --2- 172.21.15.110:0/1709579210 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f1424060550 0x7f1424062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:47.465 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:47.461+0000 7f14357fa700 1 -- 172.21.15.110:0/1709579210 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6809+0+0 (secure 0 0 0) 0x7f143007c2b0 con 0x7f14400b1380 2023-10-06T03:29:47.465 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:47.461+0000 7f1444f48700 1 --2- 172.21.15.110:0/1709579210 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f1424060550 0x7f1424062a00 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f1440157320 tx=0x7f143c019000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:29:47.470 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:47.465+0000 7f14357fa700 1 -- 172.21.15.110:0/1709579210 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f143004c450 con 0x7f14400b1380 2023-10-06T03:29:47.673 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:47.669+0000 7f144674b700 1 -- 172.21.15.110:0/1709579210 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f14400c7bc0 con 0x7f14400b1380 2023-10-06T03:29:47.674 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:47.669+0000 7f14357fa700 1 -- 172.21.15.110:0/1709579210 <== mon.0 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4602 (secure 0 0 0) 0x7f143004fa70 con 0x7f14400b1380 2023-10-06T03:29:47.674 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-10-06T03:29:47.674 INFO:teuthology.orchestra.run.smithi110.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4279,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6833/1434505700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6832","nonce":1434505700},{"type":"v1","addr":"172.21.15.192:6833","nonce":1434505700}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4281,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/1168370789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6834","nonce":1168370789},{"type":"v1","addr":"172.21.15.110:6835","nonce":1168370789}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4288,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6835/1271810086","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":1271810086},{"type":"v1","addr":"172.21.15.192:6835","nonce":1271810086}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-06T03:29:33.894998+0000","modified":"2023-10-06T03:29:47.307075+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4293},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4293":{"gid":4293,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.110:6837/540320464","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6836","nonce":540320464},{"type":"v1","addr":"172.21.15.110:6837","nonce":540320464}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-06T03:29:47.677 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:47.673+0000 7f144674b700 1 -- 172.21.15.110:0/1709579210 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f1424060550 msgr2=0x7f1424062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:47.677 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:47.673+0000 7f144674b700 1 --2- 172.21.15.110:0/1709579210 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f1424060550 0x7f1424062a00 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f1440157320 tx=0x7f143c019000 comp rx=0 tx=0).stop 2023-10-06T03:29:47.677 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:47.673+0000 7f144674b700 1 -- 172.21.15.110:0/1709579210 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f14400b1380 msgr2=0x7f14401592c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:47.677 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:47.673+0000 7f144674b700 1 --2- 172.21.15.110:0/1709579210 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f14400b1380 0x7f14401592c0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f143000c7c0 tx=0x7f1430007e00 comp rx=0 tx=0).stop 2023-10-06T03:29:47.677 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:47.673+0000 7f144674b700 1 -- 172.21.15.110:0/1709579210 shutdown_connections 2023-10-06T03:29:47.677 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:47.673+0000 7f144674b700 1 --2- 172.21.15.110:0/1709579210 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f1424060550 0x7f1424062a00 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:47.677 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:47.673+0000 7f144674b700 1 --2- 172.21.15.110:0/1709579210 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1440156f50 0x7f144006c790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:47.678 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:47.673+0000 7f144674b700 1 --2- 172.21.15.110:0/1709579210 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1440156b80 0x7f1440149c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:47.678 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:47.673+0000 7f144674b700 1 --2- 172.21.15.110:0/1709579210 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f14400b1380 0x7f14401592c0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:47.678 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:47.673+0000 7f144674b700 1 -- 172.21.15.110:0/1709579210 >> 172.21.15.110:0/1709579210 conn(0x7f14400cc470 msgr2=0x7f1440155e00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:29:47.678 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:47.673+0000 7f144674b700 1 -- 172.21.15.110:0/1709579210 shutdown_connections 2023-10-06T03:29:47.678 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:47.673+0000 7f144674b700 1 -- 172.21.15.110:0/1709579210 wait complete. 2023-10-06T03:29:47.678 INFO:teuthology.orchestra.run.smithi110.stderr:dumped fsmap epoch 8 2023-10-06T03:29:47.689 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-06T03:29:47.793 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:47.789+0000 7f478f59e700 1 Processor -- start 2023-10-06T03:29:47.793 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:47.789+0000 7f478f59e700 1 -- start start 2023-10-06T03:29:47.794 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:47.789+0000 7f478f59e700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f47900add50 0x7f47900af4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:47.794 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:47.789+0000 7f478f59e700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f47900b04d0 0x7f47900ca660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:47.794 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:47.789+0000 7f478f59e700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4790155780 con 0x7f47900ae120 2023-10-06T03:29:47.794 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:47.789+0000 7f478f59e700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f47900cb4a0 con 0x7f47900add50 2023-10-06T03:29:47.794 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:47.789+0000 7f478f59e700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47900b0950 con 0x7f47900b04d0 2023-10-06T03:29:47.795 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:47.789+0000 7f478dd9b700 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f47900ae120 0x7f47900c5f40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:37264/0 (socket says 172.21.15.110:37264) 2023-10-06T03:29:47.795 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:47.789+0000 7f478dd9b700 1 -- 172.21.15.110:0/2135449188 learned_addr learned my addr 172.21.15.110:0/2135449188 (peer_addr_for_me v1:172.21.15.110:0/0) 2023-10-06T03:29:47.797 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:47.789+0000 7f478e59c700 1 --2- 172.21.15.110:0/2135449188 >> v2:172.21.15.192:3300/0 conn(0x7f47900add50 0x7f47900af4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:47.797 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:47.789+0000 7f478d59a700 1 -- 172.21.15.110:0/2135449188 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2464840221 0 0) 0x7f4790155780 con 0x7f47900ae120 2023-10-06T03:29:47.797 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:47.789+0000 7f478ed9d700 1 --2- 172.21.15.110:0/2135449188 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f47900b04d0 0x7f47900ca660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:47.797 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:47.789+0000 7f478d59a700 1 -- 172.21.15.110:0/2135449188 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4778003570 con 0x7f47900ae120 2023-10-06T03:29:47.797 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:47.793+0000 7f478d59a700 1 -- 172.21.15.110:0/2135449188 <== mon.0 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3871317732 0 0) 0x7f4778003570 con 0x7f47900ae120 2023-10-06T03:29:47.797 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:47.793+0000 7f478d59a700 1 -- 172.21.15.110:0/2135449188 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f47900b04d0 msgr2=0x7f47900ca660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:29:47.798 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:47.793+0000 7f478d59a700 1 --2- 172.21.15.110:0/2135449188 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f47900b04d0 0x7f47900ca660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:47.798 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:47.793+0000 7f478d59a700 1 -- 172.21.15.110:0/2135449188 >> v2:172.21.15.192:3300/0 conn(0x7f47900add50 msgr2=0x7f47900af4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:29:47.798 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:47.793+0000 7f478d59a700 1 --2- 172.21.15.110:0/2135449188 >> v2:172.21.15.192:3300/0 conn(0x7f47900add50 0x7f47900af4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:47.798 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:47.793+0000 7f478d59a700 1 -- 172.21.15.110:0/2135449188 --> v1:172.21.15.110:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47900b0c10 con 0x7f47900ae120 2023-10-06T03:29:47.798 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:47.793+0000 7f478d59a700 1 -- 172.21.15.110:0/2135449188 <== mon.0 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 118603703 0 0) 0x7f47800017f0 con 0x7f47900ae120 2023-10-06T03:29:47.798 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:47.793+0000 7f478e59c700 1 --2- 172.21.15.110:0/2135449188 >> v2:172.21.15.192:3300/0 conn(0x7f47900add50 0x7f47900af4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-06T03:29:47.798 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:47.793+0000 7f478d59a700 1 -- 172.21.15.110:0/2135449188 >> v1:172.21.15.110:6789/0 conn(0x7f47900ae120 legacy=0x7f47900c5f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:47.798 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:47.793+0000 7f478d59a700 1 --2- 172.21.15.110:0/2135449188 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4778003a60 0x7f4778003e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:47.798 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:47.793+0000 7f478d59a700 1 --2- 172.21.15.110:0/2135449188 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4778004810 0x7f4778004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:47.798 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:47.793+0000 7f478d59a700 1 --2- 172.21.15.110:0/2135449188 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f47900b04d0 0x7f47780051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:47.798 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:47.793+0000 7f478e59c700 1 --2- 172.21.15.110:0/2135449188 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4778003a60 0x7f4778003e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:47.799 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:47.793+0000 7f478d59a700 1 -- 172.21.15.110:0/2135449188 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47900cb4a0 con 0x7f4778003a60 2023-10-06T03:29:47.799 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:47.793+0000 7f478d59a700 1 -- 172.21.15.110:0/2135449188 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47900b0950 con 0x7f4778004810 2023-10-06T03:29:47.799 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:47.793+0000 7f478d59a700 1 -- 172.21.15.110:0/2135449188 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f477800dc70 con 0x7f47900b04d0 2023-10-06T03:29:47.799 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:47.793+0000 7f478dd9b700 1 --2- 172.21.15.110:0/2135449188 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4778004810 0x7f4778004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:47.799 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:47.793+0000 7f478ed9d700 1 --2- 172.21.15.110:0/2135449188 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f47900b04d0 0x7f47780051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:47.799 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:47.793+0000 7f478e59c700 1 -- 172.21.15.110:0/2135449188 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f47900b04d0 msgr2=0x7f47780051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:29:47.799 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:47.793+0000 7f478e59c700 1 --2- 172.21.15.110:0/2135449188 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f47900b04d0 0x7f47780051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:47.799 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:47.793+0000 7f478e59c700 1 -- 172.21.15.110:0/2135449188 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4778004810 msgr2=0x7f4778004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:29:47.799 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:47.793+0000 7f478e59c700 1 --2- 172.21.15.110:0/2135449188 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4778004810 0x7f4778004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:47.799 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:47.793+0000 7f478e59c700 1 -- 172.21.15.110:0/2135449188 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4778003570 con 0x7f4778003a60 2023-10-06T03:29:47.799 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:47.793+0000 7f478dd9b700 1 --2- 172.21.15.110:0/2135449188 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4778004810 0x7f4778004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-06T03:29:47.800 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:47.793+0000 7f478e59c700 1 --2- 172.21.15.110:0/2135449188 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4778003a60 0x7f4778003e30 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f4784009df0 tx=0x7f47840091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b8e1f5b1120eecf server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:29:47.800 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:47.793+0000 7f478d59a700 1 -- 172.21.15.110:0/2135449188 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f478400e920 con 0x7f4778003a60 2023-10-06T03:29:47.800 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:47.793+0000 7f478d59a700 1 -- 172.21.15.110:0/2135449188 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f478400ea80 con 0x7f4778003a60 2023-10-06T03:29:47.800 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:47.793+0000 7f478d59a700 1 -- 172.21.15.110:0/2135449188 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f478400ee20 con 0x7f4778003a60 2023-10-06T03:29:47.803 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:47.793+0000 7f478f59e700 1 -- 172.21.15.110:0/2135449188 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4778003a60 msgr2=0x7f4778003e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:47.803 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:47.793+0000 7f478f59e700 1 --2- 172.21.15.110:0/2135449188 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4778003a60 0x7f4778003e30 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f4784009df0 tx=0x7f47840091a0 comp rx=0 tx=0).stop 2023-10-06T03:29:47.803 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:47.793+0000 7f478ed9d700 1 -- 172.21.15.110:0/2135449188 reap_dead start 2023-10-06T03:29:47.803 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:47.793+0000 7f478f59e700 1 -- 172.21.15.110:0/2135449188 shutdown_connections 2023-10-06T03:29:47.803 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:47.793+0000 7f478f59e700 1 -- 172.21.15.110:0/2135449188 >> 172.21.15.110:0/2135449188 conn(0x7f47900cc470 msgr2=0x7f47900c8410 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:29:47.803 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:47.793+0000 7f478f59e700 1 -- 172.21.15.110:0/2135449188 shutdown_connections 2023-10-06T03:29:47.803 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:47.793+0000 7f478f59e700 1 -- 172.21.15.110:0/2135449188 wait complete. 2023-10-06T03:29:47.803 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:47.797+0000 7f478f59e700 1 Processor -- start 2023-10-06T03:29:47.804 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:47.797+0000 7f478f59e700 1 -- start start 2023-10-06T03:29:47.804 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:47.797+0000 7f478f59e700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4778003a60 0x7f47900c4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:47.804 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:47.797+0000 7f478f59e700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f47900c5d20 0x7f47901655d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:47.804 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:47.797+0000 7f478f59e700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4790165b10 0x7f47900c6f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:47.804 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:47.797+0000 7f478f59e700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47900c2110 con 0x7f4790165b10 2023-10-06T03:29:47.804 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:47.797+0000 7f478f59e700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47900c5370 con 0x7f47900c5d20 2023-10-06T03:29:47.804 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:47.797+0000 7f478f59e700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47900c6270 con 0x7f4778003a60 2023-10-06T03:29:47.804 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:47.797+0000 7f478ed9d700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4790165b10 0x7f47900c6f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:47.804 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:47.797+0000 7f478ed9d700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4790165b10 0x7f47900c6f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:60438/0 (socket says 172.21.15.110:60438) 2023-10-06T03:29:47.804 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:47.797+0000 7f478ed9d700 1 -- 172.21.15.110:0/421401043 learned_addr learned my addr 172.21.15.110:0/421401043 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:29:47.804 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:47.797+0000 7f478dd9b700 1 --2- 172.21.15.110:0/421401043 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f47900c5d20 0x7f47901655d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:47.805 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:47.797+0000 7f478e59c700 1 --2- 172.21.15.110:0/421401043 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4778003a60 0x7f47900c4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:47.805 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:47.797+0000 7f478ed9d700 1 -- 172.21.15.110:0/421401043 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4778003a60 msgr2=0x7f47900c4e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:47.805 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:47.797+0000 7f478ed9d700 1 --2- 172.21.15.110:0/421401043 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4778003a60 0x7f47900c4e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:47.805 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:47.797+0000 7f478ed9d700 1 -- 172.21.15.110:0/421401043 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f47900c5d20 msgr2=0x7f47901655d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:47.805 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:47.797+0000 7f478ed9d700 1 --2- 172.21.15.110:0/421401043 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f47900c5d20 0x7f47901655d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:47.805 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:47.797+0000 7f478ed9d700 1 -- 172.21.15.110:0/421401043 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4790165fa0 con 0x7f4790165b10 2023-10-06T03:29:47.805 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:47.797+0000 7f478dd9b700 1 --2- 172.21.15.110:0/421401043 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f47900c5d20 0x7f47901655d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-06T03:29:47.805 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:47.797+0000 7f478ed9d700 1 --2- 172.21.15.110:0/421401043 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4790165b10 0x7f47900c6f10 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f479006ece0 tx=0x7f4788008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:29:47.805 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:47.797+0000 7f477e7fc700 1 -- 172.21.15.110:0/421401043 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4788015740 con 0x7f4790165b10 2023-10-06T03:29:47.805 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:47.797+0000 7f477e7fc700 1 -- 172.21.15.110:0/421401043 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47880158a0 con 0x7f4790165b10 2023-10-06T03:29:47.805 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:47.797+0000 7f477e7fc700 1 -- 172.21.15.110:0/421401043 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4788015c60 con 0x7f4790165b10 2023-10-06T03:29:47.806 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:47.797+0000 7f478f59e700 1 -- 172.21.15.110:0/421401043 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4790166280 con 0x7f4790165b10 2023-10-06T03:29:47.806 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:47.797+0000 7f478f59e700 1 -- 172.21.15.110:0/421401043 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47900c7450 con 0x7f4790165b10 2023-10-06T03:29:47.807 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:47.801+0000 7f478f59e700 1 -- 172.21.15.110:0/421401043 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4790070ff0 con 0x7f4790165b10 2023-10-06T03:29:47.807 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:47.801+0000 7f477e7fc700 1 -- 172.21.15.110:0/421401043 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7f47880145f0 con 0x7f4790165b10 2023-10-06T03:29:47.807 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:47.801+0000 7f477e7fc700 1 --2- 172.21.15.110:0/421401043 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f4770060550 0x7f4770062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:47.807 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:47.801+0000 7f477e7fc700 1 -- 172.21.15.110:0/421401043 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6809+0+0 (secure 0 0 0) 0x7f478807ca70 con 0x7f4790165b10 2023-10-06T03:29:47.807 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:47.801+0000 7f478e59c700 1 --2- 172.21.15.110:0/421401043 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f4770060550 0x7f4770062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:47.807 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:47.801+0000 7f478e59c700 1 --2- 172.21.15.110:0/421401043 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f4770060550 0x7f4770062a00 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f47840091e0 tx=0x7f478401a000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:29:47.813 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:47.809+0000 7f477e7fc700 1 -- 172.21.15.110:0/421401043 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f478804cb70 con 0x7f4790165b10 2023-10-06T03:29:47.978 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:47.973+0000 7f478f59e700 1 -- 172.21.15.110:0/421401043 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f479006b860 con 0x7f4790165b10 2023-10-06T03:29:47.980 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:47.977+0000 7f477e7fc700 1 -- 172.21.15.110:0/421401043 <== mon.0 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v28) v1 ==== 74+0+18755 (secure 0 0 0) 0x7f4788050190 con 0x7f4790165b10 2023-10-06T03:29:47.981 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-10-06T03:29:47.981 INFO:teuthology.orchestra.run.smithi110.stdout:{"epoch":28,"fsid":"93b8f6ac-4c92-4b45-b8e4-b59617024aa2","created":"2023-10-06T03:29:10.519583+0000","modified":"2023-10-06T03:29:38.951740+0000","last_up_change":"2023-10-06T03:29:21.960511+0000","last_in_change":"2023-10-06T03:29:12.560436+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-06T03:29:20.703743+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2023-10-06T03:29:28.307281+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2023-10-06T03:29:32.247251+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data","create_time":"2023-10-06T03:29:32.880086+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}}},{"pool":5,"pool_name":"cephfs_data_ec","create_time":"2023-10-06T03:29:36.613779+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"28","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"5866eb6c-2adc-46e4-be7a-e5bcda0f4065","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6824","nonce":33394},{"type":"v1","addr":"172.21.15.110:6825","nonce":33394}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6826","nonce":33394},{"type":"v1","addr":"172.21.15.110:6827","nonce":33394}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6830","nonce":33394},{"type":"v1","addr":"172.21.15.110:6831","nonce":33394}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6828","nonce":33394},{"type":"v1","addr":"172.21.15.110:6829","nonce":33394}]},"public_addr":"172.21.15.110:6825/33394","cluster_addr":"172.21.15.110:6827/33394","heartbeat_back_addr":"172.21.15.110:6831/33394","heartbeat_front_addr":"172.21.15.110:6829/33394","state":["exists","up"]},{"osd":1,"uuid":"a38de118-062b-4a05-9782-2ceddf6ff4df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6800","nonce":33396},{"type":"v1","addr":"172.21.15.110:6801","nonce":33396}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6802","nonce":33396},{"type":"v1","addr":"172.21.15.110:6803","nonce":33396}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6806","nonce":33396},{"type":"v1","addr":"172.21.15.110:6807","nonce":33396}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6804","nonce":33396},{"type":"v1","addr":"172.21.15.110:6805","nonce":33396}]},"public_addr":"172.21.15.110:6801/33396","cluster_addr":"172.21.15.110:6803/33396","heartbeat_back_addr":"172.21.15.110:6807/33396","heartbeat_front_addr":"172.21.15.110:6805/33396","state":["exists","up"]},{"osd":2,"uuid":"56e39be2-3407-4b69-877e-cf16617b06d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6816","nonce":33397},{"type":"v1","addr":"172.21.15.110:6817","nonce":33397}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6818","nonce":33397},{"type":"v1","addr":"172.21.15.110:6819","nonce":33397}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6822","nonce":33397},{"type":"v1","addr":"172.21.15.110:6823","nonce":33397}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6820","nonce":33397},{"type":"v1","addr":"172.21.15.110:6821","nonce":33397}]},"public_addr":"172.21.15.110:6817/33397","cluster_addr":"172.21.15.110:6819/33397","heartbeat_back_addr":"172.21.15.110:6823/33397","heartbeat_front_addr":"172.21.15.110:6821/33397","state":["exists","up"]},{"osd":3,"uuid":"01c82d13-c1d4-4bde-b55e-3e0089b40115","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6808","nonce":33395},{"type":"v1","addr":"172.21.15.110:6809","nonce":33395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6810","nonce":33395},{"type":"v1","addr":"172.21.15.110:6811","nonce":33395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6814","nonce":33395},{"type":"v1","addr":"172.21.15.110:6815","nonce":33395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6812","nonce":33395},{"type":"v1","addr":"172.21.15.110:6813","nonce":33395}]},"public_addr":"172.21.15.110:6809/33395","cluster_addr":"172.21.15.110:6811/33395","heartbeat_back_addr":"172.21.15.110:6815/33395","heartbeat_front_addr":"172.21.15.110:6813/33395","state":["exists","up"]},{"osd":4,"uuid":"86983430-5fa3-4d17-998f-03e2cd843237","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":33683},{"type":"v1","addr":"172.21.15.192:6817","nonce":33683}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":33683},{"type":"v1","addr":"172.21.15.192:6819","nonce":33683}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":33683},{"type":"v1","addr":"172.21.15.192:6823","nonce":33683}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6820","nonce":33683},{"type":"v1","addr":"172.21.15.192:6821","nonce":33683}]},"public_addr":"172.21.15.192:6817/33683","cluster_addr":"172.21.15.192:6819/33683","heartbeat_back_addr":"172.21.15.192:6823/33683","heartbeat_front_addr":"172.21.15.192:6821/33683","state":["exists","up"]},{"osd":5,"uuid":"6945257f-821d-4619-a63f-ebe857eba6f4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":33684},{"type":"v1","addr":"172.21.15.192:6801","nonce":33684}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":33684},{"type":"v1","addr":"172.21.15.192:6803","nonce":33684}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":33684},{"type":"v1","addr":"172.21.15.192:6807","nonce":33684}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6804","nonce":33684},{"type":"v1","addr":"172.21.15.192:6805","nonce":33684}]},"public_addr":"172.21.15.192:6801/33684","cluster_addr":"172.21.15.192:6803/33684","heartbeat_back_addr":"172.21.15.192:6807/33684","heartbeat_front_addr":"172.21.15.192:6805/33684","state":["exists","up"]},{"osd":6,"uuid":"7854d1ae-7060-4780-8f97-73cea5e7a12f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6824","nonce":33686},{"type":"v1","addr":"172.21.15.192:6825","nonce":33686}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":33686},{"type":"v1","addr":"172.21.15.192:6827","nonce":33686}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":33686},{"type":"v1","addr":"172.21.15.192:6831","nonce":33686}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6828","nonce":33686},{"type":"v1","addr":"172.21.15.192:6829","nonce":33686}]},"public_addr":"172.21.15.192:6825/33686","cluster_addr":"172.21.15.192:6827/33686","heartbeat_back_addr":"172.21.15.192:6831/33686","heartbeat_front_addr":"172.21.15.192:6829/33686","state":["exists","up"]},{"osd":7,"uuid":"f1ebdfd5-b859-482b-9f97-c18c4a26f445","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6808","nonce":33685},{"type":"v1","addr":"172.21.15.192:6809","nonce":33685}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":33685},{"type":"v1","addr":"172.21.15.192:6811","nonce":33685}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":33685},{"type":"v1","addr":"172.21.15.192:6815","nonce":33685}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6812","nonce":33685},{"type":"v1","addr":"172.21.15.192:6813","nonce":33685}]},"public_addr":"172.21.15.192:6809/33685","cluster_addr":"172.21.15.192:6811/33685","heartbeat_back_addr":"172.21.15.192:6815/33685","heartbeat_front_addr":"172.21.15.192:6813/33685","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T03:29:18.002048+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T03:29:17.478576+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T03:29:18.034944+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T03:29:17.628513+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T03:29:18.047338+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T03:29:17.772349+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T03:29:17.883958+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T03:29:18.151838+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-06T03:29:47.983 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:47.977+0000 7f478f59e700 1 -- 172.21.15.110:0/421401043 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f4770060550 msgr2=0x7f4770062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:47.983 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:47.977+0000 7f478f59e700 1 --2- 172.21.15.110:0/421401043 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f4770060550 0x7f4770062a00 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f47840091e0 tx=0x7f478401a000 comp rx=0 tx=0).stop 2023-10-06T03:29:47.983 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:47.977+0000 7f478f59e700 1 -- 172.21.15.110:0/421401043 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4790165b10 msgr2=0x7f47900c6f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:47.983 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:47.977+0000 7f478f59e700 1 --2- 172.21.15.110:0/421401043 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4790165b10 0x7f47900c6f10 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f479006ece0 tx=0x7f4788008040 comp rx=0 tx=0).stop 2023-10-06T03:29:47.984 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:47.977+0000 7f478f59e700 1 -- 172.21.15.110:0/421401043 shutdown_connections 2023-10-06T03:29:47.984 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:47.981+0000 7f478f59e700 1 --2- 172.21.15.110:0/421401043 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f4770060550 0x7f4770062a00 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:47.984 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:47.981+0000 7f478f59e700 1 --2- 172.21.15.110:0/421401043 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4790165b10 0x7f47900c6f10 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:47.984 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:47.981+0000 7f478f59e700 1 --2- 172.21.15.110:0/421401043 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f47900c5d20 0x7f47901655d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:47.984 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:47.981+0000 7f478f59e700 1 --2- 172.21.15.110:0/421401043 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4778003a60 0x7f47900c4e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:47.984 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:47.981+0000 7f478f59e700 1 -- 172.21.15.110:0/421401043 >> 172.21.15.110:0/421401043 conn(0x7f47900cc470 msgr2=0x7f4790157940 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:29:47.984 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:47.981+0000 7f478f59e700 1 -- 172.21.15.110:0/421401043 shutdown_connections 2023-10-06T03:29:47.984 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:47.981+0000 7f478f59e700 1 -- 172.21.15.110:0/421401043 wait complete. 2023-10-06T03:29:47.996 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2023-10-06T03:29:47.996 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-06T03:29:48.103 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:48.097+0000 7f89d6ec4700 1 Processor -- start 2023-10-06T03:29:48.103 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:48.101+0000 7f89d6ec4700 1 -- start start 2023-10-06T03:29:48.104 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:48.101+0000 7f89d6ec4700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f89d0063430 0x7f89d00c8480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:48.104 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:48.101+0000 7f89d6ec4700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f89d0155570 0x7f89d01559d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:48.104 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:48.101+0000 7f89d6ec4700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89d006b7c0 con 0x7f89d00c80b0 2023-10-06T03:29:48.104 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:48.101+0000 7f89d6ec4700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f89d01562e0 con 0x7f89d0063430 2023-10-06T03:29:48.104 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:48.101+0000 7f89d6ec4700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89d01565c0 con 0x7f89d0155570 2023-10-06T03:29:48.104 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:48.101+0000 7f89d56c1700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f89d0063430 0x7f89d00c8480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:48.104 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:48.101+0000 7f89d56c1700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f89d0063430 0x7f89d00c8480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:60454/0 (socket says 172.21.15.110:60454) 2023-10-06T03:29:48.105 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:48.101+0000 7f89d56c1700 1 -- 172.21.15.110:0/2115821400 learned_addr learned my addr 172.21.15.110:0/2115821400 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:29:48.106 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:48.101+0000 7f89d56c1700 1 -- 172.21.15.110:0/2115821400 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f89d0155570 msgr2=0x7f89d01559d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:29:48.106 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:48.101+0000 7f89d66c3700 1 --2- 172.21.15.110:0/2115821400 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f89d0155570 0x7f89d01559d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:48.107 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:48.101+0000 7f89d56c1700 1 --2- 172.21.15.110:0/2115821400 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f89d0155570 0x7f89d01559d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:48.107 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:48.101+0000 7f89d56c1700 1 -- 172.21.15.110:0/2115821400 >> v1:172.21.15.110:6789/0 conn(0x7f89d00c80b0 legacy=0x7f89d00ae6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:48.107 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:48.101+0000 7f89d4ec0700 1 -- 172.21.15.110:0/2115821400 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 587509925 0 0) 0x7f89d006b7c0 con 0x7f89d00c80b0 2023-10-06T03:29:48.107 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:48.101+0000 7f89d56c1700 1 -- 172.21.15.110:0/2115821400 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89d01568a0 con 0x7f89d0063430 2023-10-06T03:29:48.107 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:48.101+0000 7f89d66c3700 1 --2- 172.21.15.110:0/2115821400 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f89d0155570 0x7f89d01559d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-06T03:29:48.107 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:48.101+0000 7f89d56c1700 1 --2- 172.21.15.110:0/2115821400 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f89d0063430 0x7f89d00c8480 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f89cc009f50 tx=0x7f89cc009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=66563690b2d9687c server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:29:48.107 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:48.101+0000 7f89d4ec0700 1 -- 172.21.15.110:0/2115821400 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89cc00e070 con 0x7f89d0063430 2023-10-06T03:29:48.107 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:48.101+0000 7f89d4ec0700 1 -- 172.21.15.110:0/2115821400 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89cc00ba70 con 0x7f89d0063430 2023-10-06T03:29:48.107 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:48.101+0000 7f89d4ec0700 1 -- 172.21.15.110:0/2115821400 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89cc0092e0 con 0x7f89d0063430 2023-10-06T03:29:48.108 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:48.101+0000 7f89d6ec4700 1 -- 172.21.15.110:0/2115821400 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f89d0063430 msgr2=0x7f89d00c8480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:48.108 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:48.101+0000 7f89d6ec4700 1 --2- 172.21.15.110:0/2115821400 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f89d0063430 0x7f89d00c8480 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f89cc009f50 tx=0x7f89cc009840 comp rx=0 tx=0).stop 2023-10-06T03:29:48.108 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:48.101+0000 7f89d6ec4700 1 -- 172.21.15.110:0/2115821400 shutdown_connections 2023-10-06T03:29:48.108 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:48.101+0000 7f89d6ec4700 1 --2- 172.21.15.110:0/2115821400 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f89d0155570 0x7f89d01559d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:48.108 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:48.101+0000 7f89d6ec4700 1 --2- 172.21.15.110:0/2115821400 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f89d0063430 0x7f89d00c8480 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:48.108 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:48.101+0000 7f89d6ec4700 1 -- 172.21.15.110:0/2115821400 >> 172.21.15.110:0/2115821400 conn(0x7f89d00c8d00 msgr2=0x7f89d00c9100 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:29:48.108 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:48.101+0000 7f89d6ec4700 1 -- 172.21.15.110:0/2115821400 shutdown_connections 2023-10-06T03:29:48.108 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:48.101+0000 7f89d6ec4700 1 -- 172.21.15.110:0/2115821400 wait complete. 2023-10-06T03:29:48.109 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:48.105+0000 7f89d6ec4700 1 Processor -- start 2023-10-06T03:29:48.109 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:48.105+0000 7f89d6ec4700 1 -- start start 2023-10-06T03:29:48.112 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:48.109+0000 7f89d6ec4700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f89d0063430 0x7f89d00c6140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:48.112 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:48.109+0000 7f89d6ec4700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f89d00c80b0 0x7f89d00c3cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:48.112 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:48.109+0000 7f89d6ec4700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f89d0155570 0x7f89d014eef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:48.112 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:48.109+0000 7f89d6ec4700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89d00c4230 con 0x7f89d0063430 2023-10-06T03:29:48.112 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:48.109+0000 7f89d6ec4700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89d00c44a0 con 0x7f89d0155570 2023-10-06T03:29:48.113 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:48.109+0000 7f89d6ec4700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89d00c4780 con 0x7f89d00c80b0 2023-10-06T03:29:48.113 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:48.109+0000 7f89d5ec2700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f89d0063430 0x7f89d00c6140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:48.113 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:48.109+0000 7f89d5ec2700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f89d0063430 0x7f89d00c6140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:60456/0 (socket says 172.21.15.110:60456) 2023-10-06T03:29:48.113 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:48.109+0000 7f89d5ec2700 1 -- 172.21.15.110:0/831785518 learned_addr learned my addr 172.21.15.110:0/831785518 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:29:48.113 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:48.109+0000 7f89d56c1700 1 --2- 172.21.15.110:0/831785518 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f89d00c80b0 0x7f89d00c3cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:48.113 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:48.109+0000 7f89d66c3700 1 --2- 172.21.15.110:0/831785518 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f89d0155570 0x7f89d014eef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:48.113 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:48.109+0000 7f89d5ec2700 1 -- 172.21.15.110:0/831785518 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f89d00c80b0 msgr2=0x7f89d00c3cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:48.113 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:48.109+0000 7f89d5ec2700 1 --2- 172.21.15.110:0/831785518 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f89d00c80b0 0x7f89d00c3cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:48.113 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:48.109+0000 7f89d5ec2700 1 -- 172.21.15.110:0/831785518 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f89d0155570 msgr2=0x7f89d014eef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:48.113 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:48.109+0000 7f89d5ec2700 1 --2- 172.21.15.110:0/831785518 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f89d0155570 0x7f89d014eef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:48.114 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:48.109+0000 7f89d5ec2700 1 -- 172.21.15.110:0/831785518 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89cc009ae0 con 0x7f89d0063430 2023-10-06T03:29:48.114 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:48.109+0000 7f89d5ec2700 1 --2- 172.21.15.110:0/831785518 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f89d0063430 0x7f89d00c6140 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f89c000c810 tx=0x7f89c0007d60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:29:48.114 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:48.109+0000 7f89c5ffb700 1 -- 172.21.15.110:0/831785518 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89c000e4a0 con 0x7f89d0063430 2023-10-06T03:29:48.114 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:48.109+0000 7f89c5ffb700 1 -- 172.21.15.110:0/831785518 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89c000e620 con 0x7f89d0063430 2023-10-06T03:29:48.115 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:48.109+0000 7f89c5ffb700 1 -- 172.21.15.110:0/831785518 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89c000e9e0 con 0x7f89d0063430 2023-10-06T03:29:48.115 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:48.109+0000 7f89d6ec4700 1 -- 172.21.15.110:0/831785518 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89d00c4a60 con 0x7f89d0063430 2023-10-06T03:29:48.115 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:48.109+0000 7f89d6ec4700 1 -- 172.21.15.110:0/831785518 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89d01670f0 con 0x7f89d0063430 2023-10-06T03:29:48.115 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:48.109+0000 7f89d6ec4700 1 -- 172.21.15.110:0/831785518 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f89d0070ff0 con 0x7f89d0063430 2023-10-06T03:29:48.115 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:48.113+0000 7f89c5ffb700 1 -- 172.21.15.110:0/831785518 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7f89c000ed50 con 0x7f89d0063430 2023-10-06T03:29:48.116 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:48.113+0000 7f89c5ffb700 1 --2- 172.21.15.110:0/831785518 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f89bc060580 0x7f89bc062a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:48.116 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:48.113+0000 7f89d56c1700 1 --2- 172.21.15.110:0/831785518 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f89bc060580 0x7f89bc062a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:48.116 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:48.113+0000 7f89c5ffb700 1 -- 172.21.15.110:0/831785518 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6809+0+0 (secure 0 0 0) 0x7f89c007c270 con 0x7f89d0063430 2023-10-06T03:29:48.117 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:48.113+0000 7f89d56c1700 1 --2- 172.21.15.110:0/831785518 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f89bc060580 0x7f89bc062a30 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f89cc009f20 tx=0x7f89cc018000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:29:48.120 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:48.117+0000 7f89c5ffb700 1 -- 172.21.15.110:0/831785518 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f89c004c370 con 0x7f89d0063430 2023-10-06T03:29:48.285 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:48.281+0000 7f89d6ec4700 1 -- 172.21.15.110:0/831785518 --> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f89d0156b80 con 0x7f89bc060580 2023-10-06T03:29:48.296 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:48.293+0000 7f89c5ffb700 1 -- 172.21.15.110:0/831785518 <== mgr.4110 v2:172.21.15.110:6832/33268 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+367361 (secure 0 0 0) 0x7f89d0156b80 con 0x7f89bc060580 2023-10-06T03:29:48.299 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-10-06T03:29:48.303 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:48.297+0000 7f89d6ec4700 1 -- 172.21.15.110:0/831785518 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f89bc060580 msgr2=0x7f89bc062a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:48.304 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:48.297+0000 7f89d6ec4700 1 --2- 172.21.15.110:0/831785518 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f89bc060580 0x7f89bc062a30 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f89cc009f20 tx=0x7f89cc018000 comp rx=0 tx=0).stop 2023-10-06T03:29:48.304 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:48.297+0000 7f89d6ec4700 1 -- 172.21.15.110:0/831785518 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f89d0063430 msgr2=0x7f89d00c6140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:48.304 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:48.297+0000 7f89d6ec4700 1 --2- 172.21.15.110:0/831785518 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f89d0063430 0x7f89d00c6140 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f89c000c810 tx=0x7f89c0007d60 comp rx=0 tx=0).stop 2023-10-06T03:29:48.304 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:48.297+0000 7f89d6ec4700 1 -- 172.21.15.110:0/831785518 shutdown_connections 2023-10-06T03:29:48.304 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:48.297+0000 7f89d6ec4700 1 --2- 172.21.15.110:0/831785518 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f89bc060580 0x7f89bc062a30 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:48.304 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:48.297+0000 7f89d6ec4700 1 --2- 172.21.15.110:0/831785518 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f89d0155570 0x7f89d014eef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:48.304 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:48.297+0000 7f89d6ec4700 1 --2- 172.21.15.110:0/831785518 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f89d00c80b0 0x7f89d00c3cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:48.304 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:48.297+0000 7f89d6ec4700 1 --2- 172.21.15.110:0/831785518 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f89d0063430 0x7f89d00c6140 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:48.304 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:48.297+0000 7f89d6ec4700 1 -- 172.21.15.110:0/831785518 >> 172.21.15.110:0/831785518 conn(0x7f89d00c8d00 msgr2=0x7f89d006a8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:29:48.304 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:48.301+0000 7f89d6ec4700 1 -- 172.21.15.110:0/831785518 shutdown_connections 2023-10-06T03:29:48.305 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:48.301+0000 7f89d6ec4700 1 -- 172.21.15.110:0/831785518 wait complete. 2023-10-06T03:29:48.305 INFO:teuthology.orchestra.run.smithi110.stderr:dumped all 2023-10-06T03:29:48.316 INFO:teuthology.orchestra.run.smithi110.stdout:{"pg_ready":true,"pg_map":{"version":34,"stamp":"2023-10-06T03:29:46.651636+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592674,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2429,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":338,"acting":338,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":296,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2398184,"kb_used_data":3496,"kb_used_omap":0,"kb_used_meta":2394624,"kb_avail":752576536,"statfs":{"total":773094113280,"available":770638372864,"internally_reserved":0,"allocated":3579904,"data_stored":2145556,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2452094976},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.066433"},"pg_stats":[{"pgid":"5.19","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.624659+0000","last_change":"2023-10-06T03:29:38.620919+0000","last_active":"2023-10-06T03:29:38.624659+0000","last_peered":"2023-10-06T03:29:38.624659+0000","last_clean":"2023-10-06T03:29:38.624659+0000","last_became_active":"2023-10-06T03:29:38.620293+0000","last_became_peered":"2023-10-06T03:29:38.620293+0000","last_unstale":"2023-10-06T03:29:38.624659+0000","last_undegraded":"2023-10-06T03:29:38.624659+0000","last_fullsized":"2023-10-06T03:29:38.624659+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:37.570693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:37.570693+0000","last_clean_scrub_stamp":"2023-10-06T03:29:37.570693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T13:27:03.828118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7,2],"acting":[1,5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"23'4","reported_seq":18,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.592373+0000","last_change":"2023-10-06T03:29:33.535641+0000","last_active":"2023-10-06T03:29:38.592373+0000","last_peered":"2023-10-06T03:29:38.592373+0000","last_clean":"2023-10-06T03:29:38.592373+0000","last_became_active":"2023-10-06T03:29:33.534410+0000","last_became_peered":"2023-10-06T03:29:33.534410+0000","last_unstale":"2023-10-06T03:29:38.592373+0000","last_undegraded":"2023-10-06T03:29:38.592373+0000","last_fullsized":"2023-10-06T03:29:38.592373+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:32.495616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:32.495616+0000","last_clean_scrub_stamp":"2023-10-06T03:29:32.495616+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T14:51:49.889964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":554,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.594781+0000","last_change":"2023-10-06T03:29:34.546243+0000","last_active":"2023-10-06T03:29:38.594781+0000","last_peered":"2023-10-06T03:29:38.594781+0000","last_clean":"2023-10-06T03:29:38.594781+0000","last_became_active":"2023-10-06T03:29:34.545923+0000","last_became_peered":"2023-10-06T03:29:34.545923+0000","last_unstale":"2023-10-06T03:29:38.594781+0000","last_undegraded":"2023-10-06T03:29:38.594781+0000","last_fullsized":"2023-10-06T03:29:38.594781+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_clean_scrub_stamp":"2023-10-06T03:29:33.500929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T15:20:45.655411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.617434+0000","last_change":"2023-10-06T03:29:38.608716+0000","last_active":"2023-10-06T03:29:38.617434+0000","last_peered":"2023-10-06T03:29:38.617434+0000","last_clean":"2023-10-06T03:29:38.617434+0000","last_became_active":"2023-10-06T03:29:38.607714+0000","last_became_peered":"2023-10-06T03:29:38.607714+0000","last_unstale":"2023-10-06T03:29:38.617434+0000","last_undegraded":"2023-10-06T03:29:38.617434+0000","last_fullsized":"2023-10-06T03:29:38.617434+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:37.570693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:37.570693+0000","last_clean_scrub_stamp":"2023-10-06T03:29:37.570693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T07:11:39.919440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1,2],"acting":[4,6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"23'1","reported_seq":15,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.591195+0000","last_change":"2023-10-06T03:29:33.538145+0000","last_active":"2023-10-06T03:29:38.591195+0000","last_peered":"2023-10-06T03:29:38.591195+0000","last_clean":"2023-10-06T03:29:38.591195+0000","last_became_active":"2023-10-06T03:29:33.536028+0000","last_became_peered":"2023-10-06T03:29:33.536028+0000","last_unstale":"2023-10-06T03:29:38.591195+0000","last_undegraded":"2023-10-06T03:29:38.591195+0000","last_fullsized":"2023-10-06T03:29:38.591195+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:32.495616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:32.495616+0000","last_clean_scrub_stamp":"2023-10-06T03:29:32.495616+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T06:15:00.105181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.589205+0000","last_change":"2023-10-06T03:29:34.546777+0000","last_active":"2023-10-06T03:29:38.589205+0000","last_peered":"2023-10-06T03:29:38.589205+0000","last_clean":"2023-10-06T03:29:38.589205+0000","last_became_active":"2023-10-06T03:29:34.545461+0000","last_became_peered":"2023-10-06T03:29:34.545461+0000","last_unstale":"2023-10-06T03:29:38.589205+0000","last_undegraded":"2023-10-06T03:29:38.589205+0000","last_fullsized":"2023-10-06T03:29:38.589205+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_clean_scrub_stamp":"2023-10-06T03:29:33.500929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T06:40:43.933850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.621795+0000","last_change":"2023-10-06T03:29:38.619453+0000","last_active":"2023-10-06T03:29:38.621795+0000","last_peered":"2023-10-06T03:29:38.621795+0000","last_clean":"2023-10-06T03:29:38.621795+0000","last_became_active":"2023-10-06T03:29:38.618690+0000","last_became_peered":"2023-10-06T03:29:38.618690+0000","last_unstale":"2023-10-06T03:29:38.621795+0000","last_undegraded":"2023-10-06T03:29:38.621795+0000","last_fullsized":"2023-10-06T03:29:38.621795+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:37.570693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:37.570693+0000","last_clean_scrub_stamp":"2023-10-06T03:29:37.570693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T06:00:28.537215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7,4],"acting":[5,0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"23'2","reported_seq":16,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.602095+0000","last_change":"2023-10-06T03:29:33.542309+0000","last_active":"2023-10-06T03:29:38.602095+0000","last_peered":"2023-10-06T03:29:38.602095+0000","last_clean":"2023-10-06T03:29:38.602095+0000","last_became_active":"2023-10-06T03:29:33.542038+0000","last_became_peered":"2023-10-06T03:29:33.542038+0000","last_unstale":"2023-10-06T03:29:38.602095+0000","last_undegraded":"2023-10-06T03:29:38.602095+0000","last_fullsized":"2023-10-06T03:29:38.602095+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:32.495616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:32.495616+0000","last_clean_scrub_stamp":"2023-10-06T03:29:32.495616+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T08:04:27.115289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.592351+0000","last_change":"2023-10-06T03:29:34.540591+0000","last_active":"2023-10-06T03:29:38.592351+0000","last_peered":"2023-10-06T03:29:38.592351+0000","last_clean":"2023-10-06T03:29:38.592351+0000","last_became_active":"2023-10-06T03:29:34.540219+0000","last_became_peered":"2023-10-06T03:29:34.540219+0000","last_unstale":"2023-10-06T03:29:38.592351+0000","last_undegraded":"2023-10-06T03:29:38.592351+0000","last_fullsized":"2023-10-06T03:29:38.592351+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_clean_scrub_stamp":"2023-10-06T03:29:33.500929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T11:28:48.945575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.626642+0000","last_change":"2023-10-06T03:29:38.624230+0000","last_active":"2023-10-06T03:29:38.626642+0000","last_peered":"2023-10-06T03:29:38.626642+0000","last_clean":"2023-10-06T03:29:38.626642+0000","last_became_active":"2023-10-06T03:29:38.623638+0000","last_became_peered":"2023-10-06T03:29:38.623638+0000","last_unstale":"2023-10-06T03:29:38.626642+0000","last_undegraded":"2023-10-06T03:29:38.626642+0000","last_fullsized":"2023-10-06T03:29:38.626642+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:37.570693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:37.570693+0000","last_clean_scrub_stamp":"2023-10-06T03:29:37.570693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T12:11:50.851770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1,6],"acting":[7,4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"23'2","reported_seq":16,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.600846+0000","last_change":"2023-10-06T03:29:33.540179+0000","last_active":"2023-10-06T03:29:38.600846+0000","last_peered":"2023-10-06T03:29:38.600846+0000","last_clean":"2023-10-06T03:29:38.600846+0000","last_became_active":"2023-10-06T03:29:33.539825+0000","last_became_peered":"2023-10-06T03:29:33.539825+0000","last_unstale":"2023-10-06T03:29:38.600846+0000","last_undegraded":"2023-10-06T03:29:38.600846+0000","last_fullsized":"2023-10-06T03:29:38.600846+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:32.495616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:32.495616+0000","last_clean_scrub_stamp":"2023-10-06T03:29:32.495616+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T11:58:53.137774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.593620+0000","last_change":"2023-10-06T03:29:34.540676+0000","last_active":"2023-10-06T03:29:38.593620+0000","last_peered":"2023-10-06T03:29:38.593620+0000","last_clean":"2023-10-06T03:29:38.593620+0000","last_became_active":"2023-10-06T03:29:34.540358+0000","last_became_peered":"2023-10-06T03:29:34.540358+0000","last_unstale":"2023-10-06T03:29:38.593620+0000","last_undegraded":"2023-10-06T03:29:38.593620+0000","last_fullsized":"2023-10-06T03:29:38.593620+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_clean_scrub_stamp":"2023-10-06T03:29:33.500929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T07:51:21.462969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.624627+0000","last_change":"2023-10-06T03:29:38.620564+0000","last_active":"2023-10-06T03:29:38.624627+0000","last_peered":"2023-10-06T03:29:38.624627+0000","last_clean":"2023-10-06T03:29:38.624627+0000","last_became_active":"2023-10-06T03:29:38.620134+0000","last_became_peered":"2023-10-06T03:29:38.620134+0000","last_unstale":"2023-10-06T03:29:38.624627+0000","last_undegraded":"2023-10-06T03:29:38.624627+0000","last_fullsized":"2023-10-06T03:29:38.624627+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:37.570693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:37.570693+0000","last_clean_scrub_stamp":"2023-10-06T03:29:37.570693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T04:19:16.510300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4,0],"acting":[1,3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"23'1","reported_seq":15,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.593175+0000","last_change":"2023-10-06T03:29:33.542267+0000","last_active":"2023-10-06T03:29:38.593175+0000","last_peered":"2023-10-06T03:29:38.593175+0000","last_clean":"2023-10-06T03:29:38.593175+0000","last_became_active":"2023-10-06T03:29:33.541987+0000","last_became_peered":"2023-10-06T03:29:33.541987+0000","last_unstale":"2023-10-06T03:29:38.593175+0000","last_undegraded":"2023-10-06T03:29:38.593175+0000","last_fullsized":"2023-10-06T03:29:38.593175+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:32.495616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:32.495616+0000","last_clean_scrub_stamp":"2023-10-06T03:29:32.495616+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T12:25:10.766051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.600288+0000","last_change":"2023-10-06T03:29:34.547283+0000","last_active":"2023-10-06T03:29:38.600288+0000","last_peered":"2023-10-06T03:29:38.600288+0000","last_clean":"2023-10-06T03:29:38.600288+0000","last_became_active":"2023-10-06T03:29:34.546966+0000","last_became_peered":"2023-10-06T03:29:34.546966+0000","last_unstale":"2023-10-06T03:29:38.600288+0000","last_undegraded":"2023-10-06T03:29:38.600288+0000","last_fullsized":"2023-10-06T03:29:38.600288+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_clean_scrub_stamp":"2023-10-06T03:29:33.500929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T07:55:08.792414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.621924+0000","last_change":"2023-10-06T03:29:38.614364+0000","last_active":"2023-10-06T03:29:38.621924+0000","last_peered":"2023-10-06T03:29:38.621924+0000","last_clean":"2023-10-06T03:29:38.621924+0000","last_became_active":"2023-10-06T03:29:38.613892+0000","last_became_peered":"2023-10-06T03:29:38.613892+0000","last_unstale":"2023-10-06T03:29:38.621924+0000","last_undegraded":"2023-10-06T03:29:38.621924+0000","last_fullsized":"2023-10-06T03:29:38.621924+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:37.570693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:37.570693+0000","last_clean_scrub_stamp":"2023-10-06T03:29:37.570693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T13:00:21.693396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2,0],"acting":[4,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"23'1","reported_seq":15,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.592761+0000","last_change":"2023-10-06T03:29:33.546282+0000","last_active":"2023-10-06T03:29:38.592761+0000","last_peered":"2023-10-06T03:29:38.592761+0000","last_clean":"2023-10-06T03:29:38.592761+0000","last_became_active":"2023-10-06T03:29:33.546039+0000","last_became_peered":"2023-10-06T03:29:33.546039+0000","last_unstale":"2023-10-06T03:29:38.592761+0000","last_undegraded":"2023-10-06T03:29:38.592761+0000","last_fullsized":"2023-10-06T03:29:38.592761+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:32.495616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:32.495616+0000","last_clean_scrub_stamp":"2023-10-06T03:29:32.495616+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T12:01:27.570873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.589771+0000","last_change":"2023-10-06T03:29:34.550953+0000","last_active":"2023-10-06T03:29:38.589771+0000","last_peered":"2023-10-06T03:29:38.589771+0000","last_clean":"2023-10-06T03:29:38.589771+0000","last_became_active":"2023-10-06T03:29:34.550614+0000","last_became_peered":"2023-10-06T03:29:34.550614+0000","last_unstale":"2023-10-06T03:29:38.589771+0000","last_undegraded":"2023-10-06T03:29:38.589771+0000","last_fullsized":"2023-10-06T03:29:38.589771+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_clean_scrub_stamp":"2023-10-06T03:29:33.500929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T11:59:02.611216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.620922+0000","last_change":"2023-10-06T03:29:38.613494+0000","last_active":"2023-10-06T03:29:38.620922+0000","last_peered":"2023-10-06T03:29:38.620922+0000","last_clean":"2023-10-06T03:29:38.620922+0000","last_became_active":"2023-10-06T03:29:38.613121+0000","last_became_peered":"2023-10-06T03:29:38.613121+0000","last_unstale":"2023-10-06T03:29:38.620922+0000","last_undegraded":"2023-10-06T03:29:38.620922+0000","last_fullsized":"2023-10-06T03:29:38.620922+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:37.570693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:37.570693+0000","last_clean_scrub_stamp":"2023-10-06T03:29:37.570693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T03:45:24.009354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3,7],"acting":[6,4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.608215+0000","last_change":"2023-10-06T03:29:33.542907+0000","last_active":"2023-10-06T03:29:38.608215+0000","last_peered":"2023-10-06T03:29:38.608215+0000","last_clean":"2023-10-06T03:29:38.608215+0000","last_became_active":"2023-10-06T03:29:33.542556+0000","last_became_peered":"2023-10-06T03:29:33.542556+0000","last_unstale":"2023-10-06T03:29:38.608215+0000","last_undegraded":"2023-10-06T03:29:38.608215+0000","last_fullsized":"2023-10-06T03:29:38.608215+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:32.495616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:32.495616+0000","last_clean_scrub_stamp":"2023-10-06T03:29:32.495616+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T03:54:50.215488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.599642+0000","last_change":"2023-10-06T03:29:34.541680+0000","last_active":"2023-10-06T03:29:38.599642+0000","last_peered":"2023-10-06T03:29:38.599642+0000","last_clean":"2023-10-06T03:29:38.599642+0000","last_became_active":"2023-10-06T03:29:34.541220+0000","last_became_peered":"2023-10-06T03:29:34.541220+0000","last_unstale":"2023-10-06T03:29:38.599642+0000","last_undegraded":"2023-10-06T03:29:38.599642+0000","last_fullsized":"2023-10-06T03:29:38.599642+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_clean_scrub_stamp":"2023-10-06T03:29:33.500929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T06:09:15.729486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.627293+0000","last_change":"2023-10-06T03:29:38.623302+0000","last_active":"2023-10-06T03:29:38.627293+0000","last_peered":"2023-10-06T03:29:38.627293+0000","last_clean":"2023-10-06T03:29:38.627293+0000","last_became_active":"2023-10-06T03:29:38.622782+0000","last_became_peered":"2023-10-06T03:29:38.622782+0000","last_unstale":"2023-10-06T03:29:38.627293+0000","last_undegraded":"2023-10-06T03:29:38.627293+0000","last_fullsized":"2023-10-06T03:29:38.627293+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:37.570693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:37.570693+0000","last_clean_scrub_stamp":"2023-10-06T03:29:37.570693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T06:00:28.537215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1,2],"acting":[0,7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":14,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.589343+0000","last_change":"2023-10-06T03:29:33.543906+0000","last_active":"2023-10-06T03:29:38.589343+0000","last_peered":"2023-10-06T03:29:38.589343+0000","last_clean":"2023-10-06T03:29:38.589343+0000","last_became_active":"2023-10-06T03:29:33.543551+0000","last_became_peered":"2023-10-06T03:29:33.543551+0000","last_unstale":"2023-10-06T03:29:38.589343+0000","last_undegraded":"2023-10-06T03:29:38.589343+0000","last_fullsized":"2023-10-06T03:29:38.589343+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:32.495616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:32.495616+0000","last_clean_scrub_stamp":"2023-10-06T03:29:32.495616+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T03:34:32.799922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.609821+0000","last_change":"2023-10-06T03:29:34.549763+0000","last_active":"2023-10-06T03:29:38.609821+0000","last_peered":"2023-10-06T03:29:38.609821+0000","last_clean":"2023-10-06T03:29:38.609821+0000","last_became_active":"2023-10-06T03:29:34.549402+0000","last_became_peered":"2023-10-06T03:29:34.549402+0000","last_unstale":"2023-10-06T03:29:38.609821+0000","last_undegraded":"2023-10-06T03:29:38.609821+0000","last_fullsized":"2023-10-06T03:29:38.609821+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_clean_scrub_stamp":"2023-10-06T03:29:33.500929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T07:14:03.653521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.588889+0000","last_change":"2023-10-06T03:29:34.548269+0000","last_active":"2023-10-06T03:29:38.588889+0000","last_peered":"2023-10-06T03:29:38.588889+0000","last_clean":"2023-10-06T03:29:38.588889+0000","last_became_active":"2023-10-06T03:29:34.547982+0000","last_became_peered":"2023-10-06T03:29:34.547982+0000","last_unstale":"2023-10-06T03:29:38.588889+0000","last_undegraded":"2023-10-06T03:29:38.588889+0000","last_fullsized":"2023-10-06T03:29:38.588889+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_clean_scrub_stamp":"2023-10-06T03:29:33.500929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T03:54:50.291053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":14,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.596539+0000","last_change":"2023-10-06T03:29:33.542538+0000","last_active":"2023-10-06T03:29:38.596539+0000","last_peered":"2023-10-06T03:29:38.596539+0000","last_clean":"2023-10-06T03:29:38.596539+0000","last_became_active":"2023-10-06T03:29:33.536222+0000","last_became_peered":"2023-10-06T03:29:33.536222+0000","last_unstale":"2023-10-06T03:29:38.596539+0000","last_undegraded":"2023-10-06T03:29:38.596539+0000","last_fullsized":"2023-10-06T03:29:38.596539+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:32.495616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:32.495616+0000","last_clean_scrub_stamp":"2023-10-06T03:29:32.495616+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T07:14:02.648208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.620952+0000","last_change":"2023-10-06T03:29:38.613462+0000","last_active":"2023-10-06T03:29:38.620952+0000","last_peered":"2023-10-06T03:29:38.620952+0000","last_clean":"2023-10-06T03:29:38.620952+0000","last_became_active":"2023-10-06T03:29:38.613096+0000","last_became_peered":"2023-10-06T03:29:38.613096+0000","last_unstale":"2023-10-06T03:29:38.620952+0000","last_undegraded":"2023-10-06T03:29:38.620952+0000","last_fullsized":"2023-10-06T03:29:38.620952+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:37.570693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:37.570693+0000","last_clean_scrub_stamp":"2023-10-06T03:29:37.570693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T11:28:53.015339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7,1],"acting":[6,4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.589101+0000","last_change":"2023-10-06T03:29:34.546990+0000","last_active":"2023-10-06T03:29:38.589101+0000","last_peered":"2023-10-06T03:29:38.589101+0000","last_clean":"2023-10-06T03:29:38.589101+0000","last_became_active":"2023-10-06T03:29:34.544561+0000","last_became_peered":"2023-10-06T03:29:34.544561+0000","last_unstale":"2023-10-06T03:29:38.589101+0000","last_undegraded":"2023-10-06T03:29:38.589101+0000","last_fullsized":"2023-10-06T03:29:38.589101+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_clean_scrub_stamp":"2023-10-06T03:29:33.500929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T09:13:47.732421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":14,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.602311+0000","last_change":"2023-10-06T03:29:33.536276+0000","last_active":"2023-10-06T03:29:38.602311+0000","last_peered":"2023-10-06T03:29:38.602311+0000","last_clean":"2023-10-06T03:29:38.602311+0000","last_became_active":"2023-10-06T03:29:33.535859+0000","last_became_peered":"2023-10-06T03:29:33.535859+0000","last_unstale":"2023-10-06T03:29:38.602311+0000","last_undegraded":"2023-10-06T03:29:38.602311+0000","last_fullsized":"2023-10-06T03:29:38.602311+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:32.495616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:32.495616+0000","last_clean_scrub_stamp":"2023-10-06T03:29:32.495616+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T07:55:07.787102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.623243+0000","last_change":"2023-10-06T03:29:38.618743+0000","last_active":"2023-10-06T03:29:38.623243+0000","last_peered":"2023-10-06T03:29:38.623243+0000","last_clean":"2023-10-06T03:29:38.623243+0000","last_became_active":"2023-10-06T03:29:38.618260+0000","last_became_peered":"2023-10-06T03:29:38.618260+0000","last_unstale":"2023-10-06T03:29:38.623243+0000","last_undegraded":"2023-10-06T03:29:38.623243+0000","last_fullsized":"2023-10-06T03:29:38.623243+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:37.570693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:37.570693+0000","last_clean_scrub_stamp":"2023-10-06T03:29:37.570693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T04:14:12.838199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6,0],"acting":[7,4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.15","version":"23'3","reported_seq":16,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.606179+0000","last_change":"2023-10-06T03:29:33.541547+0000","last_active":"2023-10-06T03:29:38.606179+0000","last_peered":"2023-10-06T03:29:38.606179+0000","last_clean":"2023-10-06T03:29:38.606179+0000","last_became_active":"2023-10-06T03:29:33.541267+0000","last_became_peered":"2023-10-06T03:29:33.541267+0000","last_unstale":"2023-10-06T03:29:38.606179+0000","last_undegraded":"2023-10-06T03:29:38.606179+0000","last_fullsized":"2023-10-06T03:29:38.606179+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:32.495616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:32.495616+0000","last_clean_scrub_stamp":"2023-10-06T03:29:32.495616+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T12:25:10.766051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.608031+0000","last_change":"2023-10-06T03:29:34.550174+0000","last_active":"2023-10-06T03:29:38.608031+0000","last_peered":"2023-10-06T03:29:38.608031+0000","last_clean":"2023-10-06T03:29:38.608031+0000","last_became_active":"2023-10-06T03:29:34.549630+0000","last_became_peered":"2023-10-06T03:29:34.549630+0000","last_unstale":"2023-10-06T03:29:38.608031+0000","last_undegraded":"2023-10-06T03:29:38.608031+0000","last_fullsized":"2023-10-06T03:29:38.608031+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_clean_scrub_stamp":"2023-10-06T03:29:33.500929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T13:02:33.356490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.620450+0000","last_change":"2023-10-06T03:29:38.614643+0000","last_active":"2023-10-06T03:29:38.620450+0000","last_peered":"2023-10-06T03:29:38.620450+0000","last_clean":"2023-10-06T03:29:38.620450+0000","last_became_active":"2023-10-06T03:29:38.614242+0000","last_became_peered":"2023-10-06T03:29:38.614242+0000","last_unstale":"2023-10-06T03:29:38.620450+0000","last_undegraded":"2023-10-06T03:29:38.620450+0000","last_fullsized":"2023-10-06T03:29:38.620450+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:37.570693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:37.570693+0000","last_clean_scrub_stamp":"2023-10-06T03:29:37.570693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T06:29:58.268896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,5],"acting":[3,0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.14","version":"23'6","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.594060+0000","last_change":"2023-10-06T03:29:33.538661+0000","last_active":"2023-10-06T03:29:38.594060+0000","last_peered":"2023-10-06T03:29:38.594060+0000","last_clean":"2023-10-06T03:29:38.594060+0000","last_became_active":"2023-10-06T03:29:33.538296+0000","last_became_peered":"2023-10-06T03:29:33.538296+0000","last_unstale":"2023-10-06T03:29:38.594060+0000","last_undegraded":"2023-10-06T03:29:38.594060+0000","last_fullsized":"2023-10-06T03:29:38.594060+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:32.495616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:32.495616+0000","last_clean_scrub_stamp":"2023-10-06T03:29:32.495616+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T10:47:52.547584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1536,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.593839+0000","last_change":"2023-10-06T03:29:34.549928+0000","last_active":"2023-10-06T03:29:38.593839+0000","last_peered":"2023-10-06T03:29:38.593839+0000","last_clean":"2023-10-06T03:29:38.593839+0000","last_became_active":"2023-10-06T03:29:34.549421+0000","last_became_peered":"2023-10-06T03:29:34.549421+0000","last_unstale":"2023-10-06T03:29:38.593839+0000","last_undegraded":"2023-10-06T03:29:38.593839+0000","last_fullsized":"2023-10-06T03:29:38.593839+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_clean_scrub_stamp":"2023-10-06T03:29:33.500929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T03:45:19.939590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.624617+0000","last_change":"2023-10-06T03:29:38.620526+0000","last_active":"2023-10-06T03:29:38.624617+0000","last_peered":"2023-10-06T03:29:38.624617+0000","last_clean":"2023-10-06T03:29:38.624617+0000","last_became_active":"2023-10-06T03:29:38.620095+0000","last_became_peered":"2023-10-06T03:29:38.620095+0000","last_unstale":"2023-10-06T03:29:38.624617+0000","last_undegraded":"2023-10-06T03:29:38.624617+0000","last_fullsized":"2023-10-06T03:29:38.624617+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:37.570693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:37.570693+0000","last_clean_scrub_stamp":"2023-10-06T03:29:37.570693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T04:00:40.252544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3,7],"acting":[1,5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.594637+0000","last_change":"2023-10-06T03:29:34.538055+0000","last_active":"2023-10-06T03:29:38.594637+0000","last_peered":"2023-10-06T03:29:38.594637+0000","last_clean":"2023-10-06T03:29:38.594637+0000","last_became_active":"2023-10-06T03:29:34.537755+0000","last_became_peered":"2023-10-06T03:29:34.537755+0000","last_unstale":"2023-10-06T03:29:38.594637+0000","last_undegraded":"2023-10-06T03:29:38.594637+0000","last_fullsized":"2023-10-06T03:29:38.594637+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_clean_scrub_stamp":"2023-10-06T03:29:33.500929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T08:21:53.296989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":16,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.610021+0000","last_change":"2023-10-06T03:29:29.387402+0000","last_active":"2023-10-06T03:29:38.610021+0000","last_peered":"2023-10-06T03:29:38.610021+0000","last_clean":"2023-10-06T03:29:38.610021+0000","last_became_active":"2023-10-06T03:29:29.386812+0000","last_became_peered":"2023-10-06T03:29:29.386812+0000","last_unstale":"2023-10-06T03:29:38.610021+0000","last_undegraded":"2023-10-06T03:29:38.610021+0000","last_fullsized":"2023-10-06T03:29:38.610021+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:28.357506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:28.357506+0000","last_clean_scrub_stamp":"2023-10-06T03:29:28.357506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T12:09:43.895810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"27'1","reported_seq":15,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:39.494514+0000","last_change":"2023-10-06T03:29:33.542307+0000","last_active":"2023-10-06T03:29:39.494514+0000","last_peered":"2023-10-06T03:29:39.494514+0000","last_clean":"2023-10-06T03:29:39.494514+0000","last_became_active":"2023-10-06T03:29:33.541670+0000","last_became_peered":"2023-10-06T03:29:33.541670+0000","last_unstale":"2023-10-06T03:29:39.494514+0000","last_undegraded":"2023-10-06T03:29:39.494514+0000","last_fullsized":"2023-10-06T03:29:39.494514+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:32.495616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:32.495616+0000","last_clean_scrub_stamp":"2023-10-06T03:29:32.495616+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T05:03:47.111820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.620435+0000","last_change":"2023-10-06T03:29:38.613923+0000","last_active":"2023-10-06T03:29:38.620435+0000","last_peered":"2023-10-06T03:29:38.620435+0000","last_clean":"2023-10-06T03:29:38.620435+0000","last_became_active":"2023-10-06T03:29:38.612452+0000","last_became_peered":"2023-10-06T03:29:38.612452+0000","last_unstale":"2023-10-06T03:29:38.620435+0000","last_undegraded":"2023-10-06T03:29:38.620435+0000","last_fullsized":"2023-10-06T03:29:38.620435+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:37.570693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:37.570693+0000","last_clean_scrub_stamp":"2023-10-06T03:29:37.570693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T05:52:46.994576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4,0],"acting":[3,1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.589121+0000","last_change":"2023-10-06T03:29:34.539246+0000","last_active":"2023-10-06T03:29:38.589121+0000","last_peered":"2023-10-06T03:29:38.589121+0000","last_clean":"2023-10-06T03:29:38.589121+0000","last_became_active":"2023-10-06T03:29:34.538925+0000","last_became_peered":"2023-10-06T03:29:34.538925+0000","last_unstale":"2023-10-06T03:29:38.589121+0000","last_undegraded":"2023-10-06T03:29:38.589121+0000","last_fullsized":"2023-10-06T03:29:38.589121+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_clean_scrub_stamp":"2023-10-06T03:29:33.500929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T03:59:12.193877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":16,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.609252+0000","last_change":"2023-10-06T03:29:29.386238+0000","last_active":"2023-10-06T03:29:38.609252+0000","last_peered":"2023-10-06T03:29:38.609252+0000","last_clean":"2023-10-06T03:29:38.609252+0000","last_became_active":"2023-10-06T03:29:29.385865+0000","last_became_peered":"2023-10-06T03:29:29.385865+0000","last_unstale":"2023-10-06T03:29:38.609252+0000","last_undegraded":"2023-10-06T03:29:38.609252+0000","last_fullsized":"2023-10-06T03:29:38.609252+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:28.357506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:28.357506+0000","last_clean_scrub_stamp":"2023-10-06T03:29:28.357506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T12:43:34.714730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"23'2","reported_seq":16,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.593260+0000","last_change":"2023-10-06T03:29:33.539135+0000","last_active":"2023-10-06T03:29:38.593260+0000","last_peered":"2023-10-06T03:29:38.593260+0000","last_clean":"2023-10-06T03:29:38.593260+0000","last_became_active":"2023-10-06T03:29:33.538818+0000","last_became_peered":"2023-10-06T03:29:33.538818+0000","last_unstale":"2023-10-06T03:29:38.593260+0000","last_undegraded":"2023-10-06T03:29:38.593260+0000","last_fullsized":"2023-10-06T03:29:38.593260+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:32.495616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:32.495616+0000","last_clean_scrub_stamp":"2023-10-06T03:29:32.495616+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T03:54:50.215488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.622166+0000","last_change":"2023-10-06T03:29:38.614527+0000","last_active":"2023-10-06T03:29:38.622166+0000","last_peered":"2023-10-06T03:29:38.622166+0000","last_clean":"2023-10-06T03:29:38.622166+0000","last_became_active":"2023-10-06T03:29:38.614090+0000","last_became_peered":"2023-10-06T03:29:38.614090+0000","last_unstale":"2023-10-06T03:29:38.622166+0000","last_undegraded":"2023-10-06T03:29:38.622166+0000","last_fullsized":"2023-10-06T03:29:38.622166+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:37.570693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:37.570693+0000","last_clean_scrub_stamp":"2023-10-06T03:29:37.570693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T05:47:25.729169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7,0],"acting":[4,3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.592223+0000","last_change":"2023-10-06T03:29:34.537232+0000","last_active":"2023-10-06T03:29:38.592223+0000","last_peered":"2023-10-06T03:29:38.592223+0000","last_clean":"2023-10-06T03:29:38.592223+0000","last_became_active":"2023-10-06T03:29:34.536984+0000","last_became_peered":"2023-10-06T03:29:34.536984+0000","last_unstale":"2023-10-06T03:29:38.592223+0000","last_undegraded":"2023-10-06T03:29:38.592223+0000","last_fullsized":"2023-10-06T03:29:38.592223+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_clean_scrub_stamp":"2023-10-06T03:29:33.500929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T03:42:52.876376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":16,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.605996+0000","last_change":"2023-10-06T03:29:29.384279+0000","last_active":"2023-10-06T03:29:38.605996+0000","last_peered":"2023-10-06T03:29:38.605996+0000","last_clean":"2023-10-06T03:29:38.605996+0000","last_became_active":"2023-10-06T03:29:29.383533+0000","last_became_peered":"2023-10-06T03:29:29.383533+0000","last_unstale":"2023-10-06T03:29:38.605996+0000","last_undegraded":"2023-10-06T03:29:38.605996+0000","last_fullsized":"2023-10-06T03:29:38.605996+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:28.357506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:28.357506+0000","last_clean_scrub_stamp":"2023-10-06T03:29:28.357506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T08:34:19.641525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.609261+0000","last_change":"2023-10-06T03:29:33.542909+0000","last_active":"2023-10-06T03:29:38.609261+0000","last_peered":"2023-10-06T03:29:38.609261+0000","last_clean":"2023-10-06T03:29:38.609261+0000","last_became_active":"2023-10-06T03:29:33.542573+0000","last_became_peered":"2023-10-06T03:29:33.542573+0000","last_unstale":"2023-10-06T03:29:38.609261+0000","last_undegraded":"2023-10-06T03:29:38.609261+0000","last_fullsized":"2023-10-06T03:29:38.609261+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:32.495616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:32.495616+0000","last_clean_scrub_stamp":"2023-10-06T03:29:32.495616+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T14:26:49.262597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.620982+0000","last_change":"2023-10-06T03:29:38.615475+0000","last_active":"2023-10-06T03:29:38.620982+0000","last_peered":"2023-10-06T03:29:38.620982+0000","last_clean":"2023-10-06T03:29:38.620982+0000","last_became_active":"2023-10-06T03:29:38.615049+0000","last_became_peered":"2023-10-06T03:29:38.615049+0000","last_unstale":"2023-10-06T03:29:38.620982+0000","last_undegraded":"2023-10-06T03:29:38.620982+0000","last_fullsized":"2023-10-06T03:29:38.620982+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:37.570693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:37.570693+0000","last_clean_scrub_stamp":"2023-10-06T03:29:37.570693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T14:34:30.474789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5,2],"acting":[6,0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.608125+0000","last_change":"2023-10-06T03:29:34.543257+0000","last_active":"2023-10-06T03:29:38.608125+0000","last_peered":"2023-10-06T03:29:38.608125+0000","last_clean":"2023-10-06T03:29:38.608125+0000","last_became_active":"2023-10-06T03:29:34.537574+0000","last_became_peered":"2023-10-06T03:29:34.537574+0000","last_unstale":"2023-10-06T03:29:38.608125+0000","last_undegraded":"2023-10-06T03:29:38.608125+0000","last_fullsized":"2023-10-06T03:29:38.608125+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_clean_scrub_stamp":"2023-10-06T03:29:33.500929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T13:34:24.699976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":16,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.609348+0000","last_change":"2023-10-06T03:29:29.382183+0000","last_active":"2023-10-06T03:29:38.609348+0000","last_peered":"2023-10-06T03:29:38.609348+0000","last_clean":"2023-10-06T03:29:38.609348+0000","last_became_active":"2023-10-06T03:29:29.381918+0000","last_became_peered":"2023-10-06T03:29:29.381918+0000","last_unstale":"2023-10-06T03:29:38.609348+0000","last_undegraded":"2023-10-06T03:29:38.609348+0000","last_fullsized":"2023-10-06T03:29:38.609348+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:28.357506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:28.357506+0000","last_clean_scrub_stamp":"2023-10-06T03:29:28.357506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T11:56:26.013716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":14,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.602189+0000","last_change":"2023-10-06T03:29:33.540375+0000","last_active":"2023-10-06T03:29:38.602189+0000","last_peered":"2023-10-06T03:29:38.602189+0000","last_clean":"2023-10-06T03:29:38.602189+0000","last_became_active":"2023-10-06T03:29:33.540016+0000","last_became_peered":"2023-10-06T03:29:33.540016+0000","last_unstale":"2023-10-06T03:29:38.602189+0000","last_undegraded":"2023-10-06T03:29:38.602189+0000","last_fullsized":"2023-10-06T03:29:38.602189+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:32.495616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:32.495616+0000","last_clean_scrub_stamp":"2023-10-06T03:29:32.495616+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T10:47:52.547584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.623995+0000","last_change":"2023-10-06T03:29:38.613181+0000","last_active":"2023-10-06T03:29:38.623995+0000","last_peered":"2023-10-06T03:29:38.623995+0000","last_clean":"2023-10-06T03:29:38.623995+0000","last_became_active":"2023-10-06T03:29:38.612787+0000","last_became_peered":"2023-10-06T03:29:38.612787+0000","last_unstale":"2023-10-06T03:29:38.623995+0000","last_undegraded":"2023-10-06T03:29:38.623995+0000","last_fullsized":"2023-10-06T03:29:38.623995+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:37.570693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:37.570693+0000","last_clean_scrub_stamp":"2023-10-06T03:29:37.570693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T07:38:31.118869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6,5],"acting":[0,1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.609205+0000","last_change":"2023-10-06T03:29:34.543207+0000","last_active":"2023-10-06T03:29:38.609205+0000","last_peered":"2023-10-06T03:29:38.609205+0000","last_clean":"2023-10-06T03:29:38.609205+0000","last_became_active":"2023-10-06T03:29:34.537291+0000","last_became_peered":"2023-10-06T03:29:34.537291+0000","last_unstale":"2023-10-06T03:29:38.609205+0000","last_undegraded":"2023-10-06T03:29:38.609205+0000","last_fullsized":"2023-10-06T03:29:38.609205+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_clean_scrub_stamp":"2023-10-06T03:29:33.500929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T06:40:43.933850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":14,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.594240+0000","last_change":"2023-10-06T03:29:33.543412+0000","last_active":"2023-10-06T03:29:38.594240+0000","last_peered":"2023-10-06T03:29:38.594240+0000","last_clean":"2023-10-06T03:29:38.594240+0000","last_became_active":"2023-10-06T03:29:33.543114+0000","last_became_peered":"2023-10-06T03:29:33.543114+0000","last_unstale":"2023-10-06T03:29:38.594240+0000","last_undegraded":"2023-10-06T03:29:38.594240+0000","last_fullsized":"2023-10-06T03:29:38.594240+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:32.495616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:32.495616+0000","last_clean_scrub_stamp":"2023-10-06T03:29:32.495616+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T15:12:57.750941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":17,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.600966+0000","last_change":"2023-10-06T03:29:29.384415+0000","last_active":"2023-10-06T03:29:38.600966+0000","last_peered":"2023-10-06T03:29:38.600966+0000","last_clean":"2023-10-06T03:29:38.600966+0000","last_became_active":"2023-10-06T03:29:29.383894+0000","last_became_peered":"2023-10-06T03:29:29.383894+0000","last_unstale":"2023-10-06T03:29:38.600966+0000","last_undegraded":"2023-10-06T03:29:38.600966+0000","last_fullsized":"2023-10-06T03:29:38.600966+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:28.357506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:28.357506+0000","last_clean_scrub_stamp":"2023-10-06T03:29:28.357506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T13:09:00.657107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.622959+0000","last_change":"2023-10-06T03:29:38.613076+0000","last_active":"2023-10-06T03:29:38.622959+0000","last_peered":"2023-10-06T03:29:38.622959+0000","last_clean":"2023-10-06T03:29:38.622959+0000","last_became_active":"2023-10-06T03:29:38.612646+0000","last_became_peered":"2023-10-06T03:29:38.612646+0000","last_unstale":"2023-10-06T03:29:38.622959+0000","last_undegraded":"2023-10-06T03:29:38.622959+0000","last_fullsized":"2023-10-06T03:29:38.622959+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:37.570693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:37.570693+0000","last_clean_scrub_stamp":"2023-10-06T03:29:37.570693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T06:07:31.871068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4,6],"acting":[0,1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.609278+0000","last_change":"2023-10-06T03:29:34.543346+0000","last_active":"2023-10-06T03:29:38.609278+0000","last_peered":"2023-10-06T03:29:38.609278+0000","last_clean":"2023-10-06T03:29:38.609278+0000","last_became_active":"2023-10-06T03:29:34.537438+0000","last_became_peered":"2023-10-06T03:29:34.537438+0000","last_unstale":"2023-10-06T03:29:38.609278+0000","last_undegraded":"2023-10-06T03:29:38.609278+0000","last_fullsized":"2023-10-06T03:29:38.609278+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_clean_scrub_stamp":"2023-10-06T03:29:33.500929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T09:13:47.732421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.609403+0000","last_change":"2023-10-06T03:29:33.544242+0000","last_active":"2023-10-06T03:29:38.609403+0000","last_peered":"2023-10-06T03:29:38.609403+0000","last_clean":"2023-10-06T03:29:38.609403+0000","last_became_active":"2023-10-06T03:29:33.543938+0000","last_became_peered":"2023-10-06T03:29:33.543938+0000","last_unstale":"2023-10-06T03:29:38.609403+0000","last_undegraded":"2023-10-06T03:29:38.609403+0000","last_fullsized":"2023-10-06T03:29:38.609403+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:32.495616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:32.495616+0000","last_clean_scrub_stamp":"2023-10-06T03:29:32.495616+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T10:09:45.061520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":16,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.599964+0000","last_change":"2023-10-06T03:29:29.606195+0000","last_active":"2023-10-06T03:29:38.599964+0000","last_peered":"2023-10-06T03:29:38.599964+0000","last_clean":"2023-10-06T03:29:38.599964+0000","last_became_active":"2023-10-06T03:29:29.605489+0000","last_became_peered":"2023-10-06T03:29:29.605489+0000","last_unstale":"2023-10-06T03:29:38.599964+0000","last_undegraded":"2023-10-06T03:29:38.599964+0000","last_fullsized":"2023-10-06T03:29:38.599964+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:28.357506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:28.357506+0000","last_clean_scrub_stamp":"2023-10-06T03:29:28.357506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T10:21:35.864895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.626333+0000","last_change":"2023-10-06T03:29:38.624434+0000","last_active":"2023-10-06T03:29:38.626333+0000","last_peered":"2023-10-06T03:29:38.626333+0000","last_clean":"2023-10-06T03:29:38.626333+0000","last_became_active":"2023-10-06T03:29:38.623676+0000","last_became_peered":"2023-10-06T03:29:38.623676+0000","last_unstale":"2023-10-06T03:29:38.626333+0000","last_undegraded":"2023-10-06T03:29:38.626333+0000","last_fullsized":"2023-10-06T03:29:38.626333+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:37.570693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:37.570693+0000","last_clean_scrub_stamp":"2023-10-06T03:29:37.570693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T12:26:20.152406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,1,7],"acting":[2,5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.599731+0000","last_change":"2023-10-06T03:29:34.550793+0000","last_active":"2023-10-06T03:29:38.599731+0000","last_peered":"2023-10-06T03:29:38.599731+0000","last_clean":"2023-10-06T03:29:38.599731+0000","last_became_active":"2023-10-06T03:29:34.550345+0000","last_became_peered":"2023-10-06T03:29:34.550345+0000","last_unstale":"2023-10-06T03:29:38.599731+0000","last_undegraded":"2023-10-06T03:29:38.599731+0000","last_fullsized":"2023-10-06T03:29:38.599731+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_clean_scrub_stamp":"2023-10-06T03:29:33.500929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T11:28:48.945575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":14,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.591494+0000","last_change":"2023-10-06T03:29:33.543304+0000","last_active":"2023-10-06T03:29:38.591494+0000","last_peered":"2023-10-06T03:29:38.591494+0000","last_clean":"2023-10-06T03:29:38.591494+0000","last_became_active":"2023-10-06T03:29:33.542483+0000","last_became_peered":"2023-10-06T03:29:33.542483+0000","last_unstale":"2023-10-06T03:29:38.591494+0000","last_undegraded":"2023-10-06T03:29:38.591494+0000","last_fullsized":"2023-10-06T03:29:38.591494+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:32.495616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:32.495616+0000","last_clean_scrub_stamp":"2023-10-06T03:29:32.495616+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T11:58:53.137774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":16,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.603498+0000","last_change":"2023-10-06T03:29:29.384353+0000","last_active":"2023-10-06T03:29:38.603498+0000","last_peered":"2023-10-06T03:29:38.603498+0000","last_clean":"2023-10-06T03:29:38.603498+0000","last_became_active":"2023-10-06T03:29:29.383884+0000","last_became_peered":"2023-10-06T03:29:29.383884+0000","last_unstale":"2023-10-06T03:29:38.603498+0000","last_undegraded":"2023-10-06T03:29:38.603498+0000","last_fullsized":"2023-10-06T03:29:38.603498+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:28.357506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:28.357506+0000","last_clean_scrub_stamp":"2023-10-06T03:29:28.357506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T14:43:09.107140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.629976+0000","last_change":"2023-10-06T03:29:38.626568+0000","last_active":"2023-10-06T03:29:38.629976+0000","last_peered":"2023-10-06T03:29:38.629976+0000","last_clean":"2023-10-06T03:29:38.629976+0000","last_became_active":"2023-10-06T03:29:38.625323+0000","last_became_peered":"2023-10-06T03:29:38.625323+0000","last_unstale":"2023-10-06T03:29:38.629976+0000","last_undegraded":"2023-10-06T03:29:38.629976+0000","last_fullsized":"2023-10-06T03:29:38.629976+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:37.570693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:37.570693+0000","last_clean_scrub_stamp":"2023-10-06T03:29:37.570693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T04:51:55.443872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0,4],"acting":[5,1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.609554+0000","last_change":"2023-10-06T03:29:34.549281+0000","last_active":"2023-10-06T03:29:38.609554+0000","last_peered":"2023-10-06T03:29:38.609554+0000","last_clean":"2023-10-06T03:29:38.609554+0000","last_became_active":"2023-10-06T03:29:34.548900+0000","last_became_peered":"2023-10-06T03:29:34.548900+0000","last_unstale":"2023-10-06T03:29:38.609554+0000","last_undegraded":"2023-10-06T03:29:38.609554+0000","last_fullsized":"2023-10-06T03:29:38.609554+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_clean_scrub_stamp":"2023-10-06T03:29:33.500929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T09:06:08.844003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":14,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.589353+0000","last_change":"2023-10-06T03:29:33.534969+0000","last_active":"2023-10-06T03:29:38.589353+0000","last_peered":"2023-10-06T03:29:38.589353+0000","last_clean":"2023-10-06T03:29:38.589353+0000","last_became_active":"2023-10-06T03:29:33.534402+0000","last_became_peered":"2023-10-06T03:29:33.534402+0000","last_unstale":"2023-10-06T03:29:38.589353+0000","last_undegraded":"2023-10-06T03:29:38.589353+0000","last_fullsized":"2023-10-06T03:29:38.589353+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:32.495616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:32.495616+0000","last_clean_scrub_stamp":"2023-10-06T03:29:32.495616+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T11:00:52.451461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":17,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.601254+0000","last_change":"2023-10-06T03:29:29.386120+0000","last_active":"2023-10-06T03:29:38.601254+0000","last_peered":"2023-10-06T03:29:38.601254+0000","last_clean":"2023-10-06T03:29:38.601254+0000","last_became_active":"2023-10-06T03:29:29.385692+0000","last_became_peered":"2023-10-06T03:29:29.385692+0000","last_unstale":"2023-10-06T03:29:38.601254+0000","last_undegraded":"2023-10-06T03:29:38.601254+0000","last_fullsized":"2023-10-06T03:29:38.601254+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:28.357506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:28.357506+0000","last_clean_scrub_stamp":"2023-10-06T03:29:28.357506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T10:21:35.864895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":169,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.603558+0000","last_change":"2023-10-06T03:29:21.978311+0000","last_active":"2023-10-06T03:29:38.603558+0000","last_peered":"2023-10-06T03:29:38.603558+0000","last_clean":"2023-10-06T03:29:38.603558+0000","last_became_active":"2023-10-06T03:29:21.978017+0000","last_became_peered":"2023-10-06T03:29:21.978017+0000","last_unstale":"2023-10-06T03:29:38.603558+0000","last_undegraded":"2023-10-06T03:29:38.603558+0000","last_fullsized":"2023-10-06T03:29:38.603558+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:20.957955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:20.957955+0000","last_clean_scrub_stamp":"2023-10-06T03:29:20.957955+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T07:33:10.429084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.614806+0000","last_change":"2023-10-06T03:29:38.612549+0000","last_active":"2023-10-06T03:29:38.614806+0000","last_peered":"2023-10-06T03:29:38.614806+0000","last_clean":"2023-10-06T03:29:38.614806+0000","last_became_active":"2023-10-06T03:29:38.611790+0000","last_became_peered":"2023-10-06T03:29:38.611790+0000","last_unstale":"2023-10-06T03:29:38.614806+0000","last_undegraded":"2023-10-06T03:29:38.614806+0000","last_fullsized":"2023-10-06T03:29:38.614806+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:37.570693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:37.570693+0000","last_clean_scrub_stamp":"2023-10-06T03:29:37.570693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T12:17:41.605048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5,3],"acting":[7,2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.608679+0000","last_change":"2023-10-06T03:29:34.545006+0000","last_active":"2023-10-06T03:29:38.608679+0000","last_peered":"2023-10-06T03:29:38.608679+0000","last_clean":"2023-10-06T03:29:38.608679+0000","last_became_active":"2023-10-06T03:29:34.544682+0000","last_became_peered":"2023-10-06T03:29:34.544682+0000","last_unstale":"2023-10-06T03:29:38.608679+0000","last_undegraded":"2023-10-06T03:29:38.608679+0000","last_fullsized":"2023-10-06T03:29:38.608679+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_clean_scrub_stamp":"2023-10-06T03:29:33.500929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T03:59:12.193877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":14,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.589559+0000","last_change":"2023-10-06T03:29:33.545053+0000","last_active":"2023-10-06T03:29:38.589559+0000","last_peered":"2023-10-06T03:29:38.589559+0000","last_clean":"2023-10-06T03:29:38.589559+0000","last_became_active":"2023-10-06T03:29:33.544749+0000","last_became_peered":"2023-10-06T03:29:33.544749+0000","last_unstale":"2023-10-06T03:29:38.589559+0000","last_undegraded":"2023-10-06T03:29:38.589559+0000","last_fullsized":"2023-10-06T03:29:38.589559+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:32.495616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:32.495616+0000","last_clean_scrub_stamp":"2023-10-06T03:29:32.495616+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T14:26:49.262597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.621981+0000","last_change":"2023-10-06T03:29:38.614480+0000","last_active":"2023-10-06T03:29:38.621981+0000","last_peered":"2023-10-06T03:29:38.621981+0000","last_clean":"2023-10-06T03:29:38.621981+0000","last_became_active":"2023-10-06T03:29:38.613976+0000","last_became_peered":"2023-10-06T03:29:38.613976+0000","last_unstale":"2023-10-06T03:29:38.621981+0000","last_undegraded":"2023-10-06T03:29:38.621981+0000","last_fullsized":"2023-10-06T03:29:38.621981+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:37.570693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:37.570693+0000","last_clean_scrub_stamp":"2023-10-06T03:29:37.570693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T09:04:13.741349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0,2],"acting":[4,5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.592722+0000","last_change":"2023-10-06T03:29:34.551311+0000","last_active":"2023-10-06T03:29:38.592722+0000","last_peered":"2023-10-06T03:29:38.592722+0000","last_clean":"2023-10-06T03:29:38.592722+0000","last_became_active":"2023-10-06T03:29:34.550961+0000","last_became_peered":"2023-10-06T03:29:34.550961+0000","last_unstale":"2023-10-06T03:29:38.592722+0000","last_undegraded":"2023-10-06T03:29:38.592722+0000","last_fullsized":"2023-10-06T03:29:38.592722+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_clean_scrub_stamp":"2023-10-06T03:29:33.500929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T14:20:50.168977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":14,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.592869+0000","last_change":"2023-10-06T03:29:33.538732+0000","last_active":"2023-10-06T03:29:38.592869+0000","last_peered":"2023-10-06T03:29:38.592869+0000","last_clean":"2023-10-06T03:29:38.592869+0000","last_became_active":"2023-10-06T03:29:33.538430+0000","last_became_peered":"2023-10-06T03:29:33.538430+0000","last_unstale":"2023-10-06T03:29:38.592869+0000","last_undegraded":"2023-10-06T03:29:38.592869+0000","last_fullsized":"2023-10-06T03:29:38.592869+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:32.495616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:32.495616+0000","last_clean_scrub_stamp":"2023-10-06T03:29:32.495616+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T12:57:50.614907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.621727+0000","last_change":"2023-10-06T03:29:38.617352+0000","last_active":"2023-10-06T03:29:38.621727+0000","last_peered":"2023-10-06T03:29:38.621727+0000","last_clean":"2023-10-06T03:29:38.621727+0000","last_became_active":"2023-10-06T03:29:38.616718+0000","last_became_peered":"2023-10-06T03:29:38.616718+0000","last_unstale":"2023-10-06T03:29:38.621727+0000","last_undegraded":"2023-10-06T03:29:38.621727+0000","last_fullsized":"2023-10-06T03:29:38.621727+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:37.570693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:37.570693+0000","last_clean_scrub_stamp":"2023-10-06T03:29:37.570693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T11:16:01.323266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6,2],"acting":[5,4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.592923+0000","last_change":"2023-10-06T03:29:34.541404+0000","last_active":"2023-10-06T03:29:38.592923+0000","last_peered":"2023-10-06T03:29:38.592923+0000","last_clean":"2023-10-06T03:29:38.592923+0000","last_became_active":"2023-10-06T03:29:34.541037+0000","last_became_peered":"2023-10-06T03:29:34.541037+0000","last_unstale":"2023-10-06T03:29:38.592923+0000","last_undegraded":"2023-10-06T03:29:38.592923+0000","last_fullsized":"2023-10-06T03:29:38.592923+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_clean_scrub_stamp":"2023-10-06T03:29:33.500929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T08:57:07.064845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.610030+0000","last_change":"2023-10-06T03:29:33.542889+0000","last_active":"2023-10-06T03:29:38.610030+0000","last_peered":"2023-10-06T03:29:38.610030+0000","last_clean":"2023-10-06T03:29:38.610030+0000","last_became_active":"2023-10-06T03:29:33.542527+0000","last_became_peered":"2023-10-06T03:29:33.542527+0000","last_unstale":"2023-10-06T03:29:38.610030+0000","last_undegraded":"2023-10-06T03:29:38.610030+0000","last_fullsized":"2023-10-06T03:29:38.610030+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:32.495616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:32.495616+0000","last_clean_scrub_stamp":"2023-10-06T03:29:32.495616+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T09:16:59.387601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.624663+0000","last_change":"2023-10-06T03:29:38.616847+0000","last_active":"2023-10-06T03:29:38.624663+0000","last_peered":"2023-10-06T03:29:38.624663+0000","last_clean":"2023-10-06T03:29:38.624663+0000","last_became_active":"2023-10-06T03:29:38.616359+0000","last_became_peered":"2023-10-06T03:29:38.616359+0000","last_unstale":"2023-10-06T03:29:38.624663+0000","last_undegraded":"2023-10-06T03:29:38.624663+0000","last_fullsized":"2023-10-06T03:29:38.624663+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:37.570693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:37.570693+0000","last_clean_scrub_stamp":"2023-10-06T03:29:37.570693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T10:33:46.594949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0,6],"acting":[1,4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.593466+0000","last_change":"2023-10-06T03:29:34.542358+0000","last_active":"2023-10-06T03:29:38.593466+0000","last_peered":"2023-10-06T03:29:38.593466+0000","last_clean":"2023-10-06T03:29:38.593466+0000","last_became_active":"2023-10-06T03:29:34.542038+0000","last_became_peered":"2023-10-06T03:29:34.542038+0000","last_unstale":"2023-10-06T03:29:38.593466+0000","last_undegraded":"2023-10-06T03:29:38.593466+0000","last_fullsized":"2023-10-06T03:29:38.593466+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_clean_scrub_stamp":"2023-10-06T03:29:33.500929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T11:19:26.387732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"23'2","reported_seq":16,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.590740+0000","last_change":"2023-10-06T03:29:33.544626+0000","last_active":"2023-10-06T03:29:38.590740+0000","last_peered":"2023-10-06T03:29:38.590740+0000","last_clean":"2023-10-06T03:29:38.590740+0000","last_became_active":"2023-10-06T03:29:33.544301+0000","last_became_peered":"2023-10-06T03:29:33.544301+0000","last_unstale":"2023-10-06T03:29:38.590740+0000","last_undegraded":"2023-10-06T03:29:38.590740+0000","last_fullsized":"2023-10-06T03:29:38.590740+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:32.495616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:32.495616+0000","last_clean_scrub_stamp":"2023-10-06T03:29:32.495616+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T09:13:04.451900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.613447+0000","last_change":"2023-10-06T03:29:38.610963+0000","last_active":"2023-10-06T03:29:38.613447+0000","last_peered":"2023-10-06T03:29:38.613447+0000","last_clean":"2023-10-06T03:29:38.613447+0000","last_became_active":"2023-10-06T03:29:38.610492+0000","last_became_peered":"2023-10-06T03:29:38.610492+0000","last_unstale":"2023-10-06T03:29:38.613447+0000","last_undegraded":"2023-10-06T03:29:38.613447+0000","last_fullsized":"2023-10-06T03:29:38.613447+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:37.570693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:37.570693+0000","last_clean_scrub_stamp":"2023-10-06T03:29:37.570693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T04:15:31.186577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5,6],"acting":[2,7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.592107+0000","last_change":"2023-10-06T03:29:34.547048+0000","last_active":"2023-10-06T03:29:38.592107+0000","last_peered":"2023-10-06T03:29:38.592107+0000","last_clean":"2023-10-06T03:29:38.592107+0000","last_became_active":"2023-10-06T03:29:34.546757+0000","last_became_peered":"2023-10-06T03:29:34.546757+0000","last_unstale":"2023-10-06T03:29:38.592107+0000","last_undegraded":"2023-10-06T03:29:38.592107+0000","last_fullsized":"2023-10-06T03:29:38.592107+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_clean_scrub_stamp":"2023-10-06T03:29:33.500929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T14:20:50.168977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":14,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.600978+0000","last_change":"2023-10-06T03:29:33.537618+0000","last_active":"2023-10-06T03:29:38.600978+0000","last_peered":"2023-10-06T03:29:38.600978+0000","last_clean":"2023-10-06T03:29:38.600978+0000","last_became_active":"2023-10-06T03:29:33.537140+0000","last_became_peered":"2023-10-06T03:29:33.537140+0000","last_unstale":"2023-10-06T03:29:38.600978+0000","last_undegraded":"2023-10-06T03:29:38.600978+0000","last_fullsized":"2023-10-06T03:29:38.600978+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:32.495616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:32.495616+0000","last_clean_scrub_stamp":"2023-10-06T03:29:32.495616+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T12:25:10.766051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.623146+0000","last_change":"2023-10-06T03:29:38.616303+0000","last_active":"2023-10-06T03:29:38.623146+0000","last_peered":"2023-10-06T03:29:38.623146+0000","last_clean":"2023-10-06T03:29:38.623146+0000","last_became_active":"2023-10-06T03:29:38.615882+0000","last_became_peered":"2023-10-06T03:29:38.615882+0000","last_unstale":"2023-10-06T03:29:38.623146+0000","last_undegraded":"2023-10-06T03:29:38.623146+0000","last_fullsized":"2023-10-06T03:29:38.623146+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:37.570693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:37.570693+0000","last_clean_scrub_stamp":"2023-10-06T03:29:37.570693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T11:19:30.457497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3,6],"acting":[2,4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.610059+0000","last_change":"2023-10-06T03:29:34.546395+0000","last_active":"2023-10-06T03:29:38.610059+0000","last_peered":"2023-10-06T03:29:38.610059+0000","last_clean":"2023-10-06T03:29:38.610059+0000","last_became_active":"2023-10-06T03:29:34.546016+0000","last_became_peered":"2023-10-06T03:29:34.546016+0000","last_unstale":"2023-10-06T03:29:38.610059+0000","last_undegraded":"2023-10-06T03:29:38.610059+0000","last_fullsized":"2023-10-06T03:29:38.610059+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_clean_scrub_stamp":"2023-10-06T03:29:33.500929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T14:51:50.895276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"23'1","reported_seq":14,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.605833+0000","last_change":"2023-10-06T03:29:33.536994+0000","last_active":"2023-10-06T03:29:38.605833+0000","last_peered":"2023-10-06T03:29:38.605833+0000","last_clean":"2023-10-06T03:29:38.605833+0000","last_became_active":"2023-10-06T03:29:33.535768+0000","last_became_peered":"2023-10-06T03:29:33.535768+0000","last_unstale":"2023-10-06T03:29:38.605833+0000","last_undegraded":"2023-10-06T03:29:38.605833+0000","last_fullsized":"2023-10-06T03:29:38.605833+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:32.495616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:32.495616+0000","last_clean_scrub_stamp":"2023-10-06T03:29:32.495616+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T06:53:50.776266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.615680+0000","last_change":"2023-10-06T03:29:38.614617+0000","last_active":"2023-10-06T03:29:38.615680+0000","last_peered":"2023-10-06T03:29:38.615680+0000","last_clean":"2023-10-06T03:29:38.615680+0000","last_became_active":"2023-10-06T03:29:38.614134+0000","last_became_peered":"2023-10-06T03:29:38.614134+0000","last_unstale":"2023-10-06T03:29:38.615680+0000","last_undegraded":"2023-10-06T03:29:38.615680+0000","last_fullsized":"2023-10-06T03:29:38.615680+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:37.570693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:37.570693+0000","last_clean_scrub_stamp":"2023-10-06T03:29:37.570693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T07:51:25.532733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5,6],"acting":[2,0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.592039+0000","last_change":"2023-10-06T03:29:34.535355+0000","last_active":"2023-10-06T03:29:38.592039+0000","last_peered":"2023-10-06T03:29:38.592039+0000","last_clean":"2023-10-06T03:29:38.592039+0000","last_became_active":"2023-10-06T03:29:34.534944+0000","last_became_peered":"2023-10-06T03:29:34.534944+0000","last_unstale":"2023-10-06T03:29:38.592039+0000","last_undegraded":"2023-10-06T03:29:38.592039+0000","last_fullsized":"2023-10-06T03:29:38.592039+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_clean_scrub_stamp":"2023-10-06T03:29:33.500929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T09:39:33.168497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"23'2","reported_seq":15,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.605956+0000","last_change":"2023-10-06T03:29:33.545575+0000","last_active":"2023-10-06T03:29:38.605956+0000","last_peered":"2023-10-06T03:29:38.605956+0000","last_clean":"2023-10-06T03:29:38.605956+0000","last_became_active":"2023-10-06T03:29:33.545110+0000","last_became_peered":"2023-10-06T03:29:33.545110+0000","last_unstale":"2023-10-06T03:29:38.605956+0000","last_undegraded":"2023-10-06T03:29:38.605956+0000","last_fullsized":"2023-10-06T03:29:38.605956+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:32.495616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:32.495616+0000","last_clean_scrub_stamp":"2023-10-06T03:29:32.495616+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T14:51:49.889964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.623767+0000","last_change":"2023-10-06T03:29:38.620856+0000","last_active":"2023-10-06T03:29:38.623767+0000","last_peered":"2023-10-06T03:29:38.623767+0000","last_clean":"2023-10-06T03:29:38.623767+0000","last_became_active":"2023-10-06T03:29:38.620374+0000","last_became_peered":"2023-10-06T03:29:38.620374+0000","last_unstale":"2023-10-06T03:29:38.623767+0000","last_undegraded":"2023-10-06T03:29:38.623767+0000","last_fullsized":"2023-10-06T03:29:38.623767+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:37.570693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:37.570693+0000","last_clean_scrub_stamp":"2023-10-06T03:29:37.570693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T08:05:22.622058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1,5],"acting":[2,0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.600871+0000","last_change":"2023-10-06T03:29:34.537767+0000","last_active":"2023-10-06T03:29:38.600871+0000","last_peered":"2023-10-06T03:29:38.600871+0000","last_clean":"2023-10-06T03:29:38.600871+0000","last_became_active":"2023-10-06T03:29:34.537414+0000","last_became_peered":"2023-10-06T03:29:34.537414+0000","last_unstale":"2023-10-06T03:29:38.600871+0000","last_undegraded":"2023-10-06T03:29:38.600871+0000","last_fullsized":"2023-10-06T03:29:38.600871+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_clean_scrub_stamp":"2023-10-06T03:29:33.500929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T08:29:43.429905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"23'1","reported_seq":14,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.606271+0000","last_change":"2023-10-06T03:29:33.543577+0000","last_active":"2023-10-06T03:29:38.606271+0000","last_peered":"2023-10-06T03:29:38.606271+0000","last_clean":"2023-10-06T03:29:38.606271+0000","last_became_active":"2023-10-06T03:29:33.543265+0000","last_became_peered":"2023-10-06T03:29:33.543265+0000","last_unstale":"2023-10-06T03:29:38.606271+0000","last_undegraded":"2023-10-06T03:29:38.606271+0000","last_fullsized":"2023-10-06T03:29:38.606271+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:32.495616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:32.495616+0000","last_clean_scrub_stamp":"2023-10-06T03:29:32.495616+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T09:06:07.838690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.623296+0000","last_change":"2023-10-06T03:29:38.619910+0000","last_active":"2023-10-06T03:29:38.623296+0000","last_peered":"2023-10-06T03:29:38.623296+0000","last_clean":"2023-10-06T03:29:38.623296+0000","last_became_active":"2023-10-06T03:29:38.619517+0000","last_became_peered":"2023-10-06T03:29:38.619517+0000","last_unstale":"2023-10-06T03:29:38.623296+0000","last_undegraded":"2023-10-06T03:29:38.623296+0000","last_fullsized":"2023-10-06T03:29:38.623296+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:37.570693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:37.570693+0000","last_clean_scrub_stamp":"2023-10-06T03:29:37.570693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T04:29:21.981307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3,4],"acting":[7,6,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.627322+0000","last_change":"2023-10-06T03:29:38.623954+0000","last_active":"2023-10-06T03:29:38.627322+0000","last_peered":"2023-10-06T03:29:38.627322+0000","last_clean":"2023-10-06T03:29:38.627322+0000","last_became_active":"2023-10-06T03:29:38.623537+0000","last_became_peered":"2023-10-06T03:29:38.623537+0000","last_unstale":"2023-10-06T03:29:38.627322+0000","last_undegraded":"2023-10-06T03:29:38.627322+0000","last_fullsized":"2023-10-06T03:29:38.627322+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:37.570693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:37.570693+0000","last_clean_scrub_stamp":"2023-10-06T03:29:37.570693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T07:38:31.118869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1,6],"acting":[5,3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"23'8","reported_seq":21,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.609831+0000","last_change":"2023-10-06T03:29:33.538136+0000","last_active":"2023-10-06T03:29:38.609831+0000","last_peered":"2023-10-06T03:29:38.609831+0000","last_clean":"2023-10-06T03:29:38.609831+0000","last_became_active":"2023-10-06T03:29:33.537656+0000","last_became_peered":"2023-10-06T03:29:33.537656+0000","last_unstale":"2023-10-06T03:29:38.609831+0000","last_undegraded":"2023-10-06T03:29:38.609831+0000","last_fullsized":"2023-10-06T03:29:38.609831+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:32.495616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:32.495616+0000","last_clean_scrub_stamp":"2023-10-06T03:29:32.495616+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T07:23:17.430374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.592883+0000","last_change":"2023-10-06T03:29:34.546204+0000","last_active":"2023-10-06T03:29:38.592883+0000","last_peered":"2023-10-06T03:29:38.592883+0000","last_clean":"2023-10-06T03:29:38.592883+0000","last_became_active":"2023-10-06T03:29:34.545585+0000","last_became_peered":"2023-10-06T03:29:34.545585+0000","last_unstale":"2023-10-06T03:29:38.592883+0000","last_undegraded":"2023-10-06T03:29:38.592883+0000","last_fullsized":"2023-10-06T03:29:38.592883+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_clean_scrub_stamp":"2023-10-06T03:29:33.500929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T05:40:45.388611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.620710+0000","last_change":"2023-10-06T03:29:38.612955+0000","last_active":"2023-10-06T03:29:38.620710+0000","last_peered":"2023-10-06T03:29:38.620710+0000","last_clean":"2023-10-06T03:29:38.620710+0000","last_became_active":"2023-10-06T03:29:38.612404+0000","last_became_peered":"2023-10-06T03:29:38.612404+0000","last_unstale":"2023-10-06T03:29:38.620710+0000","last_undegraded":"2023-10-06T03:29:38.620710+0000","last_fullsized":"2023-10-06T03:29:38.620710+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:37.570693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:37.570693+0000","last_clean_scrub_stamp":"2023-10-06T03:29:37.570693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T08:25:07.069848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7,0],"acting":[3,1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.603401+0000","last_change":"2023-10-06T03:29:33.545324+0000","last_active":"2023-10-06T03:29:38.603401+0000","last_peered":"2023-10-06T03:29:38.603401+0000","last_clean":"2023-10-06T03:29:38.603401+0000","last_became_active":"2023-10-06T03:29:33.545029+0000","last_became_peered":"2023-10-06T03:29:33.545029+0000","last_unstale":"2023-10-06T03:29:38.603401+0000","last_undegraded":"2023-10-06T03:29:38.603401+0000","last_fullsized":"2023-10-06T03:29:38.603401+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:32.495616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:32.495616+0000","last_clean_scrub_stamp":"2023-10-06T03:29:32.495616+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T07:14:02.648208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.599807+0000","last_change":"2023-10-06T03:29:34.535716+0000","last_active":"2023-10-06T03:29:38.599807+0000","last_peered":"2023-10-06T03:29:38.599807+0000","last_clean":"2023-10-06T03:29:38.599807+0000","last_became_active":"2023-10-06T03:29:34.535107+0000","last_became_peered":"2023-10-06T03:29:34.535107+0000","last_unstale":"2023-10-06T03:29:38.599807+0000","last_undegraded":"2023-10-06T03:29:38.599807+0000","last_fullsized":"2023-10-06T03:29:38.599807+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_clean_scrub_stamp":"2023-10-06T03:29:33.500929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T11:19:26.387732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":14,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.591766+0000","last_change":"2023-10-06T03:29:33.541792+0000","last_active":"2023-10-06T03:29:38.591766+0000","last_peered":"2023-10-06T03:29:38.591766+0000","last_clean":"2023-10-06T03:29:38.591766+0000","last_became_active":"2023-10-06T03:29:33.535547+0000","last_became_peered":"2023-10-06T03:29:33.535547+0000","last_unstale":"2023-10-06T03:29:38.591766+0000","last_undegraded":"2023-10-06T03:29:38.591766+0000","last_fullsized":"2023-10-06T03:29:38.591766+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:32.495616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:32.495616+0000","last_clean_scrub_stamp":"2023-10-06T03:29:32.495616+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T07:55:07.787102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.602006+0000","last_change":"2023-10-06T03:29:34.540666+0000","last_active":"2023-10-06T03:29:38.602006+0000","last_peered":"2023-10-06T03:29:38.602006+0000","last_clean":"2023-10-06T03:29:38.602006+0000","last_became_active":"2023-10-06T03:29:34.540176+0000","last_became_peered":"2023-10-06T03:29:34.540176+0000","last_unstale":"2023-10-06T03:29:38.602006+0000","last_undegraded":"2023-10-06T03:29:38.602006+0000","last_fullsized":"2023-10-06T03:29:38.602006+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_clean_scrub_stamp":"2023-10-06T03:29:33.500929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T14:11:35.669872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.617535+0000","last_change":"2023-10-06T03:29:38.610899+0000","last_active":"2023-10-06T03:29:38.617535+0000","last_peered":"2023-10-06T03:29:38.617535+0000","last_clean":"2023-10-06T03:29:38.617535+0000","last_became_active":"2023-10-06T03:29:38.610494+0000","last_became_peered":"2023-10-06T03:29:38.610494+0000","last_unstale":"2023-10-06T03:29:38.617535+0000","last_undegraded":"2023-10-06T03:29:38.617535+0000","last_fullsized":"2023-10-06T03:29:38.617535+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:37.570693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:37.570693+0000","last_clean_scrub_stamp":"2023-10-06T03:29:37.570693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T08:19:47.139035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2,1],"acting":[3,7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"23'2","reported_seq":15,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.604109+0000","last_change":"2023-10-06T03:29:33.544328+0000","last_active":"2023-10-06T03:29:38.604109+0000","last_peered":"2023-10-06T03:29:38.604109+0000","last_clean":"2023-10-06T03:29:38.604109+0000","last_became_active":"2023-10-06T03:29:33.544010+0000","last_became_peered":"2023-10-06T03:29:33.544010+0000","last_unstale":"2023-10-06T03:29:38.604109+0000","last_undegraded":"2023-10-06T03:29:38.604109+0000","last_fullsized":"2023-10-06T03:29:38.604109+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:32.495616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:32.495616+0000","last_clean_scrub_stamp":"2023-10-06T03:29:32.495616+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T11:17:52.119337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.592119+0000","last_change":"2023-10-06T03:29:34.551765+0000","last_active":"2023-10-06T03:29:38.592119+0000","last_peered":"2023-10-06T03:29:38.592119+0000","last_clean":"2023-10-06T03:29:38.592119+0000","last_became_active":"2023-10-06T03:29:34.551437+0000","last_became_peered":"2023-10-06T03:29:34.551437+0000","last_unstale":"2023-10-06T03:29:38.592119+0000","last_undegraded":"2023-10-06T03:29:38.592119+0000","last_fullsized":"2023-10-06T03:29:38.592119+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_clean_scrub_stamp":"2023-10-06T03:29:33.500929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T11:47:20.603758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.629984+0000","last_change":"2023-10-06T03:29:38.626781+0000","last_active":"2023-10-06T03:29:38.629984+0000","last_peered":"2023-10-06T03:29:38.629984+0000","last_clean":"2023-10-06T03:29:38.629984+0000","last_became_active":"2023-10-06T03:29:38.625489+0000","last_became_peered":"2023-10-06T03:29:38.625489+0000","last_unstale":"2023-10-06T03:29:38.629984+0000","last_undegraded":"2023-10-06T03:29:38.629984+0000","last_fullsized":"2023-10-06T03:29:38.629984+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:37.570693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:37.570693+0000","last_clean_scrub_stamp":"2023-10-06T03:29:37.570693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T06:13:47.912662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3,0],"acting":[5,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.20","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.609684+0000","last_change":"2023-10-06T03:29:34.546226+0000","last_active":"2023-10-06T03:29:38.609684+0000","last_peered":"2023-10-06T03:29:38.609684+0000","last_clean":"2023-10-06T03:29:38.609684+0000","last_became_active":"2023-10-06T03:29:34.545400+0000","last_became_peered":"2023-10-06T03:29:34.545400+0000","last_unstale":"2023-10-06T03:29:38.609684+0000","last_undegraded":"2023-10-06T03:29:38.609684+0000","last_fullsized":"2023-10-06T03:29:38.609684+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_clean_scrub_stamp":"2023-10-06T03:29:33.500929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T07:55:08.792414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.21","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.588987+0000","last_change":"2023-10-06T03:29:34.551672+0000","last_active":"2023-10-06T03:29:38.588987+0000","last_peered":"2023-10-06T03:29:38.588987+0000","last_clean":"2023-10-06T03:29:38.588987+0000","last_became_active":"2023-10-06T03:29:34.551337+0000","last_became_peered":"2023-10-06T03:29:34.551337+0000","last_unstale":"2023-10-06T03:29:38.588987+0000","last_undegraded":"2023-10-06T03:29:38.588987+0000","last_fullsized":"2023-10-06T03:29:38.588987+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_clean_scrub_stamp":"2023-10-06T03:29:33.500929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T05:15:29.367282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.22","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.592542+0000","last_change":"2023-10-06T03:29:34.549749+0000","last_active":"2023-10-06T03:29:38.592542+0000","last_peered":"2023-10-06T03:29:38.592542+0000","last_clean":"2023-10-06T03:29:38.592542+0000","last_became_active":"2023-10-06T03:29:34.549468+0000","last_became_peered":"2023-10-06T03:29:34.549468+0000","last_unstale":"2023-10-06T03:29:38.592542+0000","last_undegraded":"2023-10-06T03:29:38.592542+0000","last_fullsized":"2023-10-06T03:29:38.592542+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_clean_scrub_stamp":"2023-10-06T03:29:33.500929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T14:11:35.669872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.23","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.590075+0000","last_change":"2023-10-06T03:29:34.545511+0000","last_active":"2023-10-06T03:29:38.590075+0000","last_peered":"2023-10-06T03:29:38.590075+0000","last_clean":"2023-10-06T03:29:38.590075+0000","last_became_active":"2023-10-06T03:29:34.545183+0000","last_became_peered":"2023-10-06T03:29:34.545183+0000","last_unstale":"2023-10-06T03:29:38.590075+0000","last_undegraded":"2023-10-06T03:29:38.590075+0000","last_fullsized":"2023-10-06T03:29:38.590075+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_clean_scrub_stamp":"2023-10-06T03:29:33.500929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T07:54:17.345077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.24","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.609286+0000","last_change":"2023-10-06T03:29:34.543928+0000","last_active":"2023-10-06T03:29:38.609286+0000","last_peered":"2023-10-06T03:29:38.609286+0000","last_clean":"2023-10-06T03:29:38.609286+0000","last_became_active":"2023-10-06T03:29:34.543604+0000","last_became_peered":"2023-10-06T03:29:34.543604+0000","last_unstale":"2023-10-06T03:29:38.609286+0000","last_undegraded":"2023-10-06T03:29:38.609286+0000","last_fullsized":"2023-10-06T03:29:38.609286+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_clean_scrub_stamp":"2023-10-06T03:29:33.500929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T09:15:40.932270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.25","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.601918+0000","last_change":"2023-10-06T03:29:34.540446+0000","last_active":"2023-10-06T03:29:38.601918+0000","last_peered":"2023-10-06T03:29:38.601918+0000","last_clean":"2023-10-06T03:29:38.601918+0000","last_became_active":"2023-10-06T03:29:34.539827+0000","last_became_peered":"2023-10-06T03:29:34.539827+0000","last_unstale":"2023-10-06T03:29:38.601918+0000","last_undegraded":"2023-10-06T03:29:38.601918+0000","last_fullsized":"2023-10-06T03:29:38.601918+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_clean_scrub_stamp":"2023-10-06T03:29:33.500929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T07:28:42.661588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.26","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.609415+0000","last_change":"2023-10-06T03:29:34.538237+0000","last_active":"2023-10-06T03:29:38.609415+0000","last_peered":"2023-10-06T03:29:38.609415+0000","last_clean":"2023-10-06T03:29:38.609415+0000","last_became_active":"2023-10-06T03:29:34.537636+0000","last_became_peered":"2023-10-06T03:29:34.537636+0000","last_unstale":"2023-10-06T03:29:38.609415+0000","last_undegraded":"2023-10-06T03:29:38.609415+0000","last_fullsized":"2023-10-06T03:29:38.609415+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_clean_scrub_stamp":"2023-10-06T03:29:33.500929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T05:09:24.657371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.27","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.608601+0000","last_change":"2023-10-06T03:29:34.549127+0000","last_active":"2023-10-06T03:29:38.608601+0000","last_peered":"2023-10-06T03:29:38.608601+0000","last_clean":"2023-10-06T03:29:38.608601+0000","last_became_active":"2023-10-06T03:29:34.548839+0000","last_became_peered":"2023-10-06T03:29:34.548839+0000","last_unstale":"2023-10-06T03:29:38.608601+0000","last_undegraded":"2023-10-06T03:29:38.608601+0000","last_fullsized":"2023-10-06T03:29:38.608601+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_clean_scrub_stamp":"2023-10-06T03:29:33.500929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T08:03:23.482409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.28","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.603462+0000","last_change":"2023-10-06T03:29:34.544177+0000","last_active":"2023-10-06T03:29:38.603462+0000","last_peered":"2023-10-06T03:29:38.603462+0000","last_clean":"2023-10-06T03:29:38.603462+0000","last_became_active":"2023-10-06T03:29:34.543872+0000","last_became_peered":"2023-10-06T03:29:34.543872+0000","last_unstale":"2023-10-06T03:29:38.603462+0000","last_undegraded":"2023-10-06T03:29:38.603462+0000","last_fullsized":"2023-10-06T03:29:38.603462+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_clean_scrub_stamp":"2023-10-06T03:29:33.500929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T06:09:15.729486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.29","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.588861+0000","last_change":"2023-10-06T03:29:34.540282+0000","last_active":"2023-10-06T03:29:38.588861+0000","last_peered":"2023-10-06T03:29:38.588861+0000","last_clean":"2023-10-06T03:29:38.588861+0000","last_became_active":"2023-10-06T03:29:34.539988+0000","last_became_peered":"2023-10-06T03:29:34.539988+0000","last_unstale":"2023-10-06T03:29:38.588861+0000","last_undegraded":"2023-10-06T03:29:38.588861+0000","last_fullsized":"2023-10-06T03:29:38.588861+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_clean_scrub_stamp":"2023-10-06T03:29:33.500929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T07:36:17.771693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.2a","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.591640+0000","last_change":"2023-10-06T03:29:34.547360+0000","last_active":"2023-10-06T03:29:38.591640+0000","last_peered":"2023-10-06T03:29:38.591640+0000","last_clean":"2023-10-06T03:29:38.591640+0000","last_became_active":"2023-10-06T03:29:34.547086+0000","last_became_peered":"2023-10-06T03:29:34.547086+0000","last_unstale":"2023-10-06T03:29:38.591640+0000","last_undegraded":"2023-10-06T03:29:38.591640+0000","last_fullsized":"2023-10-06T03:29:38.591640+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_clean_scrub_stamp":"2023-10-06T03:29:33.500929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T07:25:41.452975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2b","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.604010+0000","last_change":"2023-10-06T03:29:34.544346+0000","last_active":"2023-10-06T03:29:38.604010+0000","last_peered":"2023-10-06T03:29:38.604010+0000","last_clean":"2023-10-06T03:29:38.604010+0000","last_became_active":"2023-10-06T03:29:34.543976+0000","last_became_peered":"2023-10-06T03:29:34.543976+0000","last_unstale":"2023-10-06T03:29:38.604010+0000","last_undegraded":"2023-10-06T03:29:38.604010+0000","last_fullsized":"2023-10-06T03:29:38.604010+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_clean_scrub_stamp":"2023-10-06T03:29:33.500929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T14:11:35.669872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.2c","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.593246+0000","last_change":"2023-10-06T03:29:34.549013+0000","last_active":"2023-10-06T03:29:38.593246+0000","last_peered":"2023-10-06T03:29:38.593246+0000","last_clean":"2023-10-06T03:29:38.593246+0000","last_became_active":"2023-10-06T03:29:34.548707+0000","last_became_peered":"2023-10-06T03:29:34.548707+0000","last_unstale":"2023-10-06T03:29:38.593246+0000","last_undegraded":"2023-10-06T03:29:38.593246+0000","last_fullsized":"2023-10-06T03:29:38.593246+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_clean_scrub_stamp":"2023-10-06T03:29:33.500929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T14:34:26.405025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2d","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.598606+0000","last_change":"2023-10-06T03:29:34.545639+0000","last_active":"2023-10-06T03:29:38.598606+0000","last_peered":"2023-10-06T03:29:38.598606+0000","last_clean":"2023-10-06T03:29:38.598606+0000","last_became_active":"2023-10-06T03:29:34.545316+0000","last_became_peered":"2023-10-06T03:29:34.545316+0000","last_unstale":"2023-10-06T03:29:38.598606+0000","last_undegraded":"2023-10-06T03:29:38.598606+0000","last_fullsized":"2023-10-06T03:29:38.598606+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_clean_scrub_stamp":"2023-10-06T03:29:33.500929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T11:58:54.143086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2e","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.592479+0000","last_change":"2023-10-06T03:29:34.532132+0000","last_active":"2023-10-06T03:29:38.592479+0000","last_peered":"2023-10-06T03:29:38.592479+0000","last_clean":"2023-10-06T03:29:38.592479+0000","last_became_active":"2023-10-06T03:29:34.531862+0000","last_became_peered":"2023-10-06T03:29:34.531862+0000","last_unstale":"2023-10-06T03:29:38.592479+0000","last_undegraded":"2023-10-06T03:29:38.592479+0000","last_fullsized":"2023-10-06T03:29:38.592479+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_clean_scrub_stamp":"2023-10-06T03:29:33.500929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T07:25:41.452975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2f","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.592223+0000","last_change":"2023-10-06T03:29:34.540499+0000","last_active":"2023-10-06T03:29:38.592223+0000","last_peered":"2023-10-06T03:29:38.592223+0000","last_clean":"2023-10-06T03:29:38.592223+0000","last_became_active":"2023-10-06T03:29:34.539154+0000","last_became_peered":"2023-10-06T03:29:34.539154+0000","last_unstale":"2023-10-06T03:29:38.592223+0000","last_undegraded":"2023-10-06T03:29:38.592223+0000","last_fullsized":"2023-10-06T03:29:38.592223+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_clean_scrub_stamp":"2023-10-06T03:29:33.500929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T15:20:45.655411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.30","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.594589+0000","last_change":"2023-10-06T03:29:34.532489+0000","last_active":"2023-10-06T03:29:38.594589+0000","last_peered":"2023-10-06T03:29:38.594589+0000","last_clean":"2023-10-06T03:29:38.594589+0000","last_became_active":"2023-10-06T03:29:34.532153+0000","last_became_peered":"2023-10-06T03:29:34.532153+0000","last_unstale":"2023-10-06T03:29:38.594589+0000","last_undegraded":"2023-10-06T03:29:38.594589+0000","last_fullsized":"2023-10-06T03:29:38.594589+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_clean_scrub_stamp":"2023-10-06T03:29:33.500929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T08:05:18.552294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.31","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.603291+0000","last_change":"2023-10-06T03:29:34.551157+0000","last_active":"2023-10-06T03:29:38.603291+0000","last_peered":"2023-10-06T03:29:38.603291+0000","last_clean":"2023-10-06T03:29:38.603291+0000","last_became_active":"2023-10-06T03:29:34.550852+0000","last_became_peered":"2023-10-06T03:29:34.550852+0000","last_unstale":"2023-10-06T03:29:38.603291+0000","last_undegraded":"2023-10-06T03:29:38.603291+0000","last_fullsized":"2023-10-06T03:29:38.603291+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_clean_scrub_stamp":"2023-10-06T03:29:33.500929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T08:29:43.429905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.32","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.600883+0000","last_change":"2023-10-06T03:29:34.539103+0000","last_active":"2023-10-06T03:29:38.600883+0000","last_peered":"2023-10-06T03:29:38.600883+0000","last_clean":"2023-10-06T03:29:38.600883+0000","last_became_active":"2023-10-06T03:29:34.538817+0000","last_became_peered":"2023-10-06T03:29:34.538817+0000","last_unstale":"2023-10-06T03:29:38.600883+0000","last_undegraded":"2023-10-06T03:29:38.600883+0000","last_fullsized":"2023-10-06T03:29:38.600883+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_clean_scrub_stamp":"2023-10-06T03:29:33.500929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T07:07:03.705326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.33","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.599036+0000","last_change":"2023-10-06T03:29:34.538671+0000","last_active":"2023-10-06T03:29:38.599036+0000","last_peered":"2023-10-06T03:29:38.599036+0000","last_clean":"2023-10-06T03:29:38.599036+0000","last_became_active":"2023-10-06T03:29:34.538375+0000","last_became_peered":"2023-10-06T03:29:34.538375+0000","last_unstale":"2023-10-06T03:29:38.599036+0000","last_undegraded":"2023-10-06T03:29:38.599036+0000","last_fullsized":"2023-10-06T03:29:38.599036+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_clean_scrub_stamp":"2023-10-06T03:29:33.500929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T07:14:03.653521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.34","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.600214+0000","last_change":"2023-10-06T03:29:34.535139+0000","last_active":"2023-10-06T03:29:38.600214+0000","last_peered":"2023-10-06T03:29:38.600214+0000","last_clean":"2023-10-06T03:29:38.600214+0000","last_became_active":"2023-10-06T03:29:34.534756+0000","last_became_peered":"2023-10-06T03:29:34.534756+0000","last_unstale":"2023-10-06T03:29:38.600214+0000","last_undegraded":"2023-10-06T03:29:38.600214+0000","last_fullsized":"2023-10-06T03:29:38.600214+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_clean_scrub_stamp":"2023-10-06T03:29:33.500929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T14:51:50.895276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.35","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.592690+0000","last_change":"2023-10-06T03:29:34.550626+0000","last_active":"2023-10-06T03:29:38.592690+0000","last_peered":"2023-10-06T03:29:38.592690+0000","last_clean":"2023-10-06T03:29:38.592690+0000","last_became_active":"2023-10-06T03:29:34.550289+0000","last_became_peered":"2023-10-06T03:29:34.550289+0000","last_unstale":"2023-10-06T03:29:38.592690+0000","last_undegraded":"2023-10-06T03:29:38.592690+0000","last_fullsized":"2023-10-06T03:29:38.592690+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_clean_scrub_stamp":"2023-10-06T03:29:33.500929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T12:44:47.758971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.36","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.602276+0000","last_change":"2023-10-06T03:29:34.541344+0000","last_active":"2023-10-06T03:29:38.602276+0000","last_peered":"2023-10-06T03:29:38.602276+0000","last_clean":"2023-10-06T03:29:38.602276+0000","last_became_active":"2023-10-06T03:29:34.540994+0000","last_became_peered":"2023-10-06T03:29:34.540994+0000","last_unstale":"2023-10-06T03:29:38.602276+0000","last_undegraded":"2023-10-06T03:29:38.602276+0000","last_fullsized":"2023-10-06T03:29:38.602276+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_clean_scrub_stamp":"2023-10-06T03:29:33.500929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T05:31:20.131568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.37","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.592106+0000","last_change":"2023-10-06T03:29:34.540530+0000","last_active":"2023-10-06T03:29:38.592106+0000","last_peered":"2023-10-06T03:29:38.592106+0000","last_clean":"2023-10-06T03:29:38.592106+0000","last_became_active":"2023-10-06T03:29:34.539518+0000","last_became_peered":"2023-10-06T03:29:34.539518+0000","last_unstale":"2023-10-06T03:29:38.592106+0000","last_undegraded":"2023-10-06T03:29:38.592106+0000","last_fullsized":"2023-10-06T03:29:38.592106+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_clean_scrub_stamp":"2023-10-06T03:29:33.500929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T14:34:26.405025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.38","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.600763+0000","last_change":"2023-10-06T03:29:34.537333+0000","last_active":"2023-10-06T03:29:38.600763+0000","last_peered":"2023-10-06T03:29:38.600763+0000","last_clean":"2023-10-06T03:29:38.600763+0000","last_became_active":"2023-10-06T03:29:34.536851+0000","last_became_peered":"2023-10-06T03:29:34.536851+0000","last_unstale":"2023-10-06T03:29:38.600763+0000","last_undegraded":"2023-10-06T03:29:38.600763+0000","last_fullsized":"2023-10-06T03:29:38.600763+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_clean_scrub_stamp":"2023-10-06T03:29:33.500929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T14:20:50.168977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.39","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.603186+0000","last_change":"2023-10-06T03:29:34.551594+0000","last_active":"2023-10-06T03:29:38.603186+0000","last_peered":"2023-10-06T03:29:38.603186+0000","last_clean":"2023-10-06T03:29:38.603186+0000","last_became_active":"2023-10-06T03:29:34.551142+0000","last_became_peered":"2023-10-06T03:29:34.551142+0000","last_unstale":"2023-10-06T03:29:38.603186+0000","last_undegraded":"2023-10-06T03:29:38.603186+0000","last_fullsized":"2023-10-06T03:29:38.603186+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_clean_scrub_stamp":"2023-10-06T03:29:33.500929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T07:28:42.661588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3a","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.592149+0000","last_change":"2023-10-06T03:29:34.542011+0000","last_active":"2023-10-06T03:29:38.592149+0000","last_peered":"2023-10-06T03:29:38.592149+0000","last_clean":"2023-10-06T03:29:38.592149+0000","last_became_active":"2023-10-06T03:29:34.540577+0000","last_became_peered":"2023-10-06T03:29:34.540577+0000","last_unstale":"2023-10-06T03:29:38.592149+0000","last_undegraded":"2023-10-06T03:29:38.592149+0000","last_fullsized":"2023-10-06T03:29:38.592149+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_clean_scrub_stamp":"2023-10-06T03:29:33.500929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T12:26:16.082642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.3b","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.593039+0000","last_change":"2023-10-06T03:29:34.538708+0000","last_active":"2023-10-06T03:29:38.593039+0000","last_peered":"2023-10-06T03:29:38.593039+0000","last_clean":"2023-10-06T03:29:38.593039+0000","last_became_active":"2023-10-06T03:29:34.538419+0000","last_became_peered":"2023-10-06T03:29:34.538419+0000","last_unstale":"2023-10-06T03:29:38.593039+0000","last_undegraded":"2023-10-06T03:29:38.593039+0000","last_fullsized":"2023-10-06T03:29:38.593039+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_clean_scrub_stamp":"2023-10-06T03:29:33.500929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T03:45:19.939590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3c","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.609565+0000","last_change":"2023-10-06T03:29:34.546290+0000","last_active":"2023-10-06T03:29:38.609565+0000","last_peered":"2023-10-06T03:29:38.609565+0000","last_clean":"2023-10-06T03:29:38.609565+0000","last_became_active":"2023-10-06T03:29:34.545947+0000","last_became_peered":"2023-10-06T03:29:34.545947+0000","last_unstale":"2023-10-06T03:29:38.609565+0000","last_undegraded":"2023-10-06T03:29:38.609565+0000","last_fullsized":"2023-10-06T03:29:38.609565+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_clean_scrub_stamp":"2023-10-06T03:29:33.500929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T12:25:11.771364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3d","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.592442+0000","last_change":"2023-10-06T03:29:34.541409+0000","last_active":"2023-10-06T03:29:38.592442+0000","last_peered":"2023-10-06T03:29:38.592442+0000","last_clean":"2023-10-06T03:29:38.592442+0000","last_became_active":"2023-10-06T03:29:34.541111+0000","last_became_peered":"2023-10-06T03:29:34.541111+0000","last_unstale":"2023-10-06T03:29:38.592442+0000","last_undegraded":"2023-10-06T03:29:38.592442+0000","last_fullsized":"2023-10-06T03:29:38.592442+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_clean_scrub_stamp":"2023-10-06T03:29:33.500929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T04:15:27.116813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.3e","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.602155+0000","last_change":"2023-10-06T03:29:34.549044+0000","last_active":"2023-10-06T03:29:38.602155+0000","last_peered":"2023-10-06T03:29:38.602155+0000","last_clean":"2023-10-06T03:29:38.602155+0000","last_became_active":"2023-10-06T03:29:34.548691+0000","last_became_peered":"2023-10-06T03:29:34.548691+0000","last_unstale":"2023-10-06T03:29:38.602155+0000","last_undegraded":"2023-10-06T03:29:38.602155+0000","last_fullsized":"2023-10-06T03:29:38.602155+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_clean_scrub_stamp":"2023-10-06T03:29:33.500929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T06:09:15.729486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.3f","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.598391+0000","last_change":"2023-10-06T03:29:34.538647+0000","last_active":"2023-10-06T03:29:38.598391+0000","last_peered":"2023-10-06T03:29:38.598391+0000","last_clean":"2023-10-06T03:29:38.598391+0000","last_became_active":"2023-10-06T03:29:34.538343+0000","last_became_peered":"2023-10-06T03:29:34.538343+0000","last_unstale":"2023-10-06T03:29:38.598391+0000","last_undegraded":"2023-10-06T03:29:38.598391+0000","last_fullsized":"2023-10-06T03:29:38.598391+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_clean_scrub_stamp":"2023-10-06T03:29:33.500929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T12:25:11.771364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":4,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":2306,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":6,"up_from":16,"seq":68719476744,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297572,"kb_used_data":284,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074268,"statfs":{"total":96636764160,"available":96332050432,"internally_reserved":0,"allocated":290816,"data_stored":120026,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77800000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66900000000000004}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.94699999999999995}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0940000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.875}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0600000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.22}]}]},{"osd":5,"up_from":16,"seq":68719476744,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297580,"kb_used_data":292,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074260,"statfs":{"total":96636764160,"available":96332042240,"internally_reserved":0,"allocated":299008,"data_stored":120650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85099999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.087}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97999999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80100000000000005}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0469999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2509999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.208}]}]},{"osd":4,"up_from":16,"seq":68719476743,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297592,"kb_used_data":304,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074248,"statfs":{"total":96636764160,"available":96332029952,"internally_reserved":0,"allocated":311296,"data_stored":121618,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.052}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79400000000000004}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96899999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1399999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2589999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.117}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3859999999999999}]}]},{"osd":7,"up_from":15,"seq":64424509447,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298180,"kb_used_data":892,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94073660,"statfs":{"total":96636764160,"available":96331427840,"internally_reserved":0,"allocated":913408,"data_stored":712146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97399999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86299999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.748}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.94399999999999995}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84299999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80700000000000005}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0880000000000001}]}]},{"osd":0,"up_from":15,"seq":64424509448,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302252,"kb_used_data":868,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94069588,"statfs":{"total":96636764160,"available":96327258112,"internally_reserved":0,"allocated":888832,"data_stored":710948,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97999999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93999999999999995}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76300000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2749999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5409999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.046}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.153}]}]},{"osd":3,"up_from":14,"seq":60129542152,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":301672,"kb_used_data":288,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94070168,"statfs":{"total":96636764160,"available":96327852032,"internally_reserved":0,"allocated":294912,"data_stored":120116,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76500000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81499999999999995}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95999999999999996}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0449999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86599999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2410000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1040000000000001}]}]},{"osd":2,"up_from":14,"seq":60129542151,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":301668,"kb_used_data":284,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94070172,"statfs":{"total":96636764160,"available":96327856128,"internally_reserved":0,"allocated":290816,"data_stored":120026,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78800000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65200000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57199999999999995}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92100000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86099999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0509999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.99299999999999999}]}]},{"osd":1,"up_from":14,"seq":60129542151,"num_pgs":51,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":301668,"kb_used_data":284,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94070172,"statfs":{"total":96636764160,"available":96327856128,"internally_reserved":0,"allocated":290816,"data_stored":120026,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60599999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67200000000000004}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72899999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.875}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78500000000000003}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85199999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.05}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":554,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-06T03:29:48.326 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-06T03:29:48.434 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:48.429+0000 7fce16b19700 1 Processor -- start 2023-10-06T03:29:48.434 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:48.429+0000 7fce16b19700 1 -- start start 2023-10-06T03:29:48.434 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:48.429+0000 7fce16b19700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fce100cb380 0x7fce100c72e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:48.434 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:48.429+0000 7fce16b19700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fce1006c760 0x7fce10069ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:48.434 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:48.429+0000 7fce16b19700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce100cb750 con 0x7fce10156770 2023-10-06T03:29:48.435 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:48.429+0000 7fce16b19700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fce1006d140 con 0x7fce1006c760 2023-10-06T03:29:48.435 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:48.429+0000 7fce16b19700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fce100c7950 con 0x7fce100cb380 2023-10-06T03:29:48.435 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:48.429+0000 7fce150b6700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fce1006c760 0x7fce10069ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:48.435 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:48.429+0000 7fce150b6700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fce1006c760 0x7fce10069ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:60468/0 (socket says 172.21.15.110:60468) 2023-10-06T03:29:48.435 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:48.429+0000 7fce150b6700 1 -- 172.21.15.110:0/1427662363 learned_addr learned my addr 172.21.15.110:0/1427662363 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:29:48.435 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:48.429+0000 7fce148b5700 1 --2- 172.21.15.110:0/1427662363 >> v2:172.21.15.192:3300/0 conn(0x7fce100cb380 0x7fce100c72e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:48.435 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:48.429+0000 7fce150b6700 1 -- 172.21.15.110:0/1427662363 >> v2:172.21.15.192:3300/0 conn(0x7fce100cb380 msgr2=0x7fce100c72e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:29:48.435 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:48.429+0000 7fce150b6700 1 --2- 172.21.15.110:0/1427662363 >> v2:172.21.15.192:3300/0 conn(0x7fce100cb380 0x7fce100c72e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:48.435 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:48.429+0000 7fce150b6700 1 -- 172.21.15.110:0/1427662363 >> v1:172.21.15.192:6789/0 conn(0x7fce10156770 legacy=0x7fce1006c050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:29:48.435 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:48.429+0000 7fce150b6700 1 -- 172.21.15.110:0/1427662363 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce100c7c30 con 0x7fce1006c760 2023-10-06T03:29:48.435 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:48.429+0000 7fce150b6700 1 --2- 172.21.15.110:0/1427662363 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fce1006c760 0x7fce10069ae0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fce100640e0 tx=0x7fce08009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=728c084a44ba42c7 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:29:48.436 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:48.429+0000 7fce0f7fe700 1 -- 172.21.15.110:0/1427662363 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce0800e070 con 0x7fce1006c760 2023-10-06T03:29:48.436 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:48.429+0000 7fce0f7fe700 1 -- 172.21.15.110:0/1427662363 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce0800ba50 con 0x7fce1006c760 2023-10-06T03:29:48.436 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:48.429+0000 7fce0f7fe700 1 -- 172.21.15.110:0/1427662363 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce080092e0 con 0x7fce1006c760 2023-10-06T03:29:48.436 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:48.429+0000 7fce16b19700 1 -- 172.21.15.110:0/1427662363 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fce1006c760 msgr2=0x7fce10069ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:48.436 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:48.429+0000 7fce16b19700 1 --2- 172.21.15.110:0/1427662363 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fce1006c760 0x7fce10069ae0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fce100640e0 tx=0x7fce08009840 comp rx=0 tx=0).stop 2023-10-06T03:29:48.436 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:48.429+0000 7fce16b19700 1 -- 172.21.15.110:0/1427662363 shutdown_connections 2023-10-06T03:29:48.436 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:48.429+0000 7fce16b19700 1 --2- 172.21.15.110:0/1427662363 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fce1006c760 0x7fce10069ae0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:48.436 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:48.429+0000 7fce16b19700 1 --2- 172.21.15.110:0/1427662363 >> v2:172.21.15.192:3300/0 conn(0x7fce100cb380 0x7fce100c72e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:48.436 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:48.429+0000 7fce16b19700 1 -- 172.21.15.110:0/1427662363 >> 172.21.15.110:0/1427662363 conn(0x7fce100cbd00 msgr2=0x7fce100cc100 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:29:48.436 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:48.429+0000 7fce16b19700 1 -- 172.21.15.110:0/1427662363 shutdown_connections 2023-10-06T03:29:48.436 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:48.429+0000 7fce16b19700 1 -- 172.21.15.110:0/1427662363 wait complete. 2023-10-06T03:29:48.438 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:48.433+0000 7fce16b19700 1 Processor -- start 2023-10-06T03:29:48.438 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:48.433+0000 7fce16b19700 1 -- start start 2023-10-06T03:29:48.438 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:48.433+0000 7fce16b19700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fce1006c760 0x7fce10146930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:48.438 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:48.433+0000 7fce16b19700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fce100cb380 0x7fce1006a850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:48.439 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:48.433+0000 7fce16b19700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fce10156770 0x7fce1006ad90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:48.439 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:48.433+0000 7fce16b19700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce1016a0d0 con 0x7fce10156770 2023-10-06T03:29:48.439 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:48.433+0000 7fce16b19700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce10169df0 con 0x7fce100cb380 2023-10-06T03:29:48.439 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:48.433+0000 7fce16b19700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce1016a400 con 0x7fce1006c760 2023-10-06T03:29:48.439 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:48.433+0000 7fce150b6700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fce10156770 0x7fce1006ad90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:48.439 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:48.433+0000 7fce150b6700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fce10156770 0x7fce1006ad90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:60478/0 (socket says 172.21.15.110:60478) 2023-10-06T03:29:48.439 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:48.433+0000 7fce150b6700 1 -- 172.21.15.110:0/2787292858 learned_addr learned my addr 172.21.15.110:0/2787292858 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:29:48.439 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:48.433+0000 7fce0ffff700 1 --2- 172.21.15.110:0/2787292858 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fce100cb380 0x7fce1006a850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:48.439 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:48.433+0000 7fce148b5700 1 --2- 172.21.15.110:0/2787292858 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fce1006c760 0x7fce10146930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:48.439 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:48.433+0000 7fce150b6700 1 -- 172.21.15.110:0/2787292858 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fce1006c760 msgr2=0x7fce10146930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:48.439 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:48.433+0000 7fce150b6700 1 --2- 172.21.15.110:0/2787292858 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fce1006c760 0x7fce10146930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:48.440 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:48.433+0000 7fce150b6700 1 -- 172.21.15.110:0/2787292858 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fce100cb380 msgr2=0x7fce1006a850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:48.440 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:48.433+0000 7fce150b6700 1 --2- 172.21.15.110:0/2787292858 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fce100cb380 0x7fce1006a850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:48.440 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:48.433+0000 7fce150b6700 1 -- 172.21.15.110:0/2787292858 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce08009ae0 con 0x7fce10156770 2023-10-06T03:29:48.440 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:48.433+0000 7fce148b5700 1 --2- 172.21.15.110:0/2787292858 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fce1006c760 0x7fce10146930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-06T03:29:48.440 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:48.433+0000 7fce150b6700 1 --2- 172.21.15.110:0/2787292858 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fce10156770 0x7fce1006ad90 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fce10069300 tx=0x7fce08018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:29:48.440 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:48.433+0000 7fce0cff9700 1 -- 172.21.15.110:0/2787292858 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce080094e0 con 0x7fce10156770 2023-10-06T03:29:48.440 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:48.433+0000 7fce0cff9700 1 -- 172.21.15.110:0/2787292858 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce0800be60 con 0x7fce10156770 2023-10-06T03:29:48.440 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:48.433+0000 7fce16b19700 1 -- 172.21.15.110:0/2787292858 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce1016a6e0 con 0x7fce10156770 2023-10-06T03:29:48.440 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:48.433+0000 7fce0cff9700 1 -- 172.21.15.110:0/2787292858 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce08025460 con 0x7fce10156770 2023-10-06T03:29:48.440 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:48.433+0000 7fce16b19700 1 -- 172.21.15.110:0/2787292858 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce10146e70 con 0x7fce10156770 2023-10-06T03:29:48.441 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:48.437+0000 7fce16b19700 1 -- 172.21.15.110:0/2787292858 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce1016abe0 con 0x7fce10156770 2023-10-06T03:29:48.441 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:48.437+0000 7fce0cff9700 1 -- 172.21.15.110:0/2787292858 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7fce08003040 con 0x7fce10156770 2023-10-06T03:29:48.441 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:48.437+0000 7fce0cff9700 1 --2- 172.21.15.110:0/2787292858 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fcdf0060620 0x7fcdf0062ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:48.441 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:48.437+0000 7fce0cff9700 1 -- 172.21.15.110:0/2787292858 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6809+0+0 (secure 0 0 0) 0x7fce0807e820 con 0x7fce10156770 2023-10-06T03:29:48.441 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:48.437+0000 7fce148b5700 1 --2- 172.21.15.110:0/2787292858 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fcdf0060620 0x7fcdf0062ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:48.442 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:48.437+0000 7fce148b5700 1 --2- 172.21.15.110:0/2787292858 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fcdf0060620 0x7fcdf0062ad0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fce00006fd0 tx=0x7fce00008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:29:48.448 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:48.445+0000 7fce0cff9700 1 -- 172.21.15.110:0/2787292858 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fce0804e9a0 con 0x7fce10156770 2023-10-06T03:29:48.611 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:48.609+0000 7fce16b19700 1 -- 172.21.15.110:0/2787292858 --> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fce10154b30 con 0x7fcdf0060620 2023-10-06T03:29:48.633 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:48.625+0000 7fce0cff9700 1 -- 172.21.15.110:0/2787292858 <== mgr.4110 v2:172.21.15.110:6832/33268 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+367361 (secure 0 0 0) 0x7fce10154b30 con 0x7fcdf0060620 2023-10-06T03:29:48.638 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-10-06T03:29:48.641 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:48.629+0000 7fce16b19700 1 -- 172.21.15.110:0/2787292858 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fcdf0060620 msgr2=0x7fcdf0062ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:48.642 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:48.629+0000 7fce16b19700 1 --2- 172.21.15.110:0/2787292858 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fcdf0060620 0x7fcdf0062ad0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fce00006fd0 tx=0x7fce00008040 comp rx=0 tx=0).stop 2023-10-06T03:29:48.642 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:48.629+0000 7fce16b19700 1 -- 172.21.15.110:0/2787292858 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fce10156770 msgr2=0x7fce1006ad90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:48.642 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:48.629+0000 7fce16b19700 1 --2- 172.21.15.110:0/2787292858 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fce10156770 0x7fce1006ad90 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fce10069300 tx=0x7fce08018000 comp rx=0 tx=0).stop 2023-10-06T03:29:48.642 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:48.629+0000 7fce16b19700 1 -- 172.21.15.110:0/2787292858 shutdown_connections 2023-10-06T03:29:48.642 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:48.629+0000 7fce16b19700 1 --2- 172.21.15.110:0/2787292858 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fcdf0060620 0x7fcdf0062ad0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:48.642 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:48.629+0000 7fce16b19700 1 --2- 172.21.15.110:0/2787292858 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fce10156770 0x7fce1006ad90 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:48.642 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:48.629+0000 7fce16b19700 1 --2- 172.21.15.110:0/2787292858 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fce100cb380 0x7fce1006a850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:48.642 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:48.629+0000 7fce16b19700 1 --2- 172.21.15.110:0/2787292858 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fce1006c760 0x7fce10146930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:48.642 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:48.629+0000 7fce16b19700 1 -- 172.21.15.110:0/2787292858 >> 172.21.15.110:0/2787292858 conn(0x7fce100cbd00 msgr2=0x7fce10155330 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:29:48.643 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:48.633+0000 7fce16b19700 1 -- 172.21.15.110:0/2787292858 shutdown_connections 2023-10-06T03:29:48.643 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:48.633+0000 7fce16b19700 1 -- 172.21.15.110:0/2787292858 wait complete. 2023-10-06T03:29:48.643 INFO:teuthology.orchestra.run.smithi110.stderr:dumped all 2023-10-06T03:29:48.650 INFO:teuthology.orchestra.run.smithi110.stdout:{"pg_ready":true,"pg_map":{"version":34,"stamp":"2023-10-06T03:29:46.651636+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592674,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2429,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":338,"acting":338,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":296,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2398184,"kb_used_data":3496,"kb_used_omap":0,"kb_used_meta":2394624,"kb_avail":752576536,"statfs":{"total":773094113280,"available":770638372864,"internally_reserved":0,"allocated":3579904,"data_stored":2145556,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2452094976},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.066433"},"pg_stats":[{"pgid":"5.19","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.624659+0000","last_change":"2023-10-06T03:29:38.620919+0000","last_active":"2023-10-06T03:29:38.624659+0000","last_peered":"2023-10-06T03:29:38.624659+0000","last_clean":"2023-10-06T03:29:38.624659+0000","last_became_active":"2023-10-06T03:29:38.620293+0000","last_became_peered":"2023-10-06T03:29:38.620293+0000","last_unstale":"2023-10-06T03:29:38.624659+0000","last_undegraded":"2023-10-06T03:29:38.624659+0000","last_fullsized":"2023-10-06T03:29:38.624659+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:37.570693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:37.570693+0000","last_clean_scrub_stamp":"2023-10-06T03:29:37.570693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T13:27:03.828118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7,2],"acting":[1,5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"23'4","reported_seq":18,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.592373+0000","last_change":"2023-10-06T03:29:33.535641+0000","last_active":"2023-10-06T03:29:38.592373+0000","last_peered":"2023-10-06T03:29:38.592373+0000","last_clean":"2023-10-06T03:29:38.592373+0000","last_became_active":"2023-10-06T03:29:33.534410+0000","last_became_peered":"2023-10-06T03:29:33.534410+0000","last_unstale":"2023-10-06T03:29:38.592373+0000","last_undegraded":"2023-10-06T03:29:38.592373+0000","last_fullsized":"2023-10-06T03:29:38.592373+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:32.495616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:32.495616+0000","last_clean_scrub_stamp":"2023-10-06T03:29:32.495616+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T14:51:49.889964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":554,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.594781+0000","last_change":"2023-10-06T03:29:34.546243+0000","last_active":"2023-10-06T03:29:38.594781+0000","last_peered":"2023-10-06T03:29:38.594781+0000","last_clean":"2023-10-06T03:29:38.594781+0000","last_became_active":"2023-10-06T03:29:34.545923+0000","last_became_peered":"2023-10-06T03:29:34.545923+0000","last_unstale":"2023-10-06T03:29:38.594781+0000","last_undegraded":"2023-10-06T03:29:38.594781+0000","last_fullsized":"2023-10-06T03:29:38.594781+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_clean_scrub_stamp":"2023-10-06T03:29:33.500929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T15:20:45.655411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.617434+0000","last_change":"2023-10-06T03:29:38.608716+0000","last_active":"2023-10-06T03:29:38.617434+0000","last_peered":"2023-10-06T03:29:38.617434+0000","last_clean":"2023-10-06T03:29:38.617434+0000","last_became_active":"2023-10-06T03:29:38.607714+0000","last_became_peered":"2023-10-06T03:29:38.607714+0000","last_unstale":"2023-10-06T03:29:38.617434+0000","last_undegraded":"2023-10-06T03:29:38.617434+0000","last_fullsized":"2023-10-06T03:29:38.617434+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:37.570693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:37.570693+0000","last_clean_scrub_stamp":"2023-10-06T03:29:37.570693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T07:11:39.919440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1,2],"acting":[4,6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"23'1","reported_seq":15,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.591195+0000","last_change":"2023-10-06T03:29:33.538145+0000","last_active":"2023-10-06T03:29:38.591195+0000","last_peered":"2023-10-06T03:29:38.591195+0000","last_clean":"2023-10-06T03:29:38.591195+0000","last_became_active":"2023-10-06T03:29:33.536028+0000","last_became_peered":"2023-10-06T03:29:33.536028+0000","last_unstale":"2023-10-06T03:29:38.591195+0000","last_undegraded":"2023-10-06T03:29:38.591195+0000","last_fullsized":"2023-10-06T03:29:38.591195+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:32.495616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:32.495616+0000","last_clean_scrub_stamp":"2023-10-06T03:29:32.495616+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T06:15:00.105181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.589205+0000","last_change":"2023-10-06T03:29:34.546777+0000","last_active":"2023-10-06T03:29:38.589205+0000","last_peered":"2023-10-06T03:29:38.589205+0000","last_clean":"2023-10-06T03:29:38.589205+0000","last_became_active":"2023-10-06T03:29:34.545461+0000","last_became_peered":"2023-10-06T03:29:34.545461+0000","last_unstale":"2023-10-06T03:29:38.589205+0000","last_undegraded":"2023-10-06T03:29:38.589205+0000","last_fullsized":"2023-10-06T03:29:38.589205+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_clean_scrub_stamp":"2023-10-06T03:29:33.500929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T06:40:43.933850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.621795+0000","last_change":"2023-10-06T03:29:38.619453+0000","last_active":"2023-10-06T03:29:38.621795+0000","last_peered":"2023-10-06T03:29:38.621795+0000","last_clean":"2023-10-06T03:29:38.621795+0000","last_became_active":"2023-10-06T03:29:38.618690+0000","last_became_peered":"2023-10-06T03:29:38.618690+0000","last_unstale":"2023-10-06T03:29:38.621795+0000","last_undegraded":"2023-10-06T03:29:38.621795+0000","last_fullsized":"2023-10-06T03:29:38.621795+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:37.570693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:37.570693+0000","last_clean_scrub_stamp":"2023-10-06T03:29:37.570693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T06:00:28.537215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7,4],"acting":[5,0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"23'2","reported_seq":16,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.602095+0000","last_change":"2023-10-06T03:29:33.542309+0000","last_active":"2023-10-06T03:29:38.602095+0000","last_peered":"2023-10-06T03:29:38.602095+0000","last_clean":"2023-10-06T03:29:38.602095+0000","last_became_active":"2023-10-06T03:29:33.542038+0000","last_became_peered":"2023-10-06T03:29:33.542038+0000","last_unstale":"2023-10-06T03:29:38.602095+0000","last_undegraded":"2023-10-06T03:29:38.602095+0000","last_fullsized":"2023-10-06T03:29:38.602095+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:32.495616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:32.495616+0000","last_clean_scrub_stamp":"2023-10-06T03:29:32.495616+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T08:04:27.115289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.592351+0000","last_change":"2023-10-06T03:29:34.540591+0000","last_active":"2023-10-06T03:29:38.592351+0000","last_peered":"2023-10-06T03:29:38.592351+0000","last_clean":"2023-10-06T03:29:38.592351+0000","last_became_active":"2023-10-06T03:29:34.540219+0000","last_became_peered":"2023-10-06T03:29:34.540219+0000","last_unstale":"2023-10-06T03:29:38.592351+0000","last_undegraded":"2023-10-06T03:29:38.592351+0000","last_fullsized":"2023-10-06T03:29:38.592351+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_clean_scrub_stamp":"2023-10-06T03:29:33.500929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T11:28:48.945575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.626642+0000","last_change":"2023-10-06T03:29:38.624230+0000","last_active":"2023-10-06T03:29:38.626642+0000","last_peered":"2023-10-06T03:29:38.626642+0000","last_clean":"2023-10-06T03:29:38.626642+0000","last_became_active":"2023-10-06T03:29:38.623638+0000","last_became_peered":"2023-10-06T03:29:38.623638+0000","last_unstale":"2023-10-06T03:29:38.626642+0000","last_undegraded":"2023-10-06T03:29:38.626642+0000","last_fullsized":"2023-10-06T03:29:38.626642+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:37.570693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:37.570693+0000","last_clean_scrub_stamp":"2023-10-06T03:29:37.570693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T12:11:50.851770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1,6],"acting":[7,4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"23'2","reported_seq":16,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.600846+0000","last_change":"2023-10-06T03:29:33.540179+0000","last_active":"2023-10-06T03:29:38.600846+0000","last_peered":"2023-10-06T03:29:38.600846+0000","last_clean":"2023-10-06T03:29:38.600846+0000","last_became_active":"2023-10-06T03:29:33.539825+0000","last_became_peered":"2023-10-06T03:29:33.539825+0000","last_unstale":"2023-10-06T03:29:38.600846+0000","last_undegraded":"2023-10-06T03:29:38.600846+0000","last_fullsized":"2023-10-06T03:29:38.600846+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:32.495616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:32.495616+0000","last_clean_scrub_stamp":"2023-10-06T03:29:32.495616+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T11:58:53.137774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.593620+0000","last_change":"2023-10-06T03:29:34.540676+0000","last_active":"2023-10-06T03:29:38.593620+0000","last_peered":"2023-10-06T03:29:38.593620+0000","last_clean":"2023-10-06T03:29:38.593620+0000","last_became_active":"2023-10-06T03:29:34.540358+0000","last_became_peered":"2023-10-06T03:29:34.540358+0000","last_unstale":"2023-10-06T03:29:38.593620+0000","last_undegraded":"2023-10-06T03:29:38.593620+0000","last_fullsized":"2023-10-06T03:29:38.593620+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_clean_scrub_stamp":"2023-10-06T03:29:33.500929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T07:51:21.462969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.624627+0000","last_change":"2023-10-06T03:29:38.620564+0000","last_active":"2023-10-06T03:29:38.624627+0000","last_peered":"2023-10-06T03:29:38.624627+0000","last_clean":"2023-10-06T03:29:38.624627+0000","last_became_active":"2023-10-06T03:29:38.620134+0000","last_became_peered":"2023-10-06T03:29:38.620134+0000","last_unstale":"2023-10-06T03:29:38.624627+0000","last_undegraded":"2023-10-06T03:29:38.624627+0000","last_fullsized":"2023-10-06T03:29:38.624627+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:37.570693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:37.570693+0000","last_clean_scrub_stamp":"2023-10-06T03:29:37.570693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T04:19:16.510300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4,0],"acting":[1,3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"23'1","reported_seq":15,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.593175+0000","last_change":"2023-10-06T03:29:33.542267+0000","last_active":"2023-10-06T03:29:38.593175+0000","last_peered":"2023-10-06T03:29:38.593175+0000","last_clean":"2023-10-06T03:29:38.593175+0000","last_became_active":"2023-10-06T03:29:33.541987+0000","last_became_peered":"2023-10-06T03:29:33.541987+0000","last_unstale":"2023-10-06T03:29:38.593175+0000","last_undegraded":"2023-10-06T03:29:38.593175+0000","last_fullsized":"2023-10-06T03:29:38.593175+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:32.495616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:32.495616+0000","last_clean_scrub_stamp":"2023-10-06T03:29:32.495616+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T12:25:10.766051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.600288+0000","last_change":"2023-10-06T03:29:34.547283+0000","last_active":"2023-10-06T03:29:38.600288+0000","last_peered":"2023-10-06T03:29:38.600288+0000","last_clean":"2023-10-06T03:29:38.600288+0000","last_became_active":"2023-10-06T03:29:34.546966+0000","last_became_peered":"2023-10-06T03:29:34.546966+0000","last_unstale":"2023-10-06T03:29:38.600288+0000","last_undegraded":"2023-10-06T03:29:38.600288+0000","last_fullsized":"2023-10-06T03:29:38.600288+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_clean_scrub_stamp":"2023-10-06T03:29:33.500929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T07:55:08.792414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.621924+0000","last_change":"2023-10-06T03:29:38.614364+0000","last_active":"2023-10-06T03:29:38.621924+0000","last_peered":"2023-10-06T03:29:38.621924+0000","last_clean":"2023-10-06T03:29:38.621924+0000","last_became_active":"2023-10-06T03:29:38.613892+0000","last_became_peered":"2023-10-06T03:29:38.613892+0000","last_unstale":"2023-10-06T03:29:38.621924+0000","last_undegraded":"2023-10-06T03:29:38.621924+0000","last_fullsized":"2023-10-06T03:29:38.621924+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:37.570693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:37.570693+0000","last_clean_scrub_stamp":"2023-10-06T03:29:37.570693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T13:00:21.693396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2,0],"acting":[4,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"23'1","reported_seq":15,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.592761+0000","last_change":"2023-10-06T03:29:33.546282+0000","last_active":"2023-10-06T03:29:38.592761+0000","last_peered":"2023-10-06T03:29:38.592761+0000","last_clean":"2023-10-06T03:29:38.592761+0000","last_became_active":"2023-10-06T03:29:33.546039+0000","last_became_peered":"2023-10-06T03:29:33.546039+0000","last_unstale":"2023-10-06T03:29:38.592761+0000","last_undegraded":"2023-10-06T03:29:38.592761+0000","last_fullsized":"2023-10-06T03:29:38.592761+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:32.495616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:32.495616+0000","last_clean_scrub_stamp":"2023-10-06T03:29:32.495616+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T12:01:27.570873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.589771+0000","last_change":"2023-10-06T03:29:34.550953+0000","last_active":"2023-10-06T03:29:38.589771+0000","last_peered":"2023-10-06T03:29:38.589771+0000","last_clean":"2023-10-06T03:29:38.589771+0000","last_became_active":"2023-10-06T03:29:34.550614+0000","last_became_peered":"2023-10-06T03:29:34.550614+0000","last_unstale":"2023-10-06T03:29:38.589771+0000","last_undegraded":"2023-10-06T03:29:38.589771+0000","last_fullsized":"2023-10-06T03:29:38.589771+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_clean_scrub_stamp":"2023-10-06T03:29:33.500929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T11:59:02.611216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.620922+0000","last_change":"2023-10-06T03:29:38.613494+0000","last_active":"2023-10-06T03:29:38.620922+0000","last_peered":"2023-10-06T03:29:38.620922+0000","last_clean":"2023-10-06T03:29:38.620922+0000","last_became_active":"2023-10-06T03:29:38.613121+0000","last_became_peered":"2023-10-06T03:29:38.613121+0000","last_unstale":"2023-10-06T03:29:38.620922+0000","last_undegraded":"2023-10-06T03:29:38.620922+0000","last_fullsized":"2023-10-06T03:29:38.620922+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:37.570693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:37.570693+0000","last_clean_scrub_stamp":"2023-10-06T03:29:37.570693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T03:45:24.009354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3,7],"acting":[6,4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.608215+0000","last_change":"2023-10-06T03:29:33.542907+0000","last_active":"2023-10-06T03:29:38.608215+0000","last_peered":"2023-10-06T03:29:38.608215+0000","last_clean":"2023-10-06T03:29:38.608215+0000","last_became_active":"2023-10-06T03:29:33.542556+0000","last_became_peered":"2023-10-06T03:29:33.542556+0000","last_unstale":"2023-10-06T03:29:38.608215+0000","last_undegraded":"2023-10-06T03:29:38.608215+0000","last_fullsized":"2023-10-06T03:29:38.608215+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:32.495616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:32.495616+0000","last_clean_scrub_stamp":"2023-10-06T03:29:32.495616+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T03:54:50.215488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.599642+0000","last_change":"2023-10-06T03:29:34.541680+0000","last_active":"2023-10-06T03:29:38.599642+0000","last_peered":"2023-10-06T03:29:38.599642+0000","last_clean":"2023-10-06T03:29:38.599642+0000","last_became_active":"2023-10-06T03:29:34.541220+0000","last_became_peered":"2023-10-06T03:29:34.541220+0000","last_unstale":"2023-10-06T03:29:38.599642+0000","last_undegraded":"2023-10-06T03:29:38.599642+0000","last_fullsized":"2023-10-06T03:29:38.599642+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_clean_scrub_stamp":"2023-10-06T03:29:33.500929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T06:09:15.729486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.627293+0000","last_change":"2023-10-06T03:29:38.623302+0000","last_active":"2023-10-06T03:29:38.627293+0000","last_peered":"2023-10-06T03:29:38.627293+0000","last_clean":"2023-10-06T03:29:38.627293+0000","last_became_active":"2023-10-06T03:29:38.622782+0000","last_became_peered":"2023-10-06T03:29:38.622782+0000","last_unstale":"2023-10-06T03:29:38.627293+0000","last_undegraded":"2023-10-06T03:29:38.627293+0000","last_fullsized":"2023-10-06T03:29:38.627293+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:37.570693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:37.570693+0000","last_clean_scrub_stamp":"2023-10-06T03:29:37.570693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T06:00:28.537215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1,2],"acting":[0,7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":14,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.589343+0000","last_change":"2023-10-06T03:29:33.543906+0000","last_active":"2023-10-06T03:29:38.589343+0000","last_peered":"2023-10-06T03:29:38.589343+0000","last_clean":"2023-10-06T03:29:38.589343+0000","last_became_active":"2023-10-06T03:29:33.543551+0000","last_became_peered":"2023-10-06T03:29:33.543551+0000","last_unstale":"2023-10-06T03:29:38.589343+0000","last_undegraded":"2023-10-06T03:29:38.589343+0000","last_fullsized":"2023-10-06T03:29:38.589343+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:32.495616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:32.495616+0000","last_clean_scrub_stamp":"2023-10-06T03:29:32.495616+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T03:34:32.799922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.609821+0000","last_change":"2023-10-06T03:29:34.549763+0000","last_active":"2023-10-06T03:29:38.609821+0000","last_peered":"2023-10-06T03:29:38.609821+0000","last_clean":"2023-10-06T03:29:38.609821+0000","last_became_active":"2023-10-06T03:29:34.549402+0000","last_became_peered":"2023-10-06T03:29:34.549402+0000","last_unstale":"2023-10-06T03:29:38.609821+0000","last_undegraded":"2023-10-06T03:29:38.609821+0000","last_fullsized":"2023-10-06T03:29:38.609821+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_clean_scrub_stamp":"2023-10-06T03:29:33.500929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T07:14:03.653521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.588889+0000","last_change":"2023-10-06T03:29:34.548269+0000","last_active":"2023-10-06T03:29:38.588889+0000","last_peered":"2023-10-06T03:29:38.588889+0000","last_clean":"2023-10-06T03:29:38.588889+0000","last_became_active":"2023-10-06T03:29:34.547982+0000","last_became_peered":"2023-10-06T03:29:34.547982+0000","last_unstale":"2023-10-06T03:29:38.588889+0000","last_undegraded":"2023-10-06T03:29:38.588889+0000","last_fullsized":"2023-10-06T03:29:38.588889+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_clean_scrub_stamp":"2023-10-06T03:29:33.500929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T03:54:50.291053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":14,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.596539+0000","last_change":"2023-10-06T03:29:33.542538+0000","last_active":"2023-10-06T03:29:38.596539+0000","last_peered":"2023-10-06T03:29:38.596539+0000","last_clean":"2023-10-06T03:29:38.596539+0000","last_became_active":"2023-10-06T03:29:33.536222+0000","last_became_peered":"2023-10-06T03:29:33.536222+0000","last_unstale":"2023-10-06T03:29:38.596539+0000","last_undegraded":"2023-10-06T03:29:38.596539+0000","last_fullsized":"2023-10-06T03:29:38.596539+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:32.495616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:32.495616+0000","last_clean_scrub_stamp":"2023-10-06T03:29:32.495616+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T07:14:02.648208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.620952+0000","last_change":"2023-10-06T03:29:38.613462+0000","last_active":"2023-10-06T03:29:38.620952+0000","last_peered":"2023-10-06T03:29:38.620952+0000","last_clean":"2023-10-06T03:29:38.620952+0000","last_became_active":"2023-10-06T03:29:38.613096+0000","last_became_peered":"2023-10-06T03:29:38.613096+0000","last_unstale":"2023-10-06T03:29:38.620952+0000","last_undegraded":"2023-10-06T03:29:38.620952+0000","last_fullsized":"2023-10-06T03:29:38.620952+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:37.570693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:37.570693+0000","last_clean_scrub_stamp":"2023-10-06T03:29:37.570693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T11:28:53.015339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7,1],"acting":[6,4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.589101+0000","last_change":"2023-10-06T03:29:34.546990+0000","last_active":"2023-10-06T03:29:38.589101+0000","last_peered":"2023-10-06T03:29:38.589101+0000","last_clean":"2023-10-06T03:29:38.589101+0000","last_became_active":"2023-10-06T03:29:34.544561+0000","last_became_peered":"2023-10-06T03:29:34.544561+0000","last_unstale":"2023-10-06T03:29:38.589101+0000","last_undegraded":"2023-10-06T03:29:38.589101+0000","last_fullsized":"2023-10-06T03:29:38.589101+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_clean_scrub_stamp":"2023-10-06T03:29:33.500929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T09:13:47.732421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":14,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.602311+0000","last_change":"2023-10-06T03:29:33.536276+0000","last_active":"2023-10-06T03:29:38.602311+0000","last_peered":"2023-10-06T03:29:38.602311+0000","last_clean":"2023-10-06T03:29:38.602311+0000","last_became_active":"2023-10-06T03:29:33.535859+0000","last_became_peered":"2023-10-06T03:29:33.535859+0000","last_unstale":"2023-10-06T03:29:38.602311+0000","last_undegraded":"2023-10-06T03:29:38.602311+0000","last_fullsized":"2023-10-06T03:29:38.602311+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:32.495616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:32.495616+0000","last_clean_scrub_stamp":"2023-10-06T03:29:32.495616+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T07:55:07.787102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.623243+0000","last_change":"2023-10-06T03:29:38.618743+0000","last_active":"2023-10-06T03:29:38.623243+0000","last_peered":"2023-10-06T03:29:38.623243+0000","last_clean":"2023-10-06T03:29:38.623243+0000","last_became_active":"2023-10-06T03:29:38.618260+0000","last_became_peered":"2023-10-06T03:29:38.618260+0000","last_unstale":"2023-10-06T03:29:38.623243+0000","last_undegraded":"2023-10-06T03:29:38.623243+0000","last_fullsized":"2023-10-06T03:29:38.623243+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:37.570693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:37.570693+0000","last_clean_scrub_stamp":"2023-10-06T03:29:37.570693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T04:14:12.838199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6,0],"acting":[7,4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.15","version":"23'3","reported_seq":16,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.606179+0000","last_change":"2023-10-06T03:29:33.541547+0000","last_active":"2023-10-06T03:29:38.606179+0000","last_peered":"2023-10-06T03:29:38.606179+0000","last_clean":"2023-10-06T03:29:38.606179+0000","last_became_active":"2023-10-06T03:29:33.541267+0000","last_became_peered":"2023-10-06T03:29:33.541267+0000","last_unstale":"2023-10-06T03:29:38.606179+0000","last_undegraded":"2023-10-06T03:29:38.606179+0000","last_fullsized":"2023-10-06T03:29:38.606179+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:32.495616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:32.495616+0000","last_clean_scrub_stamp":"2023-10-06T03:29:32.495616+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T12:25:10.766051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.608031+0000","last_change":"2023-10-06T03:29:34.550174+0000","last_active":"2023-10-06T03:29:38.608031+0000","last_peered":"2023-10-06T03:29:38.608031+0000","last_clean":"2023-10-06T03:29:38.608031+0000","last_became_active":"2023-10-06T03:29:34.549630+0000","last_became_peered":"2023-10-06T03:29:34.549630+0000","last_unstale":"2023-10-06T03:29:38.608031+0000","last_undegraded":"2023-10-06T03:29:38.608031+0000","last_fullsized":"2023-10-06T03:29:38.608031+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_clean_scrub_stamp":"2023-10-06T03:29:33.500929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T13:02:33.356490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.620450+0000","last_change":"2023-10-06T03:29:38.614643+0000","last_active":"2023-10-06T03:29:38.620450+0000","last_peered":"2023-10-06T03:29:38.620450+0000","last_clean":"2023-10-06T03:29:38.620450+0000","last_became_active":"2023-10-06T03:29:38.614242+0000","last_became_peered":"2023-10-06T03:29:38.614242+0000","last_unstale":"2023-10-06T03:29:38.620450+0000","last_undegraded":"2023-10-06T03:29:38.620450+0000","last_fullsized":"2023-10-06T03:29:38.620450+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:37.570693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:37.570693+0000","last_clean_scrub_stamp":"2023-10-06T03:29:37.570693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T06:29:58.268896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,5],"acting":[3,0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.14","version":"23'6","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.594060+0000","last_change":"2023-10-06T03:29:33.538661+0000","last_active":"2023-10-06T03:29:38.594060+0000","last_peered":"2023-10-06T03:29:38.594060+0000","last_clean":"2023-10-06T03:29:38.594060+0000","last_became_active":"2023-10-06T03:29:33.538296+0000","last_became_peered":"2023-10-06T03:29:33.538296+0000","last_unstale":"2023-10-06T03:29:38.594060+0000","last_undegraded":"2023-10-06T03:29:38.594060+0000","last_fullsized":"2023-10-06T03:29:38.594060+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:32.495616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:32.495616+0000","last_clean_scrub_stamp":"2023-10-06T03:29:32.495616+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T10:47:52.547584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1536,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.593839+0000","last_change":"2023-10-06T03:29:34.549928+0000","last_active":"2023-10-06T03:29:38.593839+0000","last_peered":"2023-10-06T03:29:38.593839+0000","last_clean":"2023-10-06T03:29:38.593839+0000","last_became_active":"2023-10-06T03:29:34.549421+0000","last_became_peered":"2023-10-06T03:29:34.549421+0000","last_unstale":"2023-10-06T03:29:38.593839+0000","last_undegraded":"2023-10-06T03:29:38.593839+0000","last_fullsized":"2023-10-06T03:29:38.593839+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_clean_scrub_stamp":"2023-10-06T03:29:33.500929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T03:45:19.939590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.624617+0000","last_change":"2023-10-06T03:29:38.620526+0000","last_active":"2023-10-06T03:29:38.624617+0000","last_peered":"2023-10-06T03:29:38.624617+0000","last_clean":"2023-10-06T03:29:38.624617+0000","last_became_active":"2023-10-06T03:29:38.620095+0000","last_became_peered":"2023-10-06T03:29:38.620095+0000","last_unstale":"2023-10-06T03:29:38.624617+0000","last_undegraded":"2023-10-06T03:29:38.624617+0000","last_fullsized":"2023-10-06T03:29:38.624617+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:37.570693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:37.570693+0000","last_clean_scrub_stamp":"2023-10-06T03:29:37.570693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T04:00:40.252544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3,7],"acting":[1,5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.594637+0000","last_change":"2023-10-06T03:29:34.538055+0000","last_active":"2023-10-06T03:29:38.594637+0000","last_peered":"2023-10-06T03:29:38.594637+0000","last_clean":"2023-10-06T03:29:38.594637+0000","last_became_active":"2023-10-06T03:29:34.537755+0000","last_became_peered":"2023-10-06T03:29:34.537755+0000","last_unstale":"2023-10-06T03:29:38.594637+0000","last_undegraded":"2023-10-06T03:29:38.594637+0000","last_fullsized":"2023-10-06T03:29:38.594637+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_clean_scrub_stamp":"2023-10-06T03:29:33.500929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T08:21:53.296989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":16,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.610021+0000","last_change":"2023-10-06T03:29:29.387402+0000","last_active":"2023-10-06T03:29:38.610021+0000","last_peered":"2023-10-06T03:29:38.610021+0000","last_clean":"2023-10-06T03:29:38.610021+0000","last_became_active":"2023-10-06T03:29:29.386812+0000","last_became_peered":"2023-10-06T03:29:29.386812+0000","last_unstale":"2023-10-06T03:29:38.610021+0000","last_undegraded":"2023-10-06T03:29:38.610021+0000","last_fullsized":"2023-10-06T03:29:38.610021+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:28.357506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:28.357506+0000","last_clean_scrub_stamp":"2023-10-06T03:29:28.357506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T12:09:43.895810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"27'1","reported_seq":15,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:39.494514+0000","last_change":"2023-10-06T03:29:33.542307+0000","last_active":"2023-10-06T03:29:39.494514+0000","last_peered":"2023-10-06T03:29:39.494514+0000","last_clean":"2023-10-06T03:29:39.494514+0000","last_became_active":"2023-10-06T03:29:33.541670+0000","last_became_peered":"2023-10-06T03:29:33.541670+0000","last_unstale":"2023-10-06T03:29:39.494514+0000","last_undegraded":"2023-10-06T03:29:39.494514+0000","last_fullsized":"2023-10-06T03:29:39.494514+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:32.495616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:32.495616+0000","last_clean_scrub_stamp":"2023-10-06T03:29:32.495616+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T05:03:47.111820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.620435+0000","last_change":"2023-10-06T03:29:38.613923+0000","last_active":"2023-10-06T03:29:38.620435+0000","last_peered":"2023-10-06T03:29:38.620435+0000","last_clean":"2023-10-06T03:29:38.620435+0000","last_became_active":"2023-10-06T03:29:38.612452+0000","last_became_peered":"2023-10-06T03:29:38.612452+0000","last_unstale":"2023-10-06T03:29:38.620435+0000","last_undegraded":"2023-10-06T03:29:38.620435+0000","last_fullsized":"2023-10-06T03:29:38.620435+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:37.570693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:37.570693+0000","last_clean_scrub_stamp":"2023-10-06T03:29:37.570693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T05:52:46.994576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4,0],"acting":[3,1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.589121+0000","last_change":"2023-10-06T03:29:34.539246+0000","last_active":"2023-10-06T03:29:38.589121+0000","last_peered":"2023-10-06T03:29:38.589121+0000","last_clean":"2023-10-06T03:29:38.589121+0000","last_became_active":"2023-10-06T03:29:34.538925+0000","last_became_peered":"2023-10-06T03:29:34.538925+0000","last_unstale":"2023-10-06T03:29:38.589121+0000","last_undegraded":"2023-10-06T03:29:38.589121+0000","last_fullsized":"2023-10-06T03:29:38.589121+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_clean_scrub_stamp":"2023-10-06T03:29:33.500929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T03:59:12.193877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":16,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.609252+0000","last_change":"2023-10-06T03:29:29.386238+0000","last_active":"2023-10-06T03:29:38.609252+0000","last_peered":"2023-10-06T03:29:38.609252+0000","last_clean":"2023-10-06T03:29:38.609252+0000","last_became_active":"2023-10-06T03:29:29.385865+0000","last_became_peered":"2023-10-06T03:29:29.385865+0000","last_unstale":"2023-10-06T03:29:38.609252+0000","last_undegraded":"2023-10-06T03:29:38.609252+0000","last_fullsized":"2023-10-06T03:29:38.609252+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:28.357506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:28.357506+0000","last_clean_scrub_stamp":"2023-10-06T03:29:28.357506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T12:43:34.714730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"23'2","reported_seq":16,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.593260+0000","last_change":"2023-10-06T03:29:33.539135+0000","last_active":"2023-10-06T03:29:38.593260+0000","last_peered":"2023-10-06T03:29:38.593260+0000","last_clean":"2023-10-06T03:29:38.593260+0000","last_became_active":"2023-10-06T03:29:33.538818+0000","last_became_peered":"2023-10-06T03:29:33.538818+0000","last_unstale":"2023-10-06T03:29:38.593260+0000","last_undegraded":"2023-10-06T03:29:38.593260+0000","last_fullsized":"2023-10-06T03:29:38.593260+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:32.495616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:32.495616+0000","last_clean_scrub_stamp":"2023-10-06T03:29:32.495616+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T03:54:50.215488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.622166+0000","last_change":"2023-10-06T03:29:38.614527+0000","last_active":"2023-10-06T03:29:38.622166+0000","last_peered":"2023-10-06T03:29:38.622166+0000","last_clean":"2023-10-06T03:29:38.622166+0000","last_became_active":"2023-10-06T03:29:38.614090+0000","last_became_peered":"2023-10-06T03:29:38.614090+0000","last_unstale":"2023-10-06T03:29:38.622166+0000","last_undegraded":"2023-10-06T03:29:38.622166+0000","last_fullsized":"2023-10-06T03:29:38.622166+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:37.570693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:37.570693+0000","last_clean_scrub_stamp":"2023-10-06T03:29:37.570693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T05:47:25.729169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7,0],"acting":[4,3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.592223+0000","last_change":"2023-10-06T03:29:34.537232+0000","last_active":"2023-10-06T03:29:38.592223+0000","last_peered":"2023-10-06T03:29:38.592223+0000","last_clean":"2023-10-06T03:29:38.592223+0000","last_became_active":"2023-10-06T03:29:34.536984+0000","last_became_peered":"2023-10-06T03:29:34.536984+0000","last_unstale":"2023-10-06T03:29:38.592223+0000","last_undegraded":"2023-10-06T03:29:38.592223+0000","last_fullsized":"2023-10-06T03:29:38.592223+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_clean_scrub_stamp":"2023-10-06T03:29:33.500929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T03:42:52.876376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":16,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.605996+0000","last_change":"2023-10-06T03:29:29.384279+0000","last_active":"2023-10-06T03:29:38.605996+0000","last_peered":"2023-10-06T03:29:38.605996+0000","last_clean":"2023-10-06T03:29:38.605996+0000","last_became_active":"2023-10-06T03:29:29.383533+0000","last_became_peered":"2023-10-06T03:29:29.383533+0000","last_unstale":"2023-10-06T03:29:38.605996+0000","last_undegraded":"2023-10-06T03:29:38.605996+0000","last_fullsized":"2023-10-06T03:29:38.605996+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:28.357506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:28.357506+0000","last_clean_scrub_stamp":"2023-10-06T03:29:28.357506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T08:34:19.641525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.609261+0000","last_change":"2023-10-06T03:29:33.542909+0000","last_active":"2023-10-06T03:29:38.609261+0000","last_peered":"2023-10-06T03:29:38.609261+0000","last_clean":"2023-10-06T03:29:38.609261+0000","last_became_active":"2023-10-06T03:29:33.542573+0000","last_became_peered":"2023-10-06T03:29:33.542573+0000","last_unstale":"2023-10-06T03:29:38.609261+0000","last_undegraded":"2023-10-06T03:29:38.609261+0000","last_fullsized":"2023-10-06T03:29:38.609261+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:32.495616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:32.495616+0000","last_clean_scrub_stamp":"2023-10-06T03:29:32.495616+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T14:26:49.262597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.620982+0000","last_change":"2023-10-06T03:29:38.615475+0000","last_active":"2023-10-06T03:29:38.620982+0000","last_peered":"2023-10-06T03:29:38.620982+0000","last_clean":"2023-10-06T03:29:38.620982+0000","last_became_active":"2023-10-06T03:29:38.615049+0000","last_became_peered":"2023-10-06T03:29:38.615049+0000","last_unstale":"2023-10-06T03:29:38.620982+0000","last_undegraded":"2023-10-06T03:29:38.620982+0000","last_fullsized":"2023-10-06T03:29:38.620982+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:37.570693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:37.570693+0000","last_clean_scrub_stamp":"2023-10-06T03:29:37.570693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T14:34:30.474789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5,2],"acting":[6,0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.608125+0000","last_change":"2023-10-06T03:29:34.543257+0000","last_active":"2023-10-06T03:29:38.608125+0000","last_peered":"2023-10-06T03:29:38.608125+0000","last_clean":"2023-10-06T03:29:38.608125+0000","last_became_active":"2023-10-06T03:29:34.537574+0000","last_became_peered":"2023-10-06T03:29:34.537574+0000","last_unstale":"2023-10-06T03:29:38.608125+0000","last_undegraded":"2023-10-06T03:29:38.608125+0000","last_fullsized":"2023-10-06T03:29:38.608125+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_clean_scrub_stamp":"2023-10-06T03:29:33.500929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T13:34:24.699976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":16,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.609348+0000","last_change":"2023-10-06T03:29:29.382183+0000","last_active":"2023-10-06T03:29:38.609348+0000","last_peered":"2023-10-06T03:29:38.609348+0000","last_clean":"2023-10-06T03:29:38.609348+0000","last_became_active":"2023-10-06T03:29:29.381918+0000","last_became_peered":"2023-10-06T03:29:29.381918+0000","last_unstale":"2023-10-06T03:29:38.609348+0000","last_undegraded":"2023-10-06T03:29:38.609348+0000","last_fullsized":"2023-10-06T03:29:38.609348+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:28.357506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:28.357506+0000","last_clean_scrub_stamp":"2023-10-06T03:29:28.357506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T11:56:26.013716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":14,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.602189+0000","last_change":"2023-10-06T03:29:33.540375+0000","last_active":"2023-10-06T03:29:38.602189+0000","last_peered":"2023-10-06T03:29:38.602189+0000","last_clean":"2023-10-06T03:29:38.602189+0000","last_became_active":"2023-10-06T03:29:33.540016+0000","last_became_peered":"2023-10-06T03:29:33.540016+0000","last_unstale":"2023-10-06T03:29:38.602189+0000","last_undegraded":"2023-10-06T03:29:38.602189+0000","last_fullsized":"2023-10-06T03:29:38.602189+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:32.495616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:32.495616+0000","last_clean_scrub_stamp":"2023-10-06T03:29:32.495616+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T10:47:52.547584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.623995+0000","last_change":"2023-10-06T03:29:38.613181+0000","last_active":"2023-10-06T03:29:38.623995+0000","last_peered":"2023-10-06T03:29:38.623995+0000","last_clean":"2023-10-06T03:29:38.623995+0000","last_became_active":"2023-10-06T03:29:38.612787+0000","last_became_peered":"2023-10-06T03:29:38.612787+0000","last_unstale":"2023-10-06T03:29:38.623995+0000","last_undegraded":"2023-10-06T03:29:38.623995+0000","last_fullsized":"2023-10-06T03:29:38.623995+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:37.570693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:37.570693+0000","last_clean_scrub_stamp":"2023-10-06T03:29:37.570693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T07:38:31.118869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6,5],"acting":[0,1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.609205+0000","last_change":"2023-10-06T03:29:34.543207+0000","last_active":"2023-10-06T03:29:38.609205+0000","last_peered":"2023-10-06T03:29:38.609205+0000","last_clean":"2023-10-06T03:29:38.609205+0000","last_became_active":"2023-10-06T03:29:34.537291+0000","last_became_peered":"2023-10-06T03:29:34.537291+0000","last_unstale":"2023-10-06T03:29:38.609205+0000","last_undegraded":"2023-10-06T03:29:38.609205+0000","last_fullsized":"2023-10-06T03:29:38.609205+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_clean_scrub_stamp":"2023-10-06T03:29:33.500929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T06:40:43.933850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":14,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.594240+0000","last_change":"2023-10-06T03:29:33.543412+0000","last_active":"2023-10-06T03:29:38.594240+0000","last_peered":"2023-10-06T03:29:38.594240+0000","last_clean":"2023-10-06T03:29:38.594240+0000","last_became_active":"2023-10-06T03:29:33.543114+0000","last_became_peered":"2023-10-06T03:29:33.543114+0000","last_unstale":"2023-10-06T03:29:38.594240+0000","last_undegraded":"2023-10-06T03:29:38.594240+0000","last_fullsized":"2023-10-06T03:29:38.594240+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:32.495616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:32.495616+0000","last_clean_scrub_stamp":"2023-10-06T03:29:32.495616+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T15:12:57.750941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":17,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.600966+0000","last_change":"2023-10-06T03:29:29.384415+0000","last_active":"2023-10-06T03:29:38.600966+0000","last_peered":"2023-10-06T03:29:38.600966+0000","last_clean":"2023-10-06T03:29:38.600966+0000","last_became_active":"2023-10-06T03:29:29.383894+0000","last_became_peered":"2023-10-06T03:29:29.383894+0000","last_unstale":"2023-10-06T03:29:38.600966+0000","last_undegraded":"2023-10-06T03:29:38.600966+0000","last_fullsized":"2023-10-06T03:29:38.600966+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:28.357506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:28.357506+0000","last_clean_scrub_stamp":"2023-10-06T03:29:28.357506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T13:09:00.657107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.622959+0000","last_change":"2023-10-06T03:29:38.613076+0000","last_active":"2023-10-06T03:29:38.622959+0000","last_peered":"2023-10-06T03:29:38.622959+0000","last_clean":"2023-10-06T03:29:38.622959+0000","last_became_active":"2023-10-06T03:29:38.612646+0000","last_became_peered":"2023-10-06T03:29:38.612646+0000","last_unstale":"2023-10-06T03:29:38.622959+0000","last_undegraded":"2023-10-06T03:29:38.622959+0000","last_fullsized":"2023-10-06T03:29:38.622959+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:37.570693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:37.570693+0000","last_clean_scrub_stamp":"2023-10-06T03:29:37.570693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T06:07:31.871068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4,6],"acting":[0,1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.609278+0000","last_change":"2023-10-06T03:29:34.543346+0000","last_active":"2023-10-06T03:29:38.609278+0000","last_peered":"2023-10-06T03:29:38.609278+0000","last_clean":"2023-10-06T03:29:38.609278+0000","last_became_active":"2023-10-06T03:29:34.537438+0000","last_became_peered":"2023-10-06T03:29:34.537438+0000","last_unstale":"2023-10-06T03:29:38.609278+0000","last_undegraded":"2023-10-06T03:29:38.609278+0000","last_fullsized":"2023-10-06T03:29:38.609278+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_clean_scrub_stamp":"2023-10-06T03:29:33.500929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T09:13:47.732421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.609403+0000","last_change":"2023-10-06T03:29:33.544242+0000","last_active":"2023-10-06T03:29:38.609403+0000","last_peered":"2023-10-06T03:29:38.609403+0000","last_clean":"2023-10-06T03:29:38.609403+0000","last_became_active":"2023-10-06T03:29:33.543938+0000","last_became_peered":"2023-10-06T03:29:33.543938+0000","last_unstale":"2023-10-06T03:29:38.609403+0000","last_undegraded":"2023-10-06T03:29:38.609403+0000","last_fullsized":"2023-10-06T03:29:38.609403+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:32.495616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:32.495616+0000","last_clean_scrub_stamp":"2023-10-06T03:29:32.495616+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T10:09:45.061520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":16,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.599964+0000","last_change":"2023-10-06T03:29:29.606195+0000","last_active":"2023-10-06T03:29:38.599964+0000","last_peered":"2023-10-06T03:29:38.599964+0000","last_clean":"2023-10-06T03:29:38.599964+0000","last_became_active":"2023-10-06T03:29:29.605489+0000","last_became_peered":"2023-10-06T03:29:29.605489+0000","last_unstale":"2023-10-06T03:29:38.599964+0000","last_undegraded":"2023-10-06T03:29:38.599964+0000","last_fullsized":"2023-10-06T03:29:38.599964+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:28.357506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:28.357506+0000","last_clean_scrub_stamp":"2023-10-06T03:29:28.357506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T10:21:35.864895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.626333+0000","last_change":"2023-10-06T03:29:38.624434+0000","last_active":"2023-10-06T03:29:38.626333+0000","last_peered":"2023-10-06T03:29:38.626333+0000","last_clean":"2023-10-06T03:29:38.626333+0000","last_became_active":"2023-10-06T03:29:38.623676+0000","last_became_peered":"2023-10-06T03:29:38.623676+0000","last_unstale":"2023-10-06T03:29:38.626333+0000","last_undegraded":"2023-10-06T03:29:38.626333+0000","last_fullsized":"2023-10-06T03:29:38.626333+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:37.570693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:37.570693+0000","last_clean_scrub_stamp":"2023-10-06T03:29:37.570693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T12:26:20.152406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,1,7],"acting":[2,5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.599731+0000","last_change":"2023-10-06T03:29:34.550793+0000","last_active":"2023-10-06T03:29:38.599731+0000","last_peered":"2023-10-06T03:29:38.599731+0000","last_clean":"2023-10-06T03:29:38.599731+0000","last_became_active":"2023-10-06T03:29:34.550345+0000","last_became_peered":"2023-10-06T03:29:34.550345+0000","last_unstale":"2023-10-06T03:29:38.599731+0000","last_undegraded":"2023-10-06T03:29:38.599731+0000","last_fullsized":"2023-10-06T03:29:38.599731+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_clean_scrub_stamp":"2023-10-06T03:29:33.500929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T11:28:48.945575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":14,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.591494+0000","last_change":"2023-10-06T03:29:33.543304+0000","last_active":"2023-10-06T03:29:38.591494+0000","last_peered":"2023-10-06T03:29:38.591494+0000","last_clean":"2023-10-06T03:29:38.591494+0000","last_became_active":"2023-10-06T03:29:33.542483+0000","last_became_peered":"2023-10-06T03:29:33.542483+0000","last_unstale":"2023-10-06T03:29:38.591494+0000","last_undegraded":"2023-10-06T03:29:38.591494+0000","last_fullsized":"2023-10-06T03:29:38.591494+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:32.495616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:32.495616+0000","last_clean_scrub_stamp":"2023-10-06T03:29:32.495616+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T11:58:53.137774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":16,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.603498+0000","last_change":"2023-10-06T03:29:29.384353+0000","last_active":"2023-10-06T03:29:38.603498+0000","last_peered":"2023-10-06T03:29:38.603498+0000","last_clean":"2023-10-06T03:29:38.603498+0000","last_became_active":"2023-10-06T03:29:29.383884+0000","last_became_peered":"2023-10-06T03:29:29.383884+0000","last_unstale":"2023-10-06T03:29:38.603498+0000","last_undegraded":"2023-10-06T03:29:38.603498+0000","last_fullsized":"2023-10-06T03:29:38.603498+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:28.357506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:28.357506+0000","last_clean_scrub_stamp":"2023-10-06T03:29:28.357506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T14:43:09.107140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.629976+0000","last_change":"2023-10-06T03:29:38.626568+0000","last_active":"2023-10-06T03:29:38.629976+0000","last_peered":"2023-10-06T03:29:38.629976+0000","last_clean":"2023-10-06T03:29:38.629976+0000","last_became_active":"2023-10-06T03:29:38.625323+0000","last_became_peered":"2023-10-06T03:29:38.625323+0000","last_unstale":"2023-10-06T03:29:38.629976+0000","last_undegraded":"2023-10-06T03:29:38.629976+0000","last_fullsized":"2023-10-06T03:29:38.629976+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:37.570693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:37.570693+0000","last_clean_scrub_stamp":"2023-10-06T03:29:37.570693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T04:51:55.443872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0,4],"acting":[5,1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.609554+0000","last_change":"2023-10-06T03:29:34.549281+0000","last_active":"2023-10-06T03:29:38.609554+0000","last_peered":"2023-10-06T03:29:38.609554+0000","last_clean":"2023-10-06T03:29:38.609554+0000","last_became_active":"2023-10-06T03:29:34.548900+0000","last_became_peered":"2023-10-06T03:29:34.548900+0000","last_unstale":"2023-10-06T03:29:38.609554+0000","last_undegraded":"2023-10-06T03:29:38.609554+0000","last_fullsized":"2023-10-06T03:29:38.609554+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_clean_scrub_stamp":"2023-10-06T03:29:33.500929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T09:06:08.844003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":14,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.589353+0000","last_change":"2023-10-06T03:29:33.534969+0000","last_active":"2023-10-06T03:29:38.589353+0000","last_peered":"2023-10-06T03:29:38.589353+0000","last_clean":"2023-10-06T03:29:38.589353+0000","last_became_active":"2023-10-06T03:29:33.534402+0000","last_became_peered":"2023-10-06T03:29:33.534402+0000","last_unstale":"2023-10-06T03:29:38.589353+0000","last_undegraded":"2023-10-06T03:29:38.589353+0000","last_fullsized":"2023-10-06T03:29:38.589353+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:32.495616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:32.495616+0000","last_clean_scrub_stamp":"2023-10-06T03:29:32.495616+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T11:00:52.451461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":17,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.601254+0000","last_change":"2023-10-06T03:29:29.386120+0000","last_active":"2023-10-06T03:29:38.601254+0000","last_peered":"2023-10-06T03:29:38.601254+0000","last_clean":"2023-10-06T03:29:38.601254+0000","last_became_active":"2023-10-06T03:29:29.385692+0000","last_became_peered":"2023-10-06T03:29:29.385692+0000","last_unstale":"2023-10-06T03:29:38.601254+0000","last_undegraded":"2023-10-06T03:29:38.601254+0000","last_fullsized":"2023-10-06T03:29:38.601254+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:28.357506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:28.357506+0000","last_clean_scrub_stamp":"2023-10-06T03:29:28.357506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T10:21:35.864895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":169,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.603558+0000","last_change":"2023-10-06T03:29:21.978311+0000","last_active":"2023-10-06T03:29:38.603558+0000","last_peered":"2023-10-06T03:29:38.603558+0000","last_clean":"2023-10-06T03:29:38.603558+0000","last_became_active":"2023-10-06T03:29:21.978017+0000","last_became_peered":"2023-10-06T03:29:21.978017+0000","last_unstale":"2023-10-06T03:29:38.603558+0000","last_undegraded":"2023-10-06T03:29:38.603558+0000","last_fullsized":"2023-10-06T03:29:38.603558+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:20.957955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:20.957955+0000","last_clean_scrub_stamp":"2023-10-06T03:29:20.957955+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T07:33:10.429084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.614806+0000","last_change":"2023-10-06T03:29:38.612549+0000","last_active":"2023-10-06T03:29:38.614806+0000","last_peered":"2023-10-06T03:29:38.614806+0000","last_clean":"2023-10-06T03:29:38.614806+0000","last_became_active":"2023-10-06T03:29:38.611790+0000","last_became_peered":"2023-10-06T03:29:38.611790+0000","last_unstale":"2023-10-06T03:29:38.614806+0000","last_undegraded":"2023-10-06T03:29:38.614806+0000","last_fullsized":"2023-10-06T03:29:38.614806+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:37.570693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:37.570693+0000","last_clean_scrub_stamp":"2023-10-06T03:29:37.570693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T12:17:41.605048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5,3],"acting":[7,2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.608679+0000","last_change":"2023-10-06T03:29:34.545006+0000","last_active":"2023-10-06T03:29:38.608679+0000","last_peered":"2023-10-06T03:29:38.608679+0000","last_clean":"2023-10-06T03:29:38.608679+0000","last_became_active":"2023-10-06T03:29:34.544682+0000","last_became_peered":"2023-10-06T03:29:34.544682+0000","last_unstale":"2023-10-06T03:29:38.608679+0000","last_undegraded":"2023-10-06T03:29:38.608679+0000","last_fullsized":"2023-10-06T03:29:38.608679+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_clean_scrub_stamp":"2023-10-06T03:29:33.500929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T03:59:12.193877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":14,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.589559+0000","last_change":"2023-10-06T03:29:33.545053+0000","last_active":"2023-10-06T03:29:38.589559+0000","last_peered":"2023-10-06T03:29:38.589559+0000","last_clean":"2023-10-06T03:29:38.589559+0000","last_became_active":"2023-10-06T03:29:33.544749+0000","last_became_peered":"2023-10-06T03:29:33.544749+0000","last_unstale":"2023-10-06T03:29:38.589559+0000","last_undegraded":"2023-10-06T03:29:38.589559+0000","last_fullsized":"2023-10-06T03:29:38.589559+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:32.495616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:32.495616+0000","last_clean_scrub_stamp":"2023-10-06T03:29:32.495616+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T14:26:49.262597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.621981+0000","last_change":"2023-10-06T03:29:38.614480+0000","last_active":"2023-10-06T03:29:38.621981+0000","last_peered":"2023-10-06T03:29:38.621981+0000","last_clean":"2023-10-06T03:29:38.621981+0000","last_became_active":"2023-10-06T03:29:38.613976+0000","last_became_peered":"2023-10-06T03:29:38.613976+0000","last_unstale":"2023-10-06T03:29:38.621981+0000","last_undegraded":"2023-10-06T03:29:38.621981+0000","last_fullsized":"2023-10-06T03:29:38.621981+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:37.570693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:37.570693+0000","last_clean_scrub_stamp":"2023-10-06T03:29:37.570693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T09:04:13.741349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0,2],"acting":[4,5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.592722+0000","last_change":"2023-10-06T03:29:34.551311+0000","last_active":"2023-10-06T03:29:38.592722+0000","last_peered":"2023-10-06T03:29:38.592722+0000","last_clean":"2023-10-06T03:29:38.592722+0000","last_became_active":"2023-10-06T03:29:34.550961+0000","last_became_peered":"2023-10-06T03:29:34.550961+0000","last_unstale":"2023-10-06T03:29:38.592722+0000","last_undegraded":"2023-10-06T03:29:38.592722+0000","last_fullsized":"2023-10-06T03:29:38.592722+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_clean_scrub_stamp":"2023-10-06T03:29:33.500929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T14:20:50.168977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":14,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.592869+0000","last_change":"2023-10-06T03:29:33.538732+0000","last_active":"2023-10-06T03:29:38.592869+0000","last_peered":"2023-10-06T03:29:38.592869+0000","last_clean":"2023-10-06T03:29:38.592869+0000","last_became_active":"2023-10-06T03:29:33.538430+0000","last_became_peered":"2023-10-06T03:29:33.538430+0000","last_unstale":"2023-10-06T03:29:38.592869+0000","last_undegraded":"2023-10-06T03:29:38.592869+0000","last_fullsized":"2023-10-06T03:29:38.592869+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:32.495616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:32.495616+0000","last_clean_scrub_stamp":"2023-10-06T03:29:32.495616+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T12:57:50.614907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.621727+0000","last_change":"2023-10-06T03:29:38.617352+0000","last_active":"2023-10-06T03:29:38.621727+0000","last_peered":"2023-10-06T03:29:38.621727+0000","last_clean":"2023-10-06T03:29:38.621727+0000","last_became_active":"2023-10-06T03:29:38.616718+0000","last_became_peered":"2023-10-06T03:29:38.616718+0000","last_unstale":"2023-10-06T03:29:38.621727+0000","last_undegraded":"2023-10-06T03:29:38.621727+0000","last_fullsized":"2023-10-06T03:29:38.621727+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:37.570693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:37.570693+0000","last_clean_scrub_stamp":"2023-10-06T03:29:37.570693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T11:16:01.323266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6,2],"acting":[5,4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.592923+0000","last_change":"2023-10-06T03:29:34.541404+0000","last_active":"2023-10-06T03:29:38.592923+0000","last_peered":"2023-10-06T03:29:38.592923+0000","last_clean":"2023-10-06T03:29:38.592923+0000","last_became_active":"2023-10-06T03:29:34.541037+0000","last_became_peered":"2023-10-06T03:29:34.541037+0000","last_unstale":"2023-10-06T03:29:38.592923+0000","last_undegraded":"2023-10-06T03:29:38.592923+0000","last_fullsized":"2023-10-06T03:29:38.592923+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_clean_scrub_stamp":"2023-10-06T03:29:33.500929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T08:57:07.064845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.610030+0000","last_change":"2023-10-06T03:29:33.542889+0000","last_active":"2023-10-06T03:29:38.610030+0000","last_peered":"2023-10-06T03:29:38.610030+0000","last_clean":"2023-10-06T03:29:38.610030+0000","last_became_active":"2023-10-06T03:29:33.542527+0000","last_became_peered":"2023-10-06T03:29:33.542527+0000","last_unstale":"2023-10-06T03:29:38.610030+0000","last_undegraded":"2023-10-06T03:29:38.610030+0000","last_fullsized":"2023-10-06T03:29:38.610030+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:32.495616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:32.495616+0000","last_clean_scrub_stamp":"2023-10-06T03:29:32.495616+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T09:16:59.387601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.624663+0000","last_change":"2023-10-06T03:29:38.616847+0000","last_active":"2023-10-06T03:29:38.624663+0000","last_peered":"2023-10-06T03:29:38.624663+0000","last_clean":"2023-10-06T03:29:38.624663+0000","last_became_active":"2023-10-06T03:29:38.616359+0000","last_became_peered":"2023-10-06T03:29:38.616359+0000","last_unstale":"2023-10-06T03:29:38.624663+0000","last_undegraded":"2023-10-06T03:29:38.624663+0000","last_fullsized":"2023-10-06T03:29:38.624663+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:37.570693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:37.570693+0000","last_clean_scrub_stamp":"2023-10-06T03:29:37.570693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T10:33:46.594949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0,6],"acting":[1,4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.593466+0000","last_change":"2023-10-06T03:29:34.542358+0000","last_active":"2023-10-06T03:29:38.593466+0000","last_peered":"2023-10-06T03:29:38.593466+0000","last_clean":"2023-10-06T03:29:38.593466+0000","last_became_active":"2023-10-06T03:29:34.542038+0000","last_became_peered":"2023-10-06T03:29:34.542038+0000","last_unstale":"2023-10-06T03:29:38.593466+0000","last_undegraded":"2023-10-06T03:29:38.593466+0000","last_fullsized":"2023-10-06T03:29:38.593466+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_clean_scrub_stamp":"2023-10-06T03:29:33.500929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T11:19:26.387732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"23'2","reported_seq":16,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.590740+0000","last_change":"2023-10-06T03:29:33.544626+0000","last_active":"2023-10-06T03:29:38.590740+0000","last_peered":"2023-10-06T03:29:38.590740+0000","last_clean":"2023-10-06T03:29:38.590740+0000","last_became_active":"2023-10-06T03:29:33.544301+0000","last_became_peered":"2023-10-06T03:29:33.544301+0000","last_unstale":"2023-10-06T03:29:38.590740+0000","last_undegraded":"2023-10-06T03:29:38.590740+0000","last_fullsized":"2023-10-06T03:29:38.590740+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:32.495616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:32.495616+0000","last_clean_scrub_stamp":"2023-10-06T03:29:32.495616+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T09:13:04.451900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.613447+0000","last_change":"2023-10-06T03:29:38.610963+0000","last_active":"2023-10-06T03:29:38.613447+0000","last_peered":"2023-10-06T03:29:38.613447+0000","last_clean":"2023-10-06T03:29:38.613447+0000","last_became_active":"2023-10-06T03:29:38.610492+0000","last_became_peered":"2023-10-06T03:29:38.610492+0000","last_unstale":"2023-10-06T03:29:38.613447+0000","last_undegraded":"2023-10-06T03:29:38.613447+0000","last_fullsized":"2023-10-06T03:29:38.613447+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:37.570693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:37.570693+0000","last_clean_scrub_stamp":"2023-10-06T03:29:37.570693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T04:15:31.186577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5,6],"acting":[2,7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.592107+0000","last_change":"2023-10-06T03:29:34.547048+0000","last_active":"2023-10-06T03:29:38.592107+0000","last_peered":"2023-10-06T03:29:38.592107+0000","last_clean":"2023-10-06T03:29:38.592107+0000","last_became_active":"2023-10-06T03:29:34.546757+0000","last_became_peered":"2023-10-06T03:29:34.546757+0000","last_unstale":"2023-10-06T03:29:38.592107+0000","last_undegraded":"2023-10-06T03:29:38.592107+0000","last_fullsized":"2023-10-06T03:29:38.592107+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_clean_scrub_stamp":"2023-10-06T03:29:33.500929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T14:20:50.168977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":14,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.600978+0000","last_change":"2023-10-06T03:29:33.537618+0000","last_active":"2023-10-06T03:29:38.600978+0000","last_peered":"2023-10-06T03:29:38.600978+0000","last_clean":"2023-10-06T03:29:38.600978+0000","last_became_active":"2023-10-06T03:29:33.537140+0000","last_became_peered":"2023-10-06T03:29:33.537140+0000","last_unstale":"2023-10-06T03:29:38.600978+0000","last_undegraded":"2023-10-06T03:29:38.600978+0000","last_fullsized":"2023-10-06T03:29:38.600978+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:32.495616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:32.495616+0000","last_clean_scrub_stamp":"2023-10-06T03:29:32.495616+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T12:25:10.766051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.623146+0000","last_change":"2023-10-06T03:29:38.616303+0000","last_active":"2023-10-06T03:29:38.623146+0000","last_peered":"2023-10-06T03:29:38.623146+0000","last_clean":"2023-10-06T03:29:38.623146+0000","last_became_active":"2023-10-06T03:29:38.615882+0000","last_became_peered":"2023-10-06T03:29:38.615882+0000","last_unstale":"2023-10-06T03:29:38.623146+0000","last_undegraded":"2023-10-06T03:29:38.623146+0000","last_fullsized":"2023-10-06T03:29:38.623146+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:37.570693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:37.570693+0000","last_clean_scrub_stamp":"2023-10-06T03:29:37.570693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T11:19:30.457497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3,6],"acting":[2,4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.610059+0000","last_change":"2023-10-06T03:29:34.546395+0000","last_active":"2023-10-06T03:29:38.610059+0000","last_peered":"2023-10-06T03:29:38.610059+0000","last_clean":"2023-10-06T03:29:38.610059+0000","last_became_active":"2023-10-06T03:29:34.546016+0000","last_became_peered":"2023-10-06T03:29:34.546016+0000","last_unstale":"2023-10-06T03:29:38.610059+0000","last_undegraded":"2023-10-06T03:29:38.610059+0000","last_fullsized":"2023-10-06T03:29:38.610059+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_clean_scrub_stamp":"2023-10-06T03:29:33.500929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T14:51:50.895276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"23'1","reported_seq":14,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.605833+0000","last_change":"2023-10-06T03:29:33.536994+0000","last_active":"2023-10-06T03:29:38.605833+0000","last_peered":"2023-10-06T03:29:38.605833+0000","last_clean":"2023-10-06T03:29:38.605833+0000","last_became_active":"2023-10-06T03:29:33.535768+0000","last_became_peered":"2023-10-06T03:29:33.535768+0000","last_unstale":"2023-10-06T03:29:38.605833+0000","last_undegraded":"2023-10-06T03:29:38.605833+0000","last_fullsized":"2023-10-06T03:29:38.605833+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:32.495616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:32.495616+0000","last_clean_scrub_stamp":"2023-10-06T03:29:32.495616+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T06:53:50.776266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.615680+0000","last_change":"2023-10-06T03:29:38.614617+0000","last_active":"2023-10-06T03:29:38.615680+0000","last_peered":"2023-10-06T03:29:38.615680+0000","last_clean":"2023-10-06T03:29:38.615680+0000","last_became_active":"2023-10-06T03:29:38.614134+0000","last_became_peered":"2023-10-06T03:29:38.614134+0000","last_unstale":"2023-10-06T03:29:38.615680+0000","last_undegraded":"2023-10-06T03:29:38.615680+0000","last_fullsized":"2023-10-06T03:29:38.615680+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:37.570693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:37.570693+0000","last_clean_scrub_stamp":"2023-10-06T03:29:37.570693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T07:51:25.532733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5,6],"acting":[2,0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.592039+0000","last_change":"2023-10-06T03:29:34.535355+0000","last_active":"2023-10-06T03:29:38.592039+0000","last_peered":"2023-10-06T03:29:38.592039+0000","last_clean":"2023-10-06T03:29:38.592039+0000","last_became_active":"2023-10-06T03:29:34.534944+0000","last_became_peered":"2023-10-06T03:29:34.534944+0000","last_unstale":"2023-10-06T03:29:38.592039+0000","last_undegraded":"2023-10-06T03:29:38.592039+0000","last_fullsized":"2023-10-06T03:29:38.592039+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_clean_scrub_stamp":"2023-10-06T03:29:33.500929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T09:39:33.168497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"23'2","reported_seq":15,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.605956+0000","last_change":"2023-10-06T03:29:33.545575+0000","last_active":"2023-10-06T03:29:38.605956+0000","last_peered":"2023-10-06T03:29:38.605956+0000","last_clean":"2023-10-06T03:29:38.605956+0000","last_became_active":"2023-10-06T03:29:33.545110+0000","last_became_peered":"2023-10-06T03:29:33.545110+0000","last_unstale":"2023-10-06T03:29:38.605956+0000","last_undegraded":"2023-10-06T03:29:38.605956+0000","last_fullsized":"2023-10-06T03:29:38.605956+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:32.495616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:32.495616+0000","last_clean_scrub_stamp":"2023-10-06T03:29:32.495616+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T14:51:49.889964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.623767+0000","last_change":"2023-10-06T03:29:38.620856+0000","last_active":"2023-10-06T03:29:38.623767+0000","last_peered":"2023-10-06T03:29:38.623767+0000","last_clean":"2023-10-06T03:29:38.623767+0000","last_became_active":"2023-10-06T03:29:38.620374+0000","last_became_peered":"2023-10-06T03:29:38.620374+0000","last_unstale":"2023-10-06T03:29:38.623767+0000","last_undegraded":"2023-10-06T03:29:38.623767+0000","last_fullsized":"2023-10-06T03:29:38.623767+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:37.570693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:37.570693+0000","last_clean_scrub_stamp":"2023-10-06T03:29:37.570693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T08:05:22.622058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1,5],"acting":[2,0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.600871+0000","last_change":"2023-10-06T03:29:34.537767+0000","last_active":"2023-10-06T03:29:38.600871+0000","last_peered":"2023-10-06T03:29:38.600871+0000","last_clean":"2023-10-06T03:29:38.600871+0000","last_became_active":"2023-10-06T03:29:34.537414+0000","last_became_peered":"2023-10-06T03:29:34.537414+0000","last_unstale":"2023-10-06T03:29:38.600871+0000","last_undegraded":"2023-10-06T03:29:38.600871+0000","last_fullsized":"2023-10-06T03:29:38.600871+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_clean_scrub_stamp":"2023-10-06T03:29:33.500929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T08:29:43.429905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"23'1","reported_seq":14,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.606271+0000","last_change":"2023-10-06T03:29:33.543577+0000","last_active":"2023-10-06T03:29:38.606271+0000","last_peered":"2023-10-06T03:29:38.606271+0000","last_clean":"2023-10-06T03:29:38.606271+0000","last_became_active":"2023-10-06T03:29:33.543265+0000","last_became_peered":"2023-10-06T03:29:33.543265+0000","last_unstale":"2023-10-06T03:29:38.606271+0000","last_undegraded":"2023-10-06T03:29:38.606271+0000","last_fullsized":"2023-10-06T03:29:38.606271+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:32.495616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:32.495616+0000","last_clean_scrub_stamp":"2023-10-06T03:29:32.495616+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T09:06:07.838690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.623296+0000","last_change":"2023-10-06T03:29:38.619910+0000","last_active":"2023-10-06T03:29:38.623296+0000","last_peered":"2023-10-06T03:29:38.623296+0000","last_clean":"2023-10-06T03:29:38.623296+0000","last_became_active":"2023-10-06T03:29:38.619517+0000","last_became_peered":"2023-10-06T03:29:38.619517+0000","last_unstale":"2023-10-06T03:29:38.623296+0000","last_undegraded":"2023-10-06T03:29:38.623296+0000","last_fullsized":"2023-10-06T03:29:38.623296+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:37.570693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:37.570693+0000","last_clean_scrub_stamp":"2023-10-06T03:29:37.570693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T04:29:21.981307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3,4],"acting":[7,6,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.627322+0000","last_change":"2023-10-06T03:29:38.623954+0000","last_active":"2023-10-06T03:29:38.627322+0000","last_peered":"2023-10-06T03:29:38.627322+0000","last_clean":"2023-10-06T03:29:38.627322+0000","last_became_active":"2023-10-06T03:29:38.623537+0000","last_became_peered":"2023-10-06T03:29:38.623537+0000","last_unstale":"2023-10-06T03:29:38.627322+0000","last_undegraded":"2023-10-06T03:29:38.627322+0000","last_fullsized":"2023-10-06T03:29:38.627322+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:37.570693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:37.570693+0000","last_clean_scrub_stamp":"2023-10-06T03:29:37.570693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T07:38:31.118869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1,6],"acting":[5,3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"23'8","reported_seq":21,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.609831+0000","last_change":"2023-10-06T03:29:33.538136+0000","last_active":"2023-10-06T03:29:38.609831+0000","last_peered":"2023-10-06T03:29:38.609831+0000","last_clean":"2023-10-06T03:29:38.609831+0000","last_became_active":"2023-10-06T03:29:33.537656+0000","last_became_peered":"2023-10-06T03:29:33.537656+0000","last_unstale":"2023-10-06T03:29:38.609831+0000","last_undegraded":"2023-10-06T03:29:38.609831+0000","last_fullsized":"2023-10-06T03:29:38.609831+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:32.495616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:32.495616+0000","last_clean_scrub_stamp":"2023-10-06T03:29:32.495616+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T07:23:17.430374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.592883+0000","last_change":"2023-10-06T03:29:34.546204+0000","last_active":"2023-10-06T03:29:38.592883+0000","last_peered":"2023-10-06T03:29:38.592883+0000","last_clean":"2023-10-06T03:29:38.592883+0000","last_became_active":"2023-10-06T03:29:34.545585+0000","last_became_peered":"2023-10-06T03:29:34.545585+0000","last_unstale":"2023-10-06T03:29:38.592883+0000","last_undegraded":"2023-10-06T03:29:38.592883+0000","last_fullsized":"2023-10-06T03:29:38.592883+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_clean_scrub_stamp":"2023-10-06T03:29:33.500929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T05:40:45.388611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.620710+0000","last_change":"2023-10-06T03:29:38.612955+0000","last_active":"2023-10-06T03:29:38.620710+0000","last_peered":"2023-10-06T03:29:38.620710+0000","last_clean":"2023-10-06T03:29:38.620710+0000","last_became_active":"2023-10-06T03:29:38.612404+0000","last_became_peered":"2023-10-06T03:29:38.612404+0000","last_unstale":"2023-10-06T03:29:38.620710+0000","last_undegraded":"2023-10-06T03:29:38.620710+0000","last_fullsized":"2023-10-06T03:29:38.620710+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:37.570693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:37.570693+0000","last_clean_scrub_stamp":"2023-10-06T03:29:37.570693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T08:25:07.069848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7,0],"acting":[3,1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.603401+0000","last_change":"2023-10-06T03:29:33.545324+0000","last_active":"2023-10-06T03:29:38.603401+0000","last_peered":"2023-10-06T03:29:38.603401+0000","last_clean":"2023-10-06T03:29:38.603401+0000","last_became_active":"2023-10-06T03:29:33.545029+0000","last_became_peered":"2023-10-06T03:29:33.545029+0000","last_unstale":"2023-10-06T03:29:38.603401+0000","last_undegraded":"2023-10-06T03:29:38.603401+0000","last_fullsized":"2023-10-06T03:29:38.603401+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:32.495616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:32.495616+0000","last_clean_scrub_stamp":"2023-10-06T03:29:32.495616+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T07:14:02.648208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.599807+0000","last_change":"2023-10-06T03:29:34.535716+0000","last_active":"2023-10-06T03:29:38.599807+0000","last_peered":"2023-10-06T03:29:38.599807+0000","last_clean":"2023-10-06T03:29:38.599807+0000","last_became_active":"2023-10-06T03:29:34.535107+0000","last_became_peered":"2023-10-06T03:29:34.535107+0000","last_unstale":"2023-10-06T03:29:38.599807+0000","last_undegraded":"2023-10-06T03:29:38.599807+0000","last_fullsized":"2023-10-06T03:29:38.599807+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_clean_scrub_stamp":"2023-10-06T03:29:33.500929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T11:19:26.387732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":14,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.591766+0000","last_change":"2023-10-06T03:29:33.541792+0000","last_active":"2023-10-06T03:29:38.591766+0000","last_peered":"2023-10-06T03:29:38.591766+0000","last_clean":"2023-10-06T03:29:38.591766+0000","last_became_active":"2023-10-06T03:29:33.535547+0000","last_became_peered":"2023-10-06T03:29:33.535547+0000","last_unstale":"2023-10-06T03:29:38.591766+0000","last_undegraded":"2023-10-06T03:29:38.591766+0000","last_fullsized":"2023-10-06T03:29:38.591766+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:32.495616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:32.495616+0000","last_clean_scrub_stamp":"2023-10-06T03:29:32.495616+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T07:55:07.787102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.602006+0000","last_change":"2023-10-06T03:29:34.540666+0000","last_active":"2023-10-06T03:29:38.602006+0000","last_peered":"2023-10-06T03:29:38.602006+0000","last_clean":"2023-10-06T03:29:38.602006+0000","last_became_active":"2023-10-06T03:29:34.540176+0000","last_became_peered":"2023-10-06T03:29:34.540176+0000","last_unstale":"2023-10-06T03:29:38.602006+0000","last_undegraded":"2023-10-06T03:29:38.602006+0000","last_fullsized":"2023-10-06T03:29:38.602006+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_clean_scrub_stamp":"2023-10-06T03:29:33.500929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T14:11:35.669872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.617535+0000","last_change":"2023-10-06T03:29:38.610899+0000","last_active":"2023-10-06T03:29:38.617535+0000","last_peered":"2023-10-06T03:29:38.617535+0000","last_clean":"2023-10-06T03:29:38.617535+0000","last_became_active":"2023-10-06T03:29:38.610494+0000","last_became_peered":"2023-10-06T03:29:38.610494+0000","last_unstale":"2023-10-06T03:29:38.617535+0000","last_undegraded":"2023-10-06T03:29:38.617535+0000","last_fullsized":"2023-10-06T03:29:38.617535+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:37.570693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:37.570693+0000","last_clean_scrub_stamp":"2023-10-06T03:29:37.570693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T08:19:47.139035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2,1],"acting":[3,7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"23'2","reported_seq":15,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.604109+0000","last_change":"2023-10-06T03:29:33.544328+0000","last_active":"2023-10-06T03:29:38.604109+0000","last_peered":"2023-10-06T03:29:38.604109+0000","last_clean":"2023-10-06T03:29:38.604109+0000","last_became_active":"2023-10-06T03:29:33.544010+0000","last_became_peered":"2023-10-06T03:29:33.544010+0000","last_unstale":"2023-10-06T03:29:38.604109+0000","last_undegraded":"2023-10-06T03:29:38.604109+0000","last_fullsized":"2023-10-06T03:29:38.604109+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:32.495616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:32.495616+0000","last_clean_scrub_stamp":"2023-10-06T03:29:32.495616+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T11:17:52.119337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.592119+0000","last_change":"2023-10-06T03:29:34.551765+0000","last_active":"2023-10-06T03:29:38.592119+0000","last_peered":"2023-10-06T03:29:38.592119+0000","last_clean":"2023-10-06T03:29:38.592119+0000","last_became_active":"2023-10-06T03:29:34.551437+0000","last_became_peered":"2023-10-06T03:29:34.551437+0000","last_unstale":"2023-10-06T03:29:38.592119+0000","last_undegraded":"2023-10-06T03:29:38.592119+0000","last_fullsized":"2023-10-06T03:29:38.592119+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_clean_scrub_stamp":"2023-10-06T03:29:33.500929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T11:47:20.603758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.629984+0000","last_change":"2023-10-06T03:29:38.626781+0000","last_active":"2023-10-06T03:29:38.629984+0000","last_peered":"2023-10-06T03:29:38.629984+0000","last_clean":"2023-10-06T03:29:38.629984+0000","last_became_active":"2023-10-06T03:29:38.625489+0000","last_became_peered":"2023-10-06T03:29:38.625489+0000","last_unstale":"2023-10-06T03:29:38.629984+0000","last_undegraded":"2023-10-06T03:29:38.629984+0000","last_fullsized":"2023-10-06T03:29:38.629984+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:37.570693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:37.570693+0000","last_clean_scrub_stamp":"2023-10-06T03:29:37.570693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T06:13:47.912662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3,0],"acting":[5,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.20","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.609684+0000","last_change":"2023-10-06T03:29:34.546226+0000","last_active":"2023-10-06T03:29:38.609684+0000","last_peered":"2023-10-06T03:29:38.609684+0000","last_clean":"2023-10-06T03:29:38.609684+0000","last_became_active":"2023-10-06T03:29:34.545400+0000","last_became_peered":"2023-10-06T03:29:34.545400+0000","last_unstale":"2023-10-06T03:29:38.609684+0000","last_undegraded":"2023-10-06T03:29:38.609684+0000","last_fullsized":"2023-10-06T03:29:38.609684+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_clean_scrub_stamp":"2023-10-06T03:29:33.500929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T07:55:08.792414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.21","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.588987+0000","last_change":"2023-10-06T03:29:34.551672+0000","last_active":"2023-10-06T03:29:38.588987+0000","last_peered":"2023-10-06T03:29:38.588987+0000","last_clean":"2023-10-06T03:29:38.588987+0000","last_became_active":"2023-10-06T03:29:34.551337+0000","last_became_peered":"2023-10-06T03:29:34.551337+0000","last_unstale":"2023-10-06T03:29:38.588987+0000","last_undegraded":"2023-10-06T03:29:38.588987+0000","last_fullsized":"2023-10-06T03:29:38.588987+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_clean_scrub_stamp":"2023-10-06T03:29:33.500929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T05:15:29.367282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.22","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.592542+0000","last_change":"2023-10-06T03:29:34.549749+0000","last_active":"2023-10-06T03:29:38.592542+0000","last_peered":"2023-10-06T03:29:38.592542+0000","last_clean":"2023-10-06T03:29:38.592542+0000","last_became_active":"2023-10-06T03:29:34.549468+0000","last_became_peered":"2023-10-06T03:29:34.549468+0000","last_unstale":"2023-10-06T03:29:38.592542+0000","last_undegraded":"2023-10-06T03:29:38.592542+0000","last_fullsized":"2023-10-06T03:29:38.592542+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_clean_scrub_stamp":"2023-10-06T03:29:33.500929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T14:11:35.669872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.23","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.590075+0000","last_change":"2023-10-06T03:29:34.545511+0000","last_active":"2023-10-06T03:29:38.590075+0000","last_peered":"2023-10-06T03:29:38.590075+0000","last_clean":"2023-10-06T03:29:38.590075+0000","last_became_active":"2023-10-06T03:29:34.545183+0000","last_became_peered":"2023-10-06T03:29:34.545183+0000","last_unstale":"2023-10-06T03:29:38.590075+0000","last_undegraded":"2023-10-06T03:29:38.590075+0000","last_fullsized":"2023-10-06T03:29:38.590075+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_clean_scrub_stamp":"2023-10-06T03:29:33.500929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T07:54:17.345077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.24","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.609286+0000","last_change":"2023-10-06T03:29:34.543928+0000","last_active":"2023-10-06T03:29:38.609286+0000","last_peered":"2023-10-06T03:29:38.609286+0000","last_clean":"2023-10-06T03:29:38.609286+0000","last_became_active":"2023-10-06T03:29:34.543604+0000","last_became_peered":"2023-10-06T03:29:34.543604+0000","last_unstale":"2023-10-06T03:29:38.609286+0000","last_undegraded":"2023-10-06T03:29:38.609286+0000","last_fullsized":"2023-10-06T03:29:38.609286+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_clean_scrub_stamp":"2023-10-06T03:29:33.500929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T09:15:40.932270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.25","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.601918+0000","last_change":"2023-10-06T03:29:34.540446+0000","last_active":"2023-10-06T03:29:38.601918+0000","last_peered":"2023-10-06T03:29:38.601918+0000","last_clean":"2023-10-06T03:29:38.601918+0000","last_became_active":"2023-10-06T03:29:34.539827+0000","last_became_peered":"2023-10-06T03:29:34.539827+0000","last_unstale":"2023-10-06T03:29:38.601918+0000","last_undegraded":"2023-10-06T03:29:38.601918+0000","last_fullsized":"2023-10-06T03:29:38.601918+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_clean_scrub_stamp":"2023-10-06T03:29:33.500929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T07:28:42.661588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.26","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.609415+0000","last_change":"2023-10-06T03:29:34.538237+0000","last_active":"2023-10-06T03:29:38.609415+0000","last_peered":"2023-10-06T03:29:38.609415+0000","last_clean":"2023-10-06T03:29:38.609415+0000","last_became_active":"2023-10-06T03:29:34.537636+0000","last_became_peered":"2023-10-06T03:29:34.537636+0000","last_unstale":"2023-10-06T03:29:38.609415+0000","last_undegraded":"2023-10-06T03:29:38.609415+0000","last_fullsized":"2023-10-06T03:29:38.609415+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_clean_scrub_stamp":"2023-10-06T03:29:33.500929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T05:09:24.657371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.27","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.608601+0000","last_change":"2023-10-06T03:29:34.549127+0000","last_active":"2023-10-06T03:29:38.608601+0000","last_peered":"2023-10-06T03:29:38.608601+0000","last_clean":"2023-10-06T03:29:38.608601+0000","last_became_active":"2023-10-06T03:29:34.548839+0000","last_became_peered":"2023-10-06T03:29:34.548839+0000","last_unstale":"2023-10-06T03:29:38.608601+0000","last_undegraded":"2023-10-06T03:29:38.608601+0000","last_fullsized":"2023-10-06T03:29:38.608601+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_clean_scrub_stamp":"2023-10-06T03:29:33.500929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T08:03:23.482409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.28","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.603462+0000","last_change":"2023-10-06T03:29:34.544177+0000","last_active":"2023-10-06T03:29:38.603462+0000","last_peered":"2023-10-06T03:29:38.603462+0000","last_clean":"2023-10-06T03:29:38.603462+0000","last_became_active":"2023-10-06T03:29:34.543872+0000","last_became_peered":"2023-10-06T03:29:34.543872+0000","last_unstale":"2023-10-06T03:29:38.603462+0000","last_undegraded":"2023-10-06T03:29:38.603462+0000","last_fullsized":"2023-10-06T03:29:38.603462+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_clean_scrub_stamp":"2023-10-06T03:29:33.500929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T06:09:15.729486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.29","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.588861+0000","last_change":"2023-10-06T03:29:34.540282+0000","last_active":"2023-10-06T03:29:38.588861+0000","last_peered":"2023-10-06T03:29:38.588861+0000","last_clean":"2023-10-06T03:29:38.588861+0000","last_became_active":"2023-10-06T03:29:34.539988+0000","last_became_peered":"2023-10-06T03:29:34.539988+0000","last_unstale":"2023-10-06T03:29:38.588861+0000","last_undegraded":"2023-10-06T03:29:38.588861+0000","last_fullsized":"2023-10-06T03:29:38.588861+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_clean_scrub_stamp":"2023-10-06T03:29:33.500929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T07:36:17.771693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.2a","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.591640+0000","last_change":"2023-10-06T03:29:34.547360+0000","last_active":"2023-10-06T03:29:38.591640+0000","last_peered":"2023-10-06T03:29:38.591640+0000","last_clean":"2023-10-06T03:29:38.591640+0000","last_became_active":"2023-10-06T03:29:34.547086+0000","last_became_peered":"2023-10-06T03:29:34.547086+0000","last_unstale":"2023-10-06T03:29:38.591640+0000","last_undegraded":"2023-10-06T03:29:38.591640+0000","last_fullsized":"2023-10-06T03:29:38.591640+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_clean_scrub_stamp":"2023-10-06T03:29:33.500929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T07:25:41.452975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2b","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.604010+0000","last_change":"2023-10-06T03:29:34.544346+0000","last_active":"2023-10-06T03:29:38.604010+0000","last_peered":"2023-10-06T03:29:38.604010+0000","last_clean":"2023-10-06T03:29:38.604010+0000","last_became_active":"2023-10-06T03:29:34.543976+0000","last_became_peered":"2023-10-06T03:29:34.543976+0000","last_unstale":"2023-10-06T03:29:38.604010+0000","last_undegraded":"2023-10-06T03:29:38.604010+0000","last_fullsized":"2023-10-06T03:29:38.604010+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_clean_scrub_stamp":"2023-10-06T03:29:33.500929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T14:11:35.669872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.2c","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.593246+0000","last_change":"2023-10-06T03:29:34.549013+0000","last_active":"2023-10-06T03:29:38.593246+0000","last_peered":"2023-10-06T03:29:38.593246+0000","last_clean":"2023-10-06T03:29:38.593246+0000","last_became_active":"2023-10-06T03:29:34.548707+0000","last_became_peered":"2023-10-06T03:29:34.548707+0000","last_unstale":"2023-10-06T03:29:38.593246+0000","last_undegraded":"2023-10-06T03:29:38.593246+0000","last_fullsized":"2023-10-06T03:29:38.593246+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_clean_scrub_stamp":"2023-10-06T03:29:33.500929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T14:34:26.405025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2d","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.598606+0000","last_change":"2023-10-06T03:29:34.545639+0000","last_active":"2023-10-06T03:29:38.598606+0000","last_peered":"2023-10-06T03:29:38.598606+0000","last_clean":"2023-10-06T03:29:38.598606+0000","last_became_active":"2023-10-06T03:29:34.545316+0000","last_became_peered":"2023-10-06T03:29:34.545316+0000","last_unstale":"2023-10-06T03:29:38.598606+0000","last_undegraded":"2023-10-06T03:29:38.598606+0000","last_fullsized":"2023-10-06T03:29:38.598606+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_clean_scrub_stamp":"2023-10-06T03:29:33.500929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T11:58:54.143086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2e","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.592479+0000","last_change":"2023-10-06T03:29:34.532132+0000","last_active":"2023-10-06T03:29:38.592479+0000","last_peered":"2023-10-06T03:29:38.592479+0000","last_clean":"2023-10-06T03:29:38.592479+0000","last_became_active":"2023-10-06T03:29:34.531862+0000","last_became_peered":"2023-10-06T03:29:34.531862+0000","last_unstale":"2023-10-06T03:29:38.592479+0000","last_undegraded":"2023-10-06T03:29:38.592479+0000","last_fullsized":"2023-10-06T03:29:38.592479+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_clean_scrub_stamp":"2023-10-06T03:29:33.500929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T07:25:41.452975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2f","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.592223+0000","last_change":"2023-10-06T03:29:34.540499+0000","last_active":"2023-10-06T03:29:38.592223+0000","last_peered":"2023-10-06T03:29:38.592223+0000","last_clean":"2023-10-06T03:29:38.592223+0000","last_became_active":"2023-10-06T03:29:34.539154+0000","last_became_peered":"2023-10-06T03:29:34.539154+0000","last_unstale":"2023-10-06T03:29:38.592223+0000","last_undegraded":"2023-10-06T03:29:38.592223+0000","last_fullsized":"2023-10-06T03:29:38.592223+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_clean_scrub_stamp":"2023-10-06T03:29:33.500929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T15:20:45.655411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.30","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.594589+0000","last_change":"2023-10-06T03:29:34.532489+0000","last_active":"2023-10-06T03:29:38.594589+0000","last_peered":"2023-10-06T03:29:38.594589+0000","last_clean":"2023-10-06T03:29:38.594589+0000","last_became_active":"2023-10-06T03:29:34.532153+0000","last_became_peered":"2023-10-06T03:29:34.532153+0000","last_unstale":"2023-10-06T03:29:38.594589+0000","last_undegraded":"2023-10-06T03:29:38.594589+0000","last_fullsized":"2023-10-06T03:29:38.594589+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_clean_scrub_stamp":"2023-10-06T03:29:33.500929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T08:05:18.552294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.31","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.603291+0000","last_change":"2023-10-06T03:29:34.551157+0000","last_active":"2023-10-06T03:29:38.603291+0000","last_peered":"2023-10-06T03:29:38.603291+0000","last_clean":"2023-10-06T03:29:38.603291+0000","last_became_active":"2023-10-06T03:29:34.550852+0000","last_became_peered":"2023-10-06T03:29:34.550852+0000","last_unstale":"2023-10-06T03:29:38.603291+0000","last_undegraded":"2023-10-06T03:29:38.603291+0000","last_fullsized":"2023-10-06T03:29:38.603291+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_clean_scrub_stamp":"2023-10-06T03:29:33.500929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T08:29:43.429905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.32","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.600883+0000","last_change":"2023-10-06T03:29:34.539103+0000","last_active":"2023-10-06T03:29:38.600883+0000","last_peered":"2023-10-06T03:29:38.600883+0000","last_clean":"2023-10-06T03:29:38.600883+0000","last_became_active":"2023-10-06T03:29:34.538817+0000","last_became_peered":"2023-10-06T03:29:34.538817+0000","last_unstale":"2023-10-06T03:29:38.600883+0000","last_undegraded":"2023-10-06T03:29:38.600883+0000","last_fullsized":"2023-10-06T03:29:38.600883+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_clean_scrub_stamp":"2023-10-06T03:29:33.500929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T07:07:03.705326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.33","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.599036+0000","last_change":"2023-10-06T03:29:34.538671+0000","last_active":"2023-10-06T03:29:38.599036+0000","last_peered":"2023-10-06T03:29:38.599036+0000","last_clean":"2023-10-06T03:29:38.599036+0000","last_became_active":"2023-10-06T03:29:34.538375+0000","last_became_peered":"2023-10-06T03:29:34.538375+0000","last_unstale":"2023-10-06T03:29:38.599036+0000","last_undegraded":"2023-10-06T03:29:38.599036+0000","last_fullsized":"2023-10-06T03:29:38.599036+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_clean_scrub_stamp":"2023-10-06T03:29:33.500929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T07:14:03.653521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.34","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.600214+0000","last_change":"2023-10-06T03:29:34.535139+0000","last_active":"2023-10-06T03:29:38.600214+0000","last_peered":"2023-10-06T03:29:38.600214+0000","last_clean":"2023-10-06T03:29:38.600214+0000","last_became_active":"2023-10-06T03:29:34.534756+0000","last_became_peered":"2023-10-06T03:29:34.534756+0000","last_unstale":"2023-10-06T03:29:38.600214+0000","last_undegraded":"2023-10-06T03:29:38.600214+0000","last_fullsized":"2023-10-06T03:29:38.600214+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_clean_scrub_stamp":"2023-10-06T03:29:33.500929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T14:51:50.895276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.35","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.592690+0000","last_change":"2023-10-06T03:29:34.550626+0000","last_active":"2023-10-06T03:29:38.592690+0000","last_peered":"2023-10-06T03:29:38.592690+0000","last_clean":"2023-10-06T03:29:38.592690+0000","last_became_active":"2023-10-06T03:29:34.550289+0000","last_became_peered":"2023-10-06T03:29:34.550289+0000","last_unstale":"2023-10-06T03:29:38.592690+0000","last_undegraded":"2023-10-06T03:29:38.592690+0000","last_fullsized":"2023-10-06T03:29:38.592690+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_clean_scrub_stamp":"2023-10-06T03:29:33.500929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T12:44:47.758971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.36","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.602276+0000","last_change":"2023-10-06T03:29:34.541344+0000","last_active":"2023-10-06T03:29:38.602276+0000","last_peered":"2023-10-06T03:29:38.602276+0000","last_clean":"2023-10-06T03:29:38.602276+0000","last_became_active":"2023-10-06T03:29:34.540994+0000","last_became_peered":"2023-10-06T03:29:34.540994+0000","last_unstale":"2023-10-06T03:29:38.602276+0000","last_undegraded":"2023-10-06T03:29:38.602276+0000","last_fullsized":"2023-10-06T03:29:38.602276+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_clean_scrub_stamp":"2023-10-06T03:29:33.500929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T05:31:20.131568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.37","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.592106+0000","last_change":"2023-10-06T03:29:34.540530+0000","last_active":"2023-10-06T03:29:38.592106+0000","last_peered":"2023-10-06T03:29:38.592106+0000","last_clean":"2023-10-06T03:29:38.592106+0000","last_became_active":"2023-10-06T03:29:34.539518+0000","last_became_peered":"2023-10-06T03:29:34.539518+0000","last_unstale":"2023-10-06T03:29:38.592106+0000","last_undegraded":"2023-10-06T03:29:38.592106+0000","last_fullsized":"2023-10-06T03:29:38.592106+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_clean_scrub_stamp":"2023-10-06T03:29:33.500929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T14:34:26.405025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.38","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.600763+0000","last_change":"2023-10-06T03:29:34.537333+0000","last_active":"2023-10-06T03:29:38.600763+0000","last_peered":"2023-10-06T03:29:38.600763+0000","last_clean":"2023-10-06T03:29:38.600763+0000","last_became_active":"2023-10-06T03:29:34.536851+0000","last_became_peered":"2023-10-06T03:29:34.536851+0000","last_unstale":"2023-10-06T03:29:38.600763+0000","last_undegraded":"2023-10-06T03:29:38.600763+0000","last_fullsized":"2023-10-06T03:29:38.600763+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_clean_scrub_stamp":"2023-10-06T03:29:33.500929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T14:20:50.168977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.39","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.603186+0000","last_change":"2023-10-06T03:29:34.551594+0000","last_active":"2023-10-06T03:29:38.603186+0000","last_peered":"2023-10-06T03:29:38.603186+0000","last_clean":"2023-10-06T03:29:38.603186+0000","last_became_active":"2023-10-06T03:29:34.551142+0000","last_became_peered":"2023-10-06T03:29:34.551142+0000","last_unstale":"2023-10-06T03:29:38.603186+0000","last_undegraded":"2023-10-06T03:29:38.603186+0000","last_fullsized":"2023-10-06T03:29:38.603186+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_clean_scrub_stamp":"2023-10-06T03:29:33.500929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T07:28:42.661588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3a","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.592149+0000","last_change":"2023-10-06T03:29:34.542011+0000","last_active":"2023-10-06T03:29:38.592149+0000","last_peered":"2023-10-06T03:29:38.592149+0000","last_clean":"2023-10-06T03:29:38.592149+0000","last_became_active":"2023-10-06T03:29:34.540577+0000","last_became_peered":"2023-10-06T03:29:34.540577+0000","last_unstale":"2023-10-06T03:29:38.592149+0000","last_undegraded":"2023-10-06T03:29:38.592149+0000","last_fullsized":"2023-10-06T03:29:38.592149+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_clean_scrub_stamp":"2023-10-06T03:29:33.500929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T12:26:16.082642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.3b","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.593039+0000","last_change":"2023-10-06T03:29:34.538708+0000","last_active":"2023-10-06T03:29:38.593039+0000","last_peered":"2023-10-06T03:29:38.593039+0000","last_clean":"2023-10-06T03:29:38.593039+0000","last_became_active":"2023-10-06T03:29:34.538419+0000","last_became_peered":"2023-10-06T03:29:34.538419+0000","last_unstale":"2023-10-06T03:29:38.593039+0000","last_undegraded":"2023-10-06T03:29:38.593039+0000","last_fullsized":"2023-10-06T03:29:38.593039+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_clean_scrub_stamp":"2023-10-06T03:29:33.500929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T03:45:19.939590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3c","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.609565+0000","last_change":"2023-10-06T03:29:34.546290+0000","last_active":"2023-10-06T03:29:38.609565+0000","last_peered":"2023-10-06T03:29:38.609565+0000","last_clean":"2023-10-06T03:29:38.609565+0000","last_became_active":"2023-10-06T03:29:34.545947+0000","last_became_peered":"2023-10-06T03:29:34.545947+0000","last_unstale":"2023-10-06T03:29:38.609565+0000","last_undegraded":"2023-10-06T03:29:38.609565+0000","last_fullsized":"2023-10-06T03:29:38.609565+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_clean_scrub_stamp":"2023-10-06T03:29:33.500929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T12:25:11.771364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3d","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.592442+0000","last_change":"2023-10-06T03:29:34.541409+0000","last_active":"2023-10-06T03:29:38.592442+0000","last_peered":"2023-10-06T03:29:38.592442+0000","last_clean":"2023-10-06T03:29:38.592442+0000","last_became_active":"2023-10-06T03:29:34.541111+0000","last_became_peered":"2023-10-06T03:29:34.541111+0000","last_unstale":"2023-10-06T03:29:38.592442+0000","last_undegraded":"2023-10-06T03:29:38.592442+0000","last_fullsized":"2023-10-06T03:29:38.592442+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_clean_scrub_stamp":"2023-10-06T03:29:33.500929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T04:15:27.116813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.3e","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.602155+0000","last_change":"2023-10-06T03:29:34.549044+0000","last_active":"2023-10-06T03:29:38.602155+0000","last_peered":"2023-10-06T03:29:38.602155+0000","last_clean":"2023-10-06T03:29:38.602155+0000","last_became_active":"2023-10-06T03:29:34.548691+0000","last_became_peered":"2023-10-06T03:29:34.548691+0000","last_unstale":"2023-10-06T03:29:38.602155+0000","last_undegraded":"2023-10-06T03:29:38.602155+0000","last_fullsized":"2023-10-06T03:29:38.602155+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_clean_scrub_stamp":"2023-10-06T03:29:33.500929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T06:09:15.729486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.3f","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.598391+0000","last_change":"2023-10-06T03:29:34.538647+0000","last_active":"2023-10-06T03:29:38.598391+0000","last_peered":"2023-10-06T03:29:38.598391+0000","last_clean":"2023-10-06T03:29:38.598391+0000","last_became_active":"2023-10-06T03:29:34.538343+0000","last_became_peered":"2023-10-06T03:29:34.538343+0000","last_unstale":"2023-10-06T03:29:38.598391+0000","last_undegraded":"2023-10-06T03:29:38.598391+0000","last_fullsized":"2023-10-06T03:29:38.598391+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_clean_scrub_stamp":"2023-10-06T03:29:33.500929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T12:25:11.771364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":4,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":2306,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":6,"up_from":16,"seq":68719476744,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297572,"kb_used_data":284,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074268,"statfs":{"total":96636764160,"available":96332050432,"internally_reserved":0,"allocated":290816,"data_stored":120026,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77800000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66900000000000004}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.94699999999999995}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0940000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.875}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0600000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.22}]}]},{"osd":5,"up_from":16,"seq":68719476744,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297580,"kb_used_data":292,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074260,"statfs":{"total":96636764160,"available":96332042240,"internally_reserved":0,"allocated":299008,"data_stored":120650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85099999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.087}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97999999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80100000000000005}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0469999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2509999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.208}]}]},{"osd":4,"up_from":16,"seq":68719476743,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297592,"kb_used_data":304,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074248,"statfs":{"total":96636764160,"available":96332029952,"internally_reserved":0,"allocated":311296,"data_stored":121618,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.052}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79400000000000004}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96899999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1399999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2589999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.117}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3859999999999999}]}]},{"osd":7,"up_from":15,"seq":64424509447,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298180,"kb_used_data":892,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94073660,"statfs":{"total":96636764160,"available":96331427840,"internally_reserved":0,"allocated":913408,"data_stored":712146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97399999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86299999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.748}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.94399999999999995}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84299999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80700000000000005}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0880000000000001}]}]},{"osd":0,"up_from":15,"seq":64424509448,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302252,"kb_used_data":868,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94069588,"statfs":{"total":96636764160,"available":96327258112,"internally_reserved":0,"allocated":888832,"data_stored":710948,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97999999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93999999999999995}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76300000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2749999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5409999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.046}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.153}]}]},{"osd":3,"up_from":14,"seq":60129542152,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":301672,"kb_used_data":288,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94070168,"statfs":{"total":96636764160,"available":96327852032,"internally_reserved":0,"allocated":294912,"data_stored":120116,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76500000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81499999999999995}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95999999999999996}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0449999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86599999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2410000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1040000000000001}]}]},{"osd":2,"up_from":14,"seq":60129542151,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":301668,"kb_used_data":284,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94070172,"statfs":{"total":96636764160,"available":96327856128,"internally_reserved":0,"allocated":290816,"data_stored":120026,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78800000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65200000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57199999999999995}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92100000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86099999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0509999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.99299999999999999}]}]},{"osd":1,"up_from":14,"seq":60129542151,"num_pgs":51,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":301668,"kb_used_data":284,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94070172,"statfs":{"total":96636764160,"available":96327856128,"internally_reserved":0,"allocated":290816,"data_stored":120026,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60599999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67200000000000004}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72899999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.875}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78500000000000003}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85199999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.05}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":554,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-06T03:29:48.659 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2023-10-06T03:29:48.659 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2023-10-06T03:29:48.659 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2023-10-06T03:29:48.660 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2023-10-06T03:29:48.660 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2023-10-06T03:29:48.771 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:48.765+0000 7f208359e700 1 Processor -- start 2023-10-06T03:29:48.771 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:48.765+0000 7f208359e700 1 -- start start 2023-10-06T03:29:48.772 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:48.769+0000 7f208359e700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f20841569e0 0x7f20840c6b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:48.772 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:48.769+0000 7f208359e700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2084156db0 0x7f20840c7090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:48.773 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:48.769+0000 7f208359e700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20840abfe0 con 0x7f208406beb0 2023-10-06T03:29:48.773 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:48.769+0000 7f208359e700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f208406ca20 con 0x7f20841569e0 2023-10-06T03:29:48.773 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:48.769+0000 7f208359e700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20840ab9f0 con 0x7f2084156db0 2023-10-06T03:29:48.773 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:48.769+0000 7f208259c700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f20841569e0 0x7f20840c6b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:48.773 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:48.769+0000 7f208259c700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f20841569e0 0x7f20840c6b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:60492/0 (socket says 172.21.15.110:60492) 2023-10-06T03:29:48.773 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:48.769+0000 7f208259c700 1 -- 172.21.15.110:0/3234045500 learned_addr learned my addr 172.21.15.110:0/3234045500 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:29:48.773 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:48.769+0000 7f207bfff700 1 --2- 172.21.15.110:0/3234045500 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2084156db0 0x7f20840c7090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:48.775 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:48.769+0000 7f208259c700 1 -- 172.21.15.110:0/3234045500 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2084156db0 msgr2=0x7f20840c7090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:29:48.775 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:48.769+0000 7f208259c700 1 --2- 172.21.15.110:0/3234045500 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2084156db0 0x7f20840c7090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:48.775 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:48.769+0000 7f208259c700 1 -- 172.21.15.110:0/3234045500 >> v1:172.21.15.192:6789/0 conn(0x7f208406beb0 legacy=0x7f208406c310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:48.775 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:48.769+0000 7f208259c700 1 -- 172.21.15.110:0/3234045500 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f208406cc40 con 0x7f20841569e0 2023-10-06T03:29:48.775 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:48.769+0000 7f208259c700 1 --2- 172.21.15.110:0/3234045500 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f20841569e0 0x7f20840c6b50 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f20740099b0 tx=0x7f2074009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=49f6cae9ab2d7676 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:29:48.775 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:48.769+0000 7f2081d9b700 1 -- 172.21.15.110:0/3234045500 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2074009ea0 con 0x7f20841569e0 2023-10-06T03:29:48.775 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:48.769+0000 7f2081d9b700 1 -- 172.21.15.110:0/3234045500 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f207400b860 con 0x7f20841569e0 2023-10-06T03:29:48.775 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:48.769+0000 7f2081d9b700 1 -- 172.21.15.110:0/3234045500 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f207400bc00 con 0x7f20841569e0 2023-10-06T03:29:48.775 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:48.769+0000 7f208359e700 1 -- 172.21.15.110:0/3234045500 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f20841569e0 msgr2=0x7f20840c6b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:48.776 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:48.769+0000 7f208359e700 1 --2- 172.21.15.110:0/3234045500 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f20841569e0 0x7f20840c6b50 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f20740099b0 tx=0x7f2074009250 comp rx=0 tx=0).stop 2023-10-06T03:29:48.776 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:48.769+0000 7f208359e700 1 -- 172.21.15.110:0/3234045500 shutdown_connections 2023-10-06T03:29:48.776 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:48.769+0000 7f208359e700 1 --2- 172.21.15.110:0/3234045500 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2084156db0 0x7f20840c7090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:48.776 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:48.769+0000 7f208359e700 1 --2- 172.21.15.110:0/3234045500 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f20841569e0 0x7f20840c6b50 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:48.776 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:48.769+0000 7f208359e700 1 -- 172.21.15.110:0/3234045500 >> 172.21.15.110:0/3234045500 conn(0x7f20840cbe30 msgr2=0x7f20840cc230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:29:48.776 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:48.769+0000 7f208359e700 1 -- 172.21.15.110:0/3234045500 shutdown_connections 2023-10-06T03:29:48.776 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:48.769+0000 7f208359e700 1 -- 172.21.15.110:0/3234045500 wait complete. 2023-10-06T03:29:48.777 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:48.773+0000 7f208359e700 1 Processor -- start 2023-10-06T03:29:48.777 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:48.773+0000 7f208359e700 1 -- start start 2023-10-06T03:29:48.778 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:48.773+0000 7f208359e700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f208406beb0 0x7f20840c5d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:48.778 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:48.773+0000 7f208359e700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f20841569e0 0x7f20840b0340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:48.778 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:48.773+0000 7f208359e700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2084156db0 0x7f20840b0880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:48.779 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:48.773+0000 7f208359e700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2084154c20 con 0x7f2084156db0 2023-10-06T03:29:48.779 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:48.773+0000 7f208359e700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2084154660 con 0x7f208406beb0 2023-10-06T03:29:48.779 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:48.773+0000 7f208359e700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2084154940 con 0x7f20841569e0 2023-10-06T03:29:48.779 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:48.773+0000 7f2082d9d700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2084156db0 0x7f20840b0880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:48.779 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:48.773+0000 7f2082d9d700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2084156db0 0x7f20840b0880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:60500/0 (socket says 172.21.15.110:60500) 2023-10-06T03:29:48.779 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:48.773+0000 7f2082d9d700 1 -- 172.21.15.110:0/3622412999 learned_addr learned my addr 172.21.15.110:0/3622412999 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:29:48.779 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:48.773+0000 7f208259c700 1 --2- 172.21.15.110:0/3622412999 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f208406beb0 0x7f20840c5d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:48.779 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:48.773+0000 7f207bfff700 1 --2- 172.21.15.110:0/3622412999 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f20841569e0 0x7f20840b0340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:48.779 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:48.773+0000 7f2082d9d700 1 -- 172.21.15.110:0/3622412999 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f20841569e0 msgr2=0x7f20840b0340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:48.779 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:48.773+0000 7f2082d9d700 1 --2- 172.21.15.110:0/3622412999 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f20841569e0 0x7f20840b0340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:48.779 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:48.773+0000 7f2082d9d700 1 -- 172.21.15.110:0/3622412999 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f208406beb0 msgr2=0x7f20840c5d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:48.780 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:48.773+0000 7f2082d9d700 1 --2- 172.21.15.110:0/3622412999 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f208406beb0 0x7f20840c5d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:48.780 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:48.773+0000 7f2082d9d700 1 -- 172.21.15.110:0/3622412999 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20740094f0 con 0x7f2084156db0 2023-10-06T03:29:48.780 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:48.773+0000 7f2082d9d700 1 --2- 172.21.15.110:0/3622412999 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2084156db0 0x7f20840b0880 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f207c00c910 tx=0x7f207c0016d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:29:48.781 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:48.773+0000 7f207a7fc700 1 -- 172.21.15.110:0/3622412999 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f207c012630 con 0x7f2084156db0 2023-10-06T03:29:48.781 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:48.773+0000 7f207a7fc700 1 -- 172.21.15.110:0/3622412999 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f207c0127b0 con 0x7f2084156db0 2023-10-06T03:29:48.781 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:48.773+0000 7f208359e700 1 -- 172.21.15.110:0/3622412999 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f208414c530 con 0x7f2084156db0 2023-10-06T03:29:48.781 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:48.777+0000 7f207a7fc700 1 -- 172.21.15.110:0/3622412999 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f207c012b70 con 0x7f2084156db0 2023-10-06T03:29:48.781 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:48.777+0000 7f208359e700 1 -- 172.21.15.110:0/3622412999 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f208416cd20 con 0x7f2084156db0 2023-10-06T03:29:48.781 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:48.777+0000 7f208359e700 1 -- 172.21.15.110:0/3622412999 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2084070ff0 con 0x7f2084156db0 2023-10-06T03:29:48.781 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:48.777+0000 7f207a7fc700 1 -- 172.21.15.110:0/3622412999 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7f207c003030 con 0x7f2084156db0 2023-10-06T03:29:48.782 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:48.777+0000 7f207a7fc700 1 --2- 172.21.15.110:0/3622412999 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f2068060550 0x7f2068062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:48.783 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:48.777+0000 7f207a7fc700 1 -- 172.21.15.110:0/3622412999 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6809+0+0 (secure 0 0 0) 0x7f207c07cb20 con 0x7f2084156db0 2023-10-06T03:29:48.783 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:48.777+0000 7f208259c700 1 --2- 172.21.15.110:0/3622412999 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f2068060550 0x7f2068062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:48.783 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:48.781+0000 7f208259c700 1 --2- 172.21.15.110:0/3622412999 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f2068060550 0x7f2068062a00 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f20740099b0 tx=0x7f2074017000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:29:48.789 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:48.785+0000 7f207a7fc700 1 -- 172.21.15.110:0/3622412999 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f207c04ccc0 con 0x7f2084156db0 2023-10-06T03:29:48.987 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:48.985+0000 7f208359e700 1 -- 172.21.15.110:0/3622412999 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7f208406cf20 con 0x7f2084156db0 2023-10-06T03:29:48.994 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:48.989+0000 7f207a7fc700 1 -- 172.21.15.110:0/3622412999 <== mon.0 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v5) v1 ==== 74+0+154137 (secure 0 0 0) 0x7f207c0502e0 con 0x7f2084156db0 2023-10-06T03:29:48.997 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-10-06T03:29:48.999 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:48.993+0000 7f208359e700 1 -- 172.21.15.110:0/3622412999 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f2068060550 msgr2=0x7f2068062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:48.999 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:48.993+0000 7f208359e700 1 --2- 172.21.15.110:0/3622412999 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f2068060550 0x7f2068062a00 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f20740099b0 tx=0x7f2074017000 comp rx=0 tx=0).stop 2023-10-06T03:29:49.000 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:48.993+0000 7f208359e700 1 -- 172.21.15.110:0/3622412999 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2084156db0 msgr2=0x7f20840b0880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:49.000 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:48.993+0000 7f208359e700 1 --2- 172.21.15.110:0/3622412999 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2084156db0 0x7f20840b0880 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f207c00c910 tx=0x7f207c0016d0 comp rx=0 tx=0).stop 2023-10-06T03:29:49.000 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:48.993+0000 7f208359e700 1 -- 172.21.15.110:0/3622412999 shutdown_connections 2023-10-06T03:29:49.000 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:48.993+0000 7f208359e700 1 --2- 172.21.15.110:0/3622412999 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f2068060550 0x7f2068062a00 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:49.000 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:48.993+0000 7f208359e700 1 --2- 172.21.15.110:0/3622412999 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2084156db0 0x7f20840b0880 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:49.000 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:48.993+0000 7f208359e700 1 --2- 172.21.15.110:0/3622412999 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f20841569e0 0x7f20840b0340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:49.000 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:48.993+0000 7f208359e700 1 --2- 172.21.15.110:0/3622412999 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f208406beb0 0x7f20840c5d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:49.001 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:48.993+0000 7f208359e700 1 -- 172.21.15.110:0/3622412999 >> 172.21.15.110:0/3622412999 conn(0x7f20840cbe30 msgr2=0x7f208414b880 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:29:49.001 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:48.993+0000 7f208359e700 1 -- 172.21.15.110:0/3622412999 shutdown_connections 2023-10-06T03:29:49.001 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:48.993+0000 7f208359e700 1 -- 172.21.15.110:0/3622412999 wait complete. 2023-10-06T03:29:49.012 INFO:teuthology.orchestra.run.smithi110.stdout:{"epoch":5,"active_gid":4110,"active_name":"y","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":33268},{"type":"v1","addr":"172.21.15.110:6833","nonce":33268}]},"active_addr":"172.21.15.110:6833/33268","active_change":"2023-10-06T03:29:14.631818+0000","active_mgr_features":4540138320759226367,"available":true,"standbys":[{"gid":4099,"name":"x","mgr_features":4540138320759226367,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":false,"error_string":"loading asyncssh library:No module named 'asyncssh'","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.1.5","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":false,"error_string":"loading asyncssh library:No module named 'asyncssh'","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.1.5","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.110:0","nonce":1857995858}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.110:0","nonce":1318624357}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.110:0","nonce":463196587}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.110:0","nonce":1139019390}]}]} 2023-10-06T03:29:49.019 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2023-10-06T03:29:49.019 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2023-10-06T03:29:49.020 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-06T03:29:49.127 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.125+0000 7f3e9724e700 1 Processor -- start 2023-10-06T03:29:49.128 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.125+0000 7f3e9724e700 1 -- start start 2023-10-06T03:29:49.129 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.125+0000 7f3e9724e700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f3e900add50 0x7f3e900af4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:49.129 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.125+0000 7f3e9724e700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e90155780 con 0x7f3e900b04d0 2023-10-06T03:29:49.130 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.125+0000 7f3e9724e700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e900b0950 con 0x7f3e900ae120 2023-10-06T03:29:49.130 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.125+0000 7f3e9724e700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3e900cb4a0 con 0x7f3e900add50 2023-10-06T03:29:49.130 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.125+0000 7f3e9624c700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f3e900add50 0x7f3e900af4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:49.130 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.125+0000 7f3e9624c700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f3e900add50 0x7f3e900af4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:60506/0 (socket says 172.21.15.110:60506) 2023-10-06T03:29:49.130 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.125+0000 7f3e9624c700 1 -- 172.21.15.110:0/437098570 learned_addr learned my addr 172.21.15.110:0/437098570 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:29:49.130 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.125+0000 7f3e96a4d700 1 --1- 172.21.15.110:0/437098570 >> v1:172.21.15.110:6789/0 conn(0x7f3e900b04d0 0x7f3e900ca660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:37272/0 (socket says 172.21.15.110:37272) 2023-10-06T03:29:49.130 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.125+0000 7f3e9624c700 1 -- 172.21.15.110:0/437098570 >> v1:172.21.15.192:6789/0 conn(0x7f3e900ae120 legacy=0x7f3e900c5f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:49.130 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.125+0000 7f3e9624c700 1 -- 172.21.15.110:0/437098570 >> v1:172.21.15.110:6789/0 conn(0x7f3e900b04d0 legacy=0x7f3e900ca660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:49.130 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.125+0000 7f3e9624c700 1 -- 172.21.15.110:0/437098570 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e900b0c10 con 0x7f3e900add50 2023-10-06T03:29:49.130 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.125+0000 7f3e9624c700 1 --2- 172.21.15.110:0/437098570 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3e900add50 0x7f3e900af4f0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f3e8c00d1a0 tx=0x7f3e8c00ba50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=54b16bbec56d3faf server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:29:49.131 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.125+0000 7f3e9524a700 1 -- 172.21.15.110:0/437098570 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e8c010070 con 0x7f3e900add50 2023-10-06T03:29:49.131 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.125+0000 7f3e9524a700 1 -- 172.21.15.110:0/437098570 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e8c00b0d0 con 0x7f3e900add50 2023-10-06T03:29:49.131 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.125+0000 7f3e9524a700 1 -- 172.21.15.110:0/437098570 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e8c00b490 con 0x7f3e900add50 2023-10-06T03:29:49.131 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.125+0000 7f3e9724e700 1 -- 172.21.15.110:0/437098570 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3e900add50 msgr2=0x7f3e900af4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:49.132 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.125+0000 7f3e9724e700 1 --2- 172.21.15.110:0/437098570 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3e900add50 0x7f3e900af4f0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f3e8c00d1a0 tx=0x7f3e8c00ba50 comp rx=0 tx=0).stop 2023-10-06T03:29:49.132 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.125+0000 7f3e9724e700 1 -- 172.21.15.110:0/437098570 shutdown_connections 2023-10-06T03:29:49.132 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.125+0000 7f3e9724e700 1 --2- 172.21.15.110:0/437098570 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3e900add50 0x7f3e900af4f0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:49.132 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.125+0000 7f3e9724e700 1 -- 172.21.15.110:0/437098570 >> 172.21.15.110:0/437098570 conn(0x7f3e900cc470 msgr2=0x7f3e900c8410 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:29:49.132 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.125+0000 7f3e9724e700 1 -- 172.21.15.110:0/437098570 shutdown_connections 2023-10-06T03:29:49.132 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.125+0000 7f3e9724e700 1 -- 172.21.15.110:0/437098570 wait complete. 2023-10-06T03:29:49.133 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.129+0000 7f3e9724e700 1 Processor -- start 2023-10-06T03:29:49.134 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.129+0000 7f3e9724e700 1 -- start start 2023-10-06T03:29:49.135 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.129+0000 7f3e9724e700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3e900add50 0x7f3e90158480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:49.136 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.129+0000 7f3e9724e700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3e900ae120 0x7f3e900c2d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:49.136 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.129+0000 7f3e9724e700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3e900b04d0 0x7f3e90168820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:49.136 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.129+0000 7f3e9724e700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e900c4100 con 0x7f3e900b04d0 2023-10-06T03:29:49.136 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.129+0000 7f3e9724e700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e900c3270 con 0x7f3e900ae120 2023-10-06T03:29:49.136 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.129+0000 7f3e9724e700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e900c34f0 con 0x7f3e900add50 2023-10-06T03:29:49.136 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.133+0000 7f3e96a4d700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3e900b04d0 0x7f3e90168820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:49.136 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.133+0000 7f3e96a4d700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3e900b04d0 0x7f3e90168820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:60510/0 (socket says 172.21.15.110:60510) 2023-10-06T03:29:49.136 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.133+0000 7f3e96a4d700 1 -- 172.21.15.110:0/2801666537 learned_addr learned my addr 172.21.15.110:0/2801666537 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:29:49.136 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.133+0000 7f3e9624c700 1 --2- 172.21.15.110:0/2801666537 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3e900add50 0x7f3e90158480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:49.136 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.133+0000 7f3e95a4b700 1 --2- 172.21.15.110:0/2801666537 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3e900ae120 0x7f3e900c2d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:49.137 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.133+0000 7f3e96a4d700 1 -- 172.21.15.110:0/2801666537 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3e900add50 msgr2=0x7f3e90158480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:49.137 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.133+0000 7f3e96a4d700 1 --2- 172.21.15.110:0/2801666537 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3e900add50 0x7f3e90158480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:49.137 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.133+0000 7f3e96a4d700 1 -- 172.21.15.110:0/2801666537 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3e900ae120 msgr2=0x7f3e900c2d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:49.137 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.133+0000 7f3e96a4d700 1 --2- 172.21.15.110:0/2801666537 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3e900ae120 0x7f3e900c2d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:49.137 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.133+0000 7f3e96a4d700 1 -- 172.21.15.110:0/2801666537 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e8c00bcf0 con 0x7f3e900b04d0 2023-10-06T03:29:49.137 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.133+0000 7f3e9624c700 1 --2- 172.21.15.110:0/2801666537 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3e900add50 0x7f3e90158480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-06T03:29:49.137 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.133+0000 7f3e96a4d700 1 --2- 172.21.15.110:0/2801666537 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3e900b04d0 0x7f3e90168820 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f3e8800c4e0 tx=0x7f3e88007f50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:29:49.139 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.133+0000 7f3e867fc700 1 -- 172.21.15.110:0/2801666537 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e88005b40 con 0x7f3e900b04d0 2023-10-06T03:29:49.139 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.133+0000 7f3e867fc700 1 -- 172.21.15.110:0/2801666537 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e88005ca0 con 0x7f3e900b04d0 2023-10-06T03:29:49.139 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.133+0000 7f3e9724e700 1 -- 172.21.15.110:0/2801666537 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e900c37d0 con 0x7f3e900b04d0 2023-10-06T03:29:49.139 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.133+0000 7f3e867fc700 1 -- 172.21.15.110:0/2801666537 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e88007300 con 0x7f3e900b04d0 2023-10-06T03:29:49.139 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.133+0000 7f3e9724e700 1 -- 172.21.15.110:0/2801666537 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e900c3d90 con 0x7f3e900b04d0 2023-10-06T03:29:49.140 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.133+0000 7f3e867fc700 1 -- 172.21.15.110:0/2801666537 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7f3e8801b3b0 con 0x7f3e900b04d0 2023-10-06T03:29:49.140 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.133+0000 7f3e9724e700 1 -- 172.21.15.110:0/2801666537 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e900c42b0 con 0x7f3e900b04d0 2023-10-06T03:29:49.140 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.133+0000 7f3e867fc700 1 --2- 172.21.15.110:0/2801666537 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f3e74060580 0x7f3e74062a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:49.140 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.133+0000 7f3e9624c700 1 --2- 172.21.15.110:0/2801666537 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f3e74060580 0x7f3e74062a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:49.140 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.133+0000 7f3e867fc700 1 -- 172.21.15.110:0/2801666537 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6809+0+0 (secure 0 0 0) 0x7f3e8807c230 con 0x7f3e900b04d0 2023-10-06T03:29:49.140 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.137+0000 7f3e9624c700 1 --2- 172.21.15.110:0/2801666537 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f3e74060580 0x7f3e74062a30 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f3e8c00ba90 tx=0x7f3e8c01a000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:29:49.143 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.137+0000 7f3e867fc700 1 -- 172.21.15.110:0/2801666537 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f3e900c42b0 con 0x7f3e900b04d0 2023-10-06T03:29:49.311 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.305+0000 7f3e9724e700 1 -- 172.21.15.110:0/2801666537 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3e90070ff0 con 0x7f3e900b04d0 2023-10-06T03:29:49.313 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.309+0000 7f3e867fc700 1 -- 172.21.15.110:0/2801666537 <== mon.0 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v28) v1 ==== 74+0+18755 (secure 0 0 0) 0x7f3e8804c330 con 0x7f3e900b04d0 2023-10-06T03:29:49.313 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-10-06T03:29:49.314 INFO:teuthology.orchestra.run.smithi110.stdout:{"epoch":28,"fsid":"93b8f6ac-4c92-4b45-b8e4-b59617024aa2","created":"2023-10-06T03:29:10.519583+0000","modified":"2023-10-06T03:29:38.951740+0000","last_up_change":"2023-10-06T03:29:21.960511+0000","last_in_change":"2023-10-06T03:29:12.560436+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-06T03:29:20.703743+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2023-10-06T03:29:28.307281+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2023-10-06T03:29:32.247251+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data","create_time":"2023-10-06T03:29:32.880086+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}}},{"pool":5,"pool_name":"cephfs_data_ec","create_time":"2023-10-06T03:29:36.613779+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"28","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"5866eb6c-2adc-46e4-be7a-e5bcda0f4065","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6824","nonce":33394},{"type":"v1","addr":"172.21.15.110:6825","nonce":33394}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6826","nonce":33394},{"type":"v1","addr":"172.21.15.110:6827","nonce":33394}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6830","nonce":33394},{"type":"v1","addr":"172.21.15.110:6831","nonce":33394}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6828","nonce":33394},{"type":"v1","addr":"172.21.15.110:6829","nonce":33394}]},"public_addr":"172.21.15.110:6825/33394","cluster_addr":"172.21.15.110:6827/33394","heartbeat_back_addr":"172.21.15.110:6831/33394","heartbeat_front_addr":"172.21.15.110:6829/33394","state":["exists","up"]},{"osd":1,"uuid":"a38de118-062b-4a05-9782-2ceddf6ff4df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6800","nonce":33396},{"type":"v1","addr":"172.21.15.110:6801","nonce":33396}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6802","nonce":33396},{"type":"v1","addr":"172.21.15.110:6803","nonce":33396}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6806","nonce":33396},{"type":"v1","addr":"172.21.15.110:6807","nonce":33396}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6804","nonce":33396},{"type":"v1","addr":"172.21.15.110:6805","nonce":33396}]},"public_addr":"172.21.15.110:6801/33396","cluster_addr":"172.21.15.110:6803/33396","heartbeat_back_addr":"172.21.15.110:6807/33396","heartbeat_front_addr":"172.21.15.110:6805/33396","state":["exists","up"]},{"osd":2,"uuid":"56e39be2-3407-4b69-877e-cf16617b06d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6816","nonce":33397},{"type":"v1","addr":"172.21.15.110:6817","nonce":33397}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6818","nonce":33397},{"type":"v1","addr":"172.21.15.110:6819","nonce":33397}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6822","nonce":33397},{"type":"v1","addr":"172.21.15.110:6823","nonce":33397}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6820","nonce":33397},{"type":"v1","addr":"172.21.15.110:6821","nonce":33397}]},"public_addr":"172.21.15.110:6817/33397","cluster_addr":"172.21.15.110:6819/33397","heartbeat_back_addr":"172.21.15.110:6823/33397","heartbeat_front_addr":"172.21.15.110:6821/33397","state":["exists","up"]},{"osd":3,"uuid":"01c82d13-c1d4-4bde-b55e-3e0089b40115","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6808","nonce":33395},{"type":"v1","addr":"172.21.15.110:6809","nonce":33395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6810","nonce":33395},{"type":"v1","addr":"172.21.15.110:6811","nonce":33395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6814","nonce":33395},{"type":"v1","addr":"172.21.15.110:6815","nonce":33395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6812","nonce":33395},{"type":"v1","addr":"172.21.15.110:6813","nonce":33395}]},"public_addr":"172.21.15.110:6809/33395","cluster_addr":"172.21.15.110:6811/33395","heartbeat_back_addr":"172.21.15.110:6815/33395","heartbeat_front_addr":"172.21.15.110:6813/33395","state":["exists","up"]},{"osd":4,"uuid":"86983430-5fa3-4d17-998f-03e2cd843237","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":33683},{"type":"v1","addr":"172.21.15.192:6817","nonce":33683}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":33683},{"type":"v1","addr":"172.21.15.192:6819","nonce":33683}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":33683},{"type":"v1","addr":"172.21.15.192:6823","nonce":33683}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6820","nonce":33683},{"type":"v1","addr":"172.21.15.192:6821","nonce":33683}]},"public_addr":"172.21.15.192:6817/33683","cluster_addr":"172.21.15.192:6819/33683","heartbeat_back_addr":"172.21.15.192:6823/33683","heartbeat_front_addr":"172.21.15.192:6821/33683","state":["exists","up"]},{"osd":5,"uuid":"6945257f-821d-4619-a63f-ebe857eba6f4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":33684},{"type":"v1","addr":"172.21.15.192:6801","nonce":33684}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":33684},{"type":"v1","addr":"172.21.15.192:6803","nonce":33684}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":33684},{"type":"v1","addr":"172.21.15.192:6807","nonce":33684}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6804","nonce":33684},{"type":"v1","addr":"172.21.15.192:6805","nonce":33684}]},"public_addr":"172.21.15.192:6801/33684","cluster_addr":"172.21.15.192:6803/33684","heartbeat_back_addr":"172.21.15.192:6807/33684","heartbeat_front_addr":"172.21.15.192:6805/33684","state":["exists","up"]},{"osd":6,"uuid":"7854d1ae-7060-4780-8f97-73cea5e7a12f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6824","nonce":33686},{"type":"v1","addr":"172.21.15.192:6825","nonce":33686}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":33686},{"type":"v1","addr":"172.21.15.192:6827","nonce":33686}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":33686},{"type":"v1","addr":"172.21.15.192:6831","nonce":33686}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6828","nonce":33686},{"type":"v1","addr":"172.21.15.192:6829","nonce":33686}]},"public_addr":"172.21.15.192:6825/33686","cluster_addr":"172.21.15.192:6827/33686","heartbeat_back_addr":"172.21.15.192:6831/33686","heartbeat_front_addr":"172.21.15.192:6829/33686","state":["exists","up"]},{"osd":7,"uuid":"f1ebdfd5-b859-482b-9f97-c18c4a26f445","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6808","nonce":33685},{"type":"v1","addr":"172.21.15.192:6809","nonce":33685}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":33685},{"type":"v1","addr":"172.21.15.192:6811","nonce":33685}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":33685},{"type":"v1","addr":"172.21.15.192:6815","nonce":33685}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6812","nonce":33685},{"type":"v1","addr":"172.21.15.192:6813","nonce":33685}]},"public_addr":"172.21.15.192:6809/33685","cluster_addr":"172.21.15.192:6811/33685","heartbeat_back_addr":"172.21.15.192:6815/33685","heartbeat_front_addr":"172.21.15.192:6813/33685","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T03:29:18.002048+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T03:29:17.478576+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T03:29:18.034944+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T03:29:17.628513+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T03:29:18.047338+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T03:29:17.772349+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T03:29:17.883958+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T03:29:18.151838+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-06T03:29:49.316 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.313+0000 7f3e9724e700 1 -- 172.21.15.110:0/2801666537 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f3e74060580 msgr2=0x7f3e74062a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:49.316 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.313+0000 7f3e9724e700 1 --2- 172.21.15.110:0/2801666537 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f3e74060580 0x7f3e74062a30 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f3e8c00ba90 tx=0x7f3e8c01a000 comp rx=0 tx=0).stop 2023-10-06T03:29:49.316 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.313+0000 7f3e9724e700 1 -- 172.21.15.110:0/2801666537 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3e900b04d0 msgr2=0x7f3e90168820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:49.316 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.313+0000 7f3e9724e700 1 --2- 172.21.15.110:0/2801666537 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3e900b04d0 0x7f3e90168820 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f3e8800c4e0 tx=0x7f3e88007f50 comp rx=0 tx=0).stop 2023-10-06T03:29:49.316 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.313+0000 7f3e9724e700 1 -- 172.21.15.110:0/2801666537 shutdown_connections 2023-10-06T03:29:49.316 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.313+0000 7f3e9724e700 1 --2- 172.21.15.110:0/2801666537 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f3e74060580 0x7f3e74062a30 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:49.316 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.313+0000 7f3e9724e700 1 --2- 172.21.15.110:0/2801666537 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3e900b04d0 0x7f3e90168820 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:49.316 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.313+0000 7f3e9724e700 1 --2- 172.21.15.110:0/2801666537 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3e900ae120 0x7f3e900c2d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:49.316 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.313+0000 7f3e9724e700 1 --2- 172.21.15.110:0/2801666537 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3e900add50 0x7f3e90158480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:49.317 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.313+0000 7f3e9724e700 1 -- 172.21.15.110:0/2801666537 >> 172.21.15.110:0/2801666537 conn(0x7f3e900cc470 msgr2=0x7f3e900c66b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:29:49.317 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.313+0000 7f3e9724e700 1 -- 172.21.15.110:0/2801666537 shutdown_connections 2023-10-06T03:29:49.317 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.313+0000 7f3e9724e700 1 -- 172.21.15.110:0/2801666537 wait complete. 2023-10-06T03:29:49.328 INFO:tasks.ceph.ceph_manager.ceph:all up! 2023-10-06T03:29:49.328 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-06T03:29:49.427 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.421+0000 7f5fdacf4700 1 Processor -- start 2023-10-06T03:29:49.427 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.421+0000 7f5fdacf4700 1 -- start start 2023-10-06T03:29:49.427 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.425+0000 7f5fdacf4700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f5fd4069d70 0x7f5fd40ca270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:49.427 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.425+0000 7f5fdacf4700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5fd415d7f0 0x7f5fd40ca7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:49.428 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.425+0000 7f5fdacf4700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5fd406aef0 con 0x7f5fd415d420 2023-10-06T03:29:49.428 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.425+0000 7f5fdacf4700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5fd40c82a0 con 0x7f5fd4069d70 2023-10-06T03:29:49.428 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.425+0000 7f5fdacf4700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5fd406c7d0 con 0x7f5fd415d7f0 2023-10-06T03:29:49.429 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.425+0000 7f5fd3fff700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f5fd4069d70 0x7f5fd40ca270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:49.429 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.425+0000 7f5fd9291700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5fd415d7f0 0x7f5fd40ca7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:49.429 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.425+0000 7f5fd3fff700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f5fd4069d70 0x7f5fd40ca270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.110:50682/0 (socket says 172.21.15.110:50682) 2023-10-06T03:29:49.429 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.425+0000 7f5fd3fff700 1 -- 172.21.15.110:0/144261985 learned_addr learned my addr 172.21.15.110:0/144261985 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:29:49.429 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.425+0000 7f5fd8a90700 1 --1- >> v1:172.21.15.192:6789/0 conn(0x7f5fd415d420 0x7f5fd40c7b90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.110:52068/0 (socket says 172.21.15.110:52068) 2023-10-06T03:29:49.429 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.425+0000 7f5fd3fff700 1 -- 172.21.15.110:0/144261985 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5fd415d7f0 msgr2=0x7f5fd40ca7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:29:49.429 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.425+0000 7f5fd3fff700 1 --2- 172.21.15.110:0/144261985 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5fd415d7f0 0x7f5fd40ca7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:49.430 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.425+0000 7f5fd3fff700 1 -- 172.21.15.110:0/144261985 >> v1:172.21.15.192:6789/0 conn(0x7f5fd415d420 legacy=0x7f5fd40c7b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:49.430 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.425+0000 7f5fd3fff700 1 -- 172.21.15.110:0/144261985 --> v2:172.21.15.192:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fd406a2e0 con 0x7f5fd4069d70 2023-10-06T03:29:49.430 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.425+0000 7f5fd3fff700 1 --2- 172.21.15.110:0/144261985 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5fd4069d70 0x7f5fd40ca270 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f5fbc0099b0 tx=0x7f5fbc009250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=47622512db587eaa server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:29:49.430 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.425+0000 7f5fd37fe700 1 -- 172.21.15.110:0/144261985 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fbc009ea0 con 0x7f5fd4069d70 2023-10-06T03:29:49.430 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.425+0000 7f5fd37fe700 1 -- 172.21.15.110:0/144261985 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5fbc00b860 con 0x7f5fd4069d70 2023-10-06T03:29:49.430 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.425+0000 7f5fd37fe700 1 -- 172.21.15.110:0/144261985 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fbc00bc00 con 0x7f5fd4069d70 2023-10-06T03:29:49.431 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.425+0000 7f5fdacf4700 1 -- 172.21.15.110:0/144261985 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5fd4069d70 msgr2=0x7f5fd40ca270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:49.431 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.425+0000 7f5fdacf4700 1 --2- 172.21.15.110:0/144261985 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5fd4069d70 0x7f5fd40ca270 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f5fbc0099b0 tx=0x7f5fbc009250 comp rx=0 tx=0).stop 2023-10-06T03:29:49.431 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.425+0000 7f5fdacf4700 1 -- 172.21.15.110:0/144261985 shutdown_connections 2023-10-06T03:29:49.431 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.425+0000 7f5fdacf4700 1 --2- 172.21.15.110:0/144261985 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5fd415d7f0 0x7f5fd40ca7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:49.431 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.425+0000 7f5fdacf4700 1 --2- 172.21.15.110:0/144261985 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5fd4069d70 0x7f5fd40ca270 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:49.431 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.425+0000 7f5fdacf4700 1 -- 172.21.15.110:0/144261985 >> 172.21.15.110:0/144261985 conn(0x7f5fd40cbd00 msgr2=0x7f5fd40cc100 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:29:49.431 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.425+0000 7f5fdacf4700 1 -- 172.21.15.110:0/144261985 shutdown_connections 2023-10-06T03:29:49.432 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.425+0000 7f5fdacf4700 1 -- 172.21.15.110:0/144261985 wait complete. 2023-10-06T03:29:49.433 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.429+0000 7f5fdacf4700 1 Processor -- start 2023-10-06T03:29:49.434 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.429+0000 7f5fdacf4700 1 -- start start 2023-10-06T03:29:49.437 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.433+0000 7f5fdacf4700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5fd4069d70 0x7f5fd4150c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:49.438 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.433+0000 7f5fdacf4700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5fd415d420 0x7f5fd414fe10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:49.438 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.433+0000 7f5fdacf4700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5fd415d7f0 0x7f5fd4150350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:49.438 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.433+0000 7f5fdacf4700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fd4152a90 con 0x7f5fd415d7f0 2023-10-06T03:29:49.438 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.433+0000 7f5fdacf4700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fd4152d70 con 0x7f5fd4069d70 2023-10-06T03:29:49.438 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.433+0000 7f5fdacf4700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5fd4153050 con 0x7f5fd415d420 2023-10-06T03:29:49.439 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.433+0000 7f5fd9291700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5fd415d7f0 0x7f5fd4150350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:49.439 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.433+0000 7f5fd9291700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5fd415d7f0 0x7f5fd4150350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:60522/0 (socket says 172.21.15.110:60522) 2023-10-06T03:29:49.439 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.433+0000 7f5fd9291700 1 -- 172.21.15.110:0/3589777351 learned_addr learned my addr 172.21.15.110:0/3589777351 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:29:49.439 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.433+0000 7f5fd8a90700 1 --2- 172.21.15.110:0/3589777351 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5fd4069d70 0x7f5fd4150c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:49.439 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.433+0000 7f5fd3fff700 1 --2- 172.21.15.110:0/3589777351 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5fd415d420 0x7f5fd414fe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:49.440 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.433+0000 7f5fd9291700 1 -- 172.21.15.110:0/3589777351 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5fd415d420 msgr2=0x7f5fd414fe10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:49.440 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.433+0000 7f5fd9291700 1 --2- 172.21.15.110:0/3589777351 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5fd415d420 0x7f5fd414fe10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:49.440 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.433+0000 7f5fd9291700 1 -- 172.21.15.110:0/3589777351 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5fd4069d70 msgr2=0x7f5fd4150c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:49.440 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.433+0000 7f5fd9291700 1 --2- 172.21.15.110:0/3589777351 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5fd4069d70 0x7f5fd4150c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:49.440 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.433+0000 7f5fd9291700 1 -- 172.21.15.110:0/3589777351 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fbc0094f0 con 0x7f5fd415d7f0 2023-10-06T03:29:49.440 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.433+0000 7f5fd9291700 1 --2- 172.21.15.110:0/3589777351 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5fd415d7f0 0x7f5fd4150350 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f5fcc009df0 tx=0x7f5fcc00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:29:49.440 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.433+0000 7f5fd0ff9700 1 -- 172.21.15.110:0/3589777351 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fcc00caa0 con 0x7f5fd415d7f0 2023-10-06T03:29:49.440 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.433+0000 7f5fd0ff9700 1 -- 172.21.15.110:0/3589777351 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5fcc00f040 con 0x7f5fd415d7f0 2023-10-06T03:29:49.440 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.433+0000 7f5fdacf4700 1 -- 172.21.15.110:0/3589777351 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5fd4153330 con 0x7f5fd415d7f0 2023-10-06T03:29:49.440 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.433+0000 7f5fd0ff9700 1 -- 172.21.15.110:0/3589777351 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fcc00ce70 con 0x7f5fd415d7f0 2023-10-06T03:29:49.441 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.433+0000 7f5fdacf4700 1 -- 172.21.15.110:0/3589777351 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5fd4151140 con 0x7f5fd415d7f0 2023-10-06T03:29:49.441 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.433+0000 7f5fdacf4700 1 -- 172.21.15.110:0/3589777351 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5fd4153570 con 0x7f5fd415d7f0 2023-10-06T03:29:49.441 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.437+0000 7f5fd0ff9700 1 -- 172.21.15.110:0/3589777351 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7f5fcc014690 con 0x7f5fd415d7f0 2023-10-06T03:29:49.442 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.437+0000 7f5fd0ff9700 1 --2- 172.21.15.110:0/3589777351 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f5fb4060550 0x7f5fb4062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:49.442 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.437+0000 7f5fd0ff9700 1 -- 172.21.15.110:0/3589777351 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6809+0+0 (secure 0 0 0) 0x7f5fcc07d440 con 0x7f5fd415d7f0 2023-10-06T03:29:49.442 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.437+0000 7f5fd8a90700 1 --2- 172.21.15.110:0/3589777351 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f5fb4060550 0x7f5fb4062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:49.442 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.437+0000 7f5fd8a90700 1 --2- 172.21.15.110:0/3589777351 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f5fb4060550 0x7f5fb4062a00 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f5fc4004fd0 tx=0x7f5fc4007eb0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:29:49.445 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.441+0000 7f5fd0ff9700 1 -- 172.21.15.110:0/3589777351 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f5fd4153570 con 0x7f5fd415d7f0 2023-10-06T03:29:49.609 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.605+0000 7f5fdacf4700 1 -- 172.21.15.110:0/3589777351 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5fd4153570 con 0x7f5fd415d7f0 2023-10-06T03:29:49.611 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.605+0000 7f5fd0ff9700 1 -- 172.21.15.110:0/3589777351 <== mon.0 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v28) v1 ==== 74+0+18755 (secure 0 0 0) 0x7f5fd4153570 con 0x7f5fd415d7f0 2023-10-06T03:29:49.612 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-10-06T03:29:49.612 INFO:teuthology.orchestra.run.smithi110.stdout:{"epoch":28,"fsid":"93b8f6ac-4c92-4b45-b8e4-b59617024aa2","created":"2023-10-06T03:29:10.519583+0000","modified":"2023-10-06T03:29:38.951740+0000","last_up_change":"2023-10-06T03:29:21.960511+0000","last_in_change":"2023-10-06T03:29:12.560436+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-06T03:29:20.703743+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2023-10-06T03:29:28.307281+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2023-10-06T03:29:32.247251+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data","create_time":"2023-10-06T03:29:32.880086+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}}},{"pool":5,"pool_name":"cephfs_data_ec","create_time":"2023-10-06T03:29:36.613779+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"28","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"5866eb6c-2adc-46e4-be7a-e5bcda0f4065","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6824","nonce":33394},{"type":"v1","addr":"172.21.15.110:6825","nonce":33394}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6826","nonce":33394},{"type":"v1","addr":"172.21.15.110:6827","nonce":33394}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6830","nonce":33394},{"type":"v1","addr":"172.21.15.110:6831","nonce":33394}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6828","nonce":33394},{"type":"v1","addr":"172.21.15.110:6829","nonce":33394}]},"public_addr":"172.21.15.110:6825/33394","cluster_addr":"172.21.15.110:6827/33394","heartbeat_back_addr":"172.21.15.110:6831/33394","heartbeat_front_addr":"172.21.15.110:6829/33394","state":["exists","up"]},{"osd":1,"uuid":"a38de118-062b-4a05-9782-2ceddf6ff4df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6800","nonce":33396},{"type":"v1","addr":"172.21.15.110:6801","nonce":33396}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6802","nonce":33396},{"type":"v1","addr":"172.21.15.110:6803","nonce":33396}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6806","nonce":33396},{"type":"v1","addr":"172.21.15.110:6807","nonce":33396}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6804","nonce":33396},{"type":"v1","addr":"172.21.15.110:6805","nonce":33396}]},"public_addr":"172.21.15.110:6801/33396","cluster_addr":"172.21.15.110:6803/33396","heartbeat_back_addr":"172.21.15.110:6807/33396","heartbeat_front_addr":"172.21.15.110:6805/33396","state":["exists","up"]},{"osd":2,"uuid":"56e39be2-3407-4b69-877e-cf16617b06d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6816","nonce":33397},{"type":"v1","addr":"172.21.15.110:6817","nonce":33397}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6818","nonce":33397},{"type":"v1","addr":"172.21.15.110:6819","nonce":33397}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6822","nonce":33397},{"type":"v1","addr":"172.21.15.110:6823","nonce":33397}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6820","nonce":33397},{"type":"v1","addr":"172.21.15.110:6821","nonce":33397}]},"public_addr":"172.21.15.110:6817/33397","cluster_addr":"172.21.15.110:6819/33397","heartbeat_back_addr":"172.21.15.110:6823/33397","heartbeat_front_addr":"172.21.15.110:6821/33397","state":["exists","up"]},{"osd":3,"uuid":"01c82d13-c1d4-4bde-b55e-3e0089b40115","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6808","nonce":33395},{"type":"v1","addr":"172.21.15.110:6809","nonce":33395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6810","nonce":33395},{"type":"v1","addr":"172.21.15.110:6811","nonce":33395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6814","nonce":33395},{"type":"v1","addr":"172.21.15.110:6815","nonce":33395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6812","nonce":33395},{"type":"v1","addr":"172.21.15.110:6813","nonce":33395}]},"public_addr":"172.21.15.110:6809/33395","cluster_addr":"172.21.15.110:6811/33395","heartbeat_back_addr":"172.21.15.110:6815/33395","heartbeat_front_addr":"172.21.15.110:6813/33395","state":["exists","up"]},{"osd":4,"uuid":"86983430-5fa3-4d17-998f-03e2cd843237","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":33683},{"type":"v1","addr":"172.21.15.192:6817","nonce":33683}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":33683},{"type":"v1","addr":"172.21.15.192:6819","nonce":33683}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":33683},{"type":"v1","addr":"172.21.15.192:6823","nonce":33683}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6820","nonce":33683},{"type":"v1","addr":"172.21.15.192:6821","nonce":33683}]},"public_addr":"172.21.15.192:6817/33683","cluster_addr":"172.21.15.192:6819/33683","heartbeat_back_addr":"172.21.15.192:6823/33683","heartbeat_front_addr":"172.21.15.192:6821/33683","state":["exists","up"]},{"osd":5,"uuid":"6945257f-821d-4619-a63f-ebe857eba6f4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":33684},{"type":"v1","addr":"172.21.15.192:6801","nonce":33684}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":33684},{"type":"v1","addr":"172.21.15.192:6803","nonce":33684}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":33684},{"type":"v1","addr":"172.21.15.192:6807","nonce":33684}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6804","nonce":33684},{"type":"v1","addr":"172.21.15.192:6805","nonce":33684}]},"public_addr":"172.21.15.192:6801/33684","cluster_addr":"172.21.15.192:6803/33684","heartbeat_back_addr":"172.21.15.192:6807/33684","heartbeat_front_addr":"172.21.15.192:6805/33684","state":["exists","up"]},{"osd":6,"uuid":"7854d1ae-7060-4780-8f97-73cea5e7a12f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6824","nonce":33686},{"type":"v1","addr":"172.21.15.192:6825","nonce":33686}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":33686},{"type":"v1","addr":"172.21.15.192:6827","nonce":33686}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":33686},{"type":"v1","addr":"172.21.15.192:6831","nonce":33686}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6828","nonce":33686},{"type":"v1","addr":"172.21.15.192:6829","nonce":33686}]},"public_addr":"172.21.15.192:6825/33686","cluster_addr":"172.21.15.192:6827/33686","heartbeat_back_addr":"172.21.15.192:6831/33686","heartbeat_front_addr":"172.21.15.192:6829/33686","state":["exists","up"]},{"osd":7,"uuid":"f1ebdfd5-b859-482b-9f97-c18c4a26f445","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6808","nonce":33685},{"type":"v1","addr":"172.21.15.192:6809","nonce":33685}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":33685},{"type":"v1","addr":"172.21.15.192:6811","nonce":33685}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":33685},{"type":"v1","addr":"172.21.15.192:6815","nonce":33685}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6812","nonce":33685},{"type":"v1","addr":"172.21.15.192:6813","nonce":33685}]},"public_addr":"172.21.15.192:6809/33685","cluster_addr":"172.21.15.192:6811/33685","heartbeat_back_addr":"172.21.15.192:6815/33685","heartbeat_front_addr":"172.21.15.192:6813/33685","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T03:29:18.002048+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T03:29:17.478576+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T03:29:18.034944+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T03:29:17.628513+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T03:29:18.047338+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T03:29:17.772349+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T03:29:17.883958+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T03:29:18.151838+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-06T03:29:49.614 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.609+0000 7f5fdacf4700 1 -- 172.21.15.110:0/3589777351 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f5fb4060550 msgr2=0x7f5fb4062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:49.614 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.609+0000 7f5fdacf4700 1 --2- 172.21.15.110:0/3589777351 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f5fb4060550 0x7f5fb4062a00 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f5fc4004fd0 tx=0x7f5fc4007eb0 comp rx=0 tx=0).stop 2023-10-06T03:29:49.614 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.609+0000 7f5fdacf4700 1 -- 172.21.15.110:0/3589777351 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5fd415d7f0 msgr2=0x7f5fd4150350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:49.614 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.609+0000 7f5fdacf4700 1 --2- 172.21.15.110:0/3589777351 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5fd415d7f0 0x7f5fd4150350 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f5fcc009df0 tx=0x7f5fcc00b040 comp rx=0 tx=0).stop 2023-10-06T03:29:49.614 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.609+0000 7f5fdacf4700 1 -- 172.21.15.110:0/3589777351 shutdown_connections 2023-10-06T03:29:49.615 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.609+0000 7f5fdacf4700 1 --2- 172.21.15.110:0/3589777351 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f5fb4060550 0x7f5fb4062a00 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:49.615 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.609+0000 7f5fdacf4700 1 --2- 172.21.15.110:0/3589777351 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5fd415d7f0 0x7f5fd4150350 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:49.615 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.609+0000 7f5fdacf4700 1 --2- 172.21.15.110:0/3589777351 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5fd415d420 0x7f5fd414fe10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:49.615 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.609+0000 7f5fdacf4700 1 --2- 172.21.15.110:0/3589777351 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5fd4069d70 0x7f5fd4150c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:49.615 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.609+0000 7f5fdacf4700 1 -- 172.21.15.110:0/3589777351 >> 172.21.15.110:0/3589777351 conn(0x7f5fd40cbd00 msgr2=0x7f5fd40c8630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:29:49.615 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.609+0000 7f5fdacf4700 1 -- 172.21.15.110:0/3589777351 shutdown_connections 2023-10-06T03:29:49.615 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.609+0000 7f5fdacf4700 1 -- 172.21.15.110:0/3589777351 wait complete. 2023-10-06T03:29:49.626 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2023-10-06T03:29:49.626 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2023-10-06T03:29:49.627 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2023-10-06T03:29:49.627 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2023-10-06T03:29:49.628 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2023-10-06T03:29:49.628 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2023-10-06T03:29:49.628 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2023-10-06T03:29:49.629 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2023-10-06T03:29:49.765 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.761+0000 7fe493a00700 1 Processor -- start 2023-10-06T03:29:49.766 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.761+0000 7fe493a00700 1 -- start start 2023-10-06T03:29:49.767 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.761+0000 7fe493a00700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fe48c0c8c10 0x7fe48c0c9750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:49.767 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.761+0000 7fe493a00700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fe48c0c75e0 0x7fe48c154d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:49.767 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.761+0000 7fe493a00700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe48c0c7d10 con 0x7fe48c0c20d0 2023-10-06T03:29:49.767 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.761+0000 7fe493a00700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe48c0c15f0 con 0x7fe48c0c8c10 2023-10-06T03:29:49.767 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.761+0000 7fe493a00700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe48c0c18d0 con 0x7fe48c0c75e0 2023-10-06T03:29:49.768 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.765+0000 7fe490f9b700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fe48c0c75e0 0x7fe48c154d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:49.768 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.765+0000 7fe490f9b700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fe48c0c75e0 0x7fe48c154d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.110:50698/0 (socket says 172.21.15.110:50698) 2023-10-06T03:29:49.768 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.765+0000 7fe490f9b700 1 -- 172.21.15.110:0/272732853 learned_addr learned my addr 172.21.15.110:0/272732853 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:29:49.768 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.765+0000 7fe49179c700 1 --2- 172.21.15.110:0/272732853 >> v2:172.21.15.110:3300/0 conn(0x7fe48c0c8c10 0x7fe48c0c9750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:49.768 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.761+0000 7f07993da700 1 Processor -- start 2023-10-06T03:29:49.768 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.761+0000 7f07993da700 1 -- start start 2023-10-06T03:29:49.769 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.765+0000 7fe490f9b700 1 -- 172.21.15.110:0/272732853 >> v2:172.21.15.110:3300/0 conn(0x7fe48c0c8c10 msgr2=0x7fe48c0c9750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:29:49.769 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.765+0000 7fe490f9b700 1 --2- 172.21.15.110:0/272732853 >> v2:172.21.15.110:3300/0 conn(0x7fe48c0c8c10 0x7fe48c0c9750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:49.769 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.765+0000 7fe490f9b700 1 -- 172.21.15.110:0/272732853 >> v1:172.21.15.110:6789/0 conn(0x7fe48c0c20d0 legacy=0x7fe48c0ca3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:29:49.769 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.765+0000 7fe490f9b700 1 -- 172.21.15.110:0/272732853 --> v2:172.21.15.192:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe48c0c7a10 con 0x7fe48c0c75e0 2023-10-06T03:29:49.769 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.765+0000 7fe490f9b700 1 --2- 172.21.15.110:0/272732853 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe48c0c75e0 0x7fe48c154d70 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fe48c0cba60 tx=0x7fe484009840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3d9be52315522942 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:29:49.770 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.765+0000 7f07993da700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0794155160 0x7f0794069420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:49.770 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.765+0000 7f07993da700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07940c7ad0 con 0x7f0794154d90 2023-10-06T03:29:49.770 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.765+0000 7f07993da700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07940c7f60 con 0x7f0794067ab0 2023-10-06T03:29:49.770 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.765+0000 7f07993da700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07941555b0 con 0x7f0794155160 2023-10-06T03:29:49.770 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.765+0000 7fe483fff700 1 -- 172.21.15.110:0/272732853 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe48400e070 con 0x7fe48c0c75e0 2023-10-06T03:29:49.771 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.765+0000 7f0792ffd700 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f0794154d90 0x7f079406c180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:37290/0 (socket says 172.21.15.110:37290) 2023-10-06T03:29:49.771 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.765+0000 7f0792ffd700 1 -- 172.21.15.110:0/3256906571 learned_addr learned my addr 172.21.15.110:0/3256906571 (peer_addr_for_me v1:172.21.15.110:0/0) 2023-10-06T03:29:49.771 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.765+0000 7fe483fff700 1 -- 172.21.15.110:0/272732853 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe48400b9e0 con 0x7fe48c0c75e0 2023-10-06T03:29:49.771 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.765+0000 7fe483fff700 1 -- 172.21.15.110:0/272732853 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4840092e0 con 0x7fe48c0c75e0 2023-10-06T03:29:49.772 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.765+0000 7f07927fc700 1 --2- 172.21.15.110:0/3256906571 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0794155160 0x7f0794069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:49.772 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.765+0000 7f0791ffb700 1 -- 172.21.15.110:0/3256906571 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1383028785 0 0) 0x7f07940c7ad0 con 0x7f0794154d90 2023-10-06T03:29:49.772 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.765+0000 7f0791ffb700 1 -- 172.21.15.110:0/3256906571 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f077c003570 con 0x7f0794154d90 2023-10-06T03:29:49.772 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.765+0000 7f07927fc700 1 -- 172.21.15.110:0/3256906571 >> v1:172.21.15.192:6789/0 conn(0x7f0794067ab0 legacy=0x7f07940c9990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:49.772 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.765+0000 7f07927fc700 1 -- 172.21.15.110:0/3256906571 >> v1:172.21.15.110:6789/0 conn(0x7f0794154d90 legacy=0x7f079406c180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:49.772 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.765+0000 7f07927fc700 1 -- 172.21.15.110:0/3256906571 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07940c6ec0 con 0x7f0794155160 2023-10-06T03:29:49.773 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.765+0000 7f6ec0ba6700 1 Processor -- start 2023-10-06T03:29:49.773 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.765+0000 7f6ec0ba6700 1 -- start start 2023-10-06T03:29:49.773 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.765+0000 7fe493a00700 1 -- 172.21.15.110:0/272732853 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe48c0c75e0 msgr2=0x7fe48c154d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:49.773 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.765+0000 7fe493a00700 1 --2- 172.21.15.110:0/272732853 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe48c0c75e0 0x7fe48c154d70 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fe48c0cba60 tx=0x7fe484009840 comp rx=0 tx=0).stop 2023-10-06T03:29:49.774 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.765+0000 7fe493a00700 1 -- 172.21.15.110:0/272732853 shutdown_connections 2023-10-06T03:29:49.774 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.765+0000 7fe493a00700 1 --2- 172.21.15.110:0/272732853 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe48c0c75e0 0x7fe48c154d70 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:49.774 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.765+0000 7fe493a00700 1 --2- 172.21.15.110:0/272732853 >> v2:172.21.15.110:3300/0 conn(0x7fe48c0c8c10 0x7fe48c0c9750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:49.774 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.765+0000 7fe493a00700 1 -- 172.21.15.110:0/272732853 >> 172.21.15.110:0/272732853 conn(0x7fe48c0cc560 msgr2=0x7fe48c0c8500 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:29:49.774 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.765+0000 7fe493a00700 1 -- 172.21.15.110:0/272732853 shutdown_connections 2023-10-06T03:29:49.774 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.765+0000 7fe493a00700 1 -- 172.21.15.110:0/272732853 wait complete. 2023-10-06T03:29:49.774 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.765+0000 7f07927fc700 1 --2- 172.21.15.110:0/3256906571 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0794155160 0x7f0794069420 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f078c009e90 tx=0x7f078c009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=69589ea24ccc7521 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:29:49.774 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.765+0000 7f0791ffb700 1 -- 172.21.15.110:0/3256906571 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f078c00e070 con 0x7f0794155160 2023-10-06T03:29:49.775 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.765+0000 7f0791ffb700 1 -- 172.21.15.110:0/3256906571 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f078c00bc80 con 0x7f0794155160 2023-10-06T03:29:49.775 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.765+0000 7f0791ffb700 1 -- 172.21.15.110:0/3256906571 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f078c012430 con 0x7f0794155160 2023-10-06T03:29:49.775 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.765+0000 7f6ec0ba6700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6ebc15b130 0x7f6ebc159960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:49.775 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.765+0000 7f6ec0ba6700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ebc06c270 con 0x7f6ebc15ad60 2023-10-06T03:29:49.775 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.765+0000 7f6ec0ba6700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ebc06c550 con 0x7f6ebc159ea0 2023-10-06T03:29:49.775 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.765+0000 7f6ec0ba6700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ebc15aa10 con 0x7f6ebc15b130 2023-10-06T03:29:49.776 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.769+0000 7f6ebb7fe700 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f6ebc15ad60 0x7f6ebc0adc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:37298/0 (socket says 172.21.15.110:37298) 2023-10-06T03:29:49.776 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.769+0000 7f6ebb7fe700 1 -- 172.21.15.110:0/3704995055 learned_addr learned my addr 172.21.15.110:0/3704995055 (peer_addr_for_me v1:172.21.15.110:0/0) 2023-10-06T03:29:49.776 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.769+0000 7f07993da700 1 -- 172.21.15.110:0/3256906571 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0794155160 msgr2=0x7f0794069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:49.776 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.769+0000 7f07993da700 1 --2- 172.21.15.110:0/3256906571 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0794155160 0x7f0794069420 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f078c009e90 tx=0x7f078c009780 comp rx=0 tx=0).stop 2023-10-06T03:29:49.777 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.769+0000 7f6ebaffd700 1 --2- 172.21.15.110:0/3704995055 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6ebc15b130 0x7f6ebc159960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:49.777 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.769+0000 7f6eba7fc700 1 -- 172.21.15.110:0/3704995055 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2371740853 0 0) 0x7f6ebc06c270 con 0x7f6ebc15ad60 2023-10-06T03:29:49.777 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.769+0000 7f6eba7fc700 1 -- 172.21.15.110:0/3704995055 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6ea4003570 con 0x7f6ebc15ad60 2023-10-06T03:29:49.777 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.769+0000 7f6eba7fc700 1 -- 172.21.15.110:0/3704995055 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1658133866 0 0) 0x7f6ebc06c550 con 0x7f6ebc159ea0 2023-10-06T03:29:49.777 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.769+0000 7f6eba7fc700 1 -- 172.21.15.110:0/3704995055 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6ebc06c270 con 0x7f6ebc159ea0 2023-10-06T03:29:49.777 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.769+0000 7fe493a00700 1 Processor -- start 2023-10-06T03:29:49.778 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.769+0000 7f6eba7fc700 1 -- 172.21.15.110:0/3704995055 <== mon.0 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2954752096 0 0) 0x7f6ea4003570 con 0x7f6ebc15ad60 2023-10-06T03:29:49.778 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.769+0000 7f6eba7fc700 1 -- 172.21.15.110:0/3704995055 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6ebc15b130 msgr2=0x7f6ebc159960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:29:49.778 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.769+0000 7f6eba7fc700 1 --2- 172.21.15.110:0/3704995055 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6ebc15b130 0x7f6ebc159960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:49.778 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.769+0000 7f6eba7fc700 1 -- 172.21.15.110:0/3704995055 >> v1:172.21.15.192:6789/0 conn(0x7f6ebc159ea0 legacy=0x7f6ebc069a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:49.779 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.769+0000 7f07993da700 1 -- 172.21.15.110:0/3256906571 shutdown_connections 2023-10-06T03:29:49.779 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.769+0000 7f07993da700 1 --2- 172.21.15.110:0/3256906571 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0794155160 0x7f0794069420 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:49.779 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.769+0000 7f07993da700 1 -- 172.21.15.110:0/3256906571 >> 172.21.15.110:0/3256906571 conn(0x7f07940cc560 msgr2=0x7f07940c8500 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:29:49.779 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.769+0000 7f6eba7fc700 1 -- 172.21.15.110:0/3704995055 --> v1:172.21.15.110:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ebc06c830 con 0x7f6ebc15ad60 2023-10-06T03:29:49.779 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.769+0000 7f6eba7fc700 1 -- 172.21.15.110:0/3704995055 <== mon.0 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 118603703 0 0) 0x7f6eac001470 con 0x7f6ebc15ad60 2023-10-06T03:29:49.779 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.769+0000 7f6eba7fc700 1 -- 172.21.15.110:0/3704995055 >> v1:172.21.15.110:6789/0 conn(0x7f6ebc15ad60 legacy=0x7f6ebc0adc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:49.779 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.769+0000 7f6eba7fc700 1 --2- 172.21.15.110:0/3704995055 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6ea4003c20 0x7f6ea4004010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:49.780 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.769+0000 7f6eba7fc700 1 --2- 172.21.15.110:0/3704995055 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6ea4004810 0x7f6ea4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:49.780 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.769+0000 7f6eba7fc700 1 --2- 172.21.15.110:0/3704995055 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6ea40051c0 0x7f6ea400fe70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:49.780 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.769+0000 7f6eba7fc700 1 -- 172.21.15.110:0/3704995055 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ebc15aa10 con 0x7f6ea40051c0 2023-10-06T03:29:49.780 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.769+0000 7f6eba7fc700 1 -- 172.21.15.110:0/3704995055 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ea4010490 con 0x7f6ea4003c20 2023-10-06T03:29:49.780 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.769+0000 7f6eba7fc700 1 -- 172.21.15.110:0/3704995055 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ea4010650 con 0x7f6ea4004810 2023-10-06T03:29:49.780 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.769+0000 7f07993da700 1 -- 172.21.15.110:0/3256906571 shutdown_connections 2023-10-06T03:29:49.780 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.769+0000 7f07993da700 1 -- 172.21.15.110:0/3256906571 wait complete. 2023-10-06T03:29:49.781 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.769+0000 7fe493a00700 1 -- start start 2023-10-06T03:29:49.781 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.769+0000 7fe493a00700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe48c0c20d0 0x7fe48c0ad6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:49.781 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.769+0000 7fe493a00700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe48c0c75e0 0x7fe48c0c4c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:49.781 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.769+0000 7fe493a00700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe48c0c8c10 0x7fe48c0a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:49.781 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.769+0000 7fe493a00700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe48c0ac330 con 0x7fe48c0c20d0 2023-10-06T03:29:49.782 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.769+0000 7fe493a00700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe48c0ac610 con 0x7fe48c0c8c10 2023-10-06T03:29:49.782 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.769+0000 7fe493a00700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe48c0ac8f0 con 0x7fe48c0c75e0 2023-10-06T03:29:49.782 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.769+0000 7fe490f9b700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe48c0c75e0 0x7fe48c0c4c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:49.782 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.769+0000 7fe490f9b700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe48c0c75e0 0x7fe48c0c4c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3301/0 says I am v2:172.21.15.110:44052/0 (socket says 172.21.15.110:44052) 2023-10-06T03:29:49.782 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.769+0000 7fe490f9b700 1 -- 172.21.15.110:0/1617108862 learned_addr learned my addr 172.21.15.110:0/1617108862 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:29:49.782 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.769+0000 7fe491f9d700 1 --2- 172.21.15.110:0/1617108862 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe48c0c8c10 0x7fe48c0a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:49.782 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.769+0000 7f6ebbfff700 1 --2- 172.21.15.110:0/3704995055 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6ea40051c0 0x7f6ea400fe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:49.782 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.769+0000 7f6ebaffd700 1 --2- 172.21.15.110:0/3704995055 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6ea4004810 0x7f6ea4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:49.782 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.769+0000 7f6ebbfff700 1 -- 172.21.15.110:0/3704995055 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6ea4004810 msgr2=0x7f6ea4004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:29:49.783 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.769+0000 7f6ebbfff700 1 --2- 172.21.15.110:0/3704995055 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6ea4004810 0x7f6ea4004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:49.783 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.769+0000 7f6ebbfff700 1 -- 172.21.15.110:0/3704995055 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6ea4003c20 msgr2=0x7f6ea4004010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:29:49.783 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.769+0000 7f6ebbfff700 1 --2- 172.21.15.110:0/3704995055 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6ea4003c20 0x7f6ea4004010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:49.783 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.769+0000 7f6ebbfff700 1 -- 172.21.15.110:0/3704995055 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ebc06c830 con 0x7f6ea40051c0 2023-10-06T03:29:49.783 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.769+0000 7f6ebbfff700 1 --2- 172.21.15.110:0/3704995055 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6ea40051c0 0x7f6ea400fe70 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f6eb4002110 tx=0x7f6eb40017c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=314de6511d63d02 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:29:49.783 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.769+0000 7f6eba7fc700 1 -- 172.21.15.110:0/3704995055 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6eb4001ad0 con 0x7f6ea40051c0 2023-10-06T03:29:49.783 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.769+0000 7f6eba7fc700 1 -- 172.21.15.110:0/3704995055 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6eb400c980 con 0x7f6ea40051c0 2023-10-06T03:29:49.783 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.769+0000 7f6eba7fc700 1 -- 172.21.15.110:0/3704995055 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6eb400cd40 con 0x7f6ea40051c0 2023-10-06T03:29:49.784 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.769+0000 7fe49179c700 1 --2- 172.21.15.110:0/1617108862 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe48c0c20d0 0x7fe48c0ad6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:49.784 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.769+0000 7fe490f9b700 1 -- 172.21.15.110:0/1617108862 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe48c0c8c10 msgr2=0x7fe48c0a53e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:49.784 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.769+0000 7fe490f9b700 1 --2- 172.21.15.110:0/1617108862 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe48c0c8c10 0x7fe48c0a53e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:49.784 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.769+0000 7fe490f9b700 1 -- 172.21.15.110:0/1617108862 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe48c0c20d0 msgr2=0x7fe48c0ad6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:49.784 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.769+0000 7fe490f9b700 1 --2- 172.21.15.110:0/1617108862 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe48c0c20d0 0x7fe48c0ad6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:49.784 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.769+0000 7fe490f9b700 1 -- 172.21.15.110:0/1617108862 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe484009ae0 con 0x7fe48c0c75e0 2023-10-06T03:29:49.784 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.769+0000 7f6ec0ba6700 1 -- 172.21.15.110:0/3704995055 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6ea40051c0 msgr2=0x7f6ea400fe70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:49.784 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.769+0000 7f6ec0ba6700 1 --2- 172.21.15.110:0/3704995055 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6ea40051c0 0x7f6ea400fe70 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f6eb4002110 tx=0x7f6eb40017c0 comp rx=0 tx=0).stop 2023-10-06T03:29:49.785 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.769+0000 7f6ebbfff700 1 -- 172.21.15.110:0/3704995055 reap_dead start 2023-10-06T03:29:49.785 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.769+0000 7f07993da700 1 Processor -- start 2023-10-06T03:29:49.785 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.769+0000 7f07993da700 1 -- start start 2023-10-06T03:29:49.785 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.769+0000 7fe95759e700 1 Processor -- start 2023-10-06T03:29:49.786 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.769+0000 7f6ec0ba6700 1 -- 172.21.15.110:0/3704995055 shutdown_connections 2023-10-06T03:29:49.786 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.769+0000 7f6ec0ba6700 1 -- 172.21.15.110:0/3704995055 >> 172.21.15.110:0/3704995055 conn(0x7f6ebc06e760 msgr2=0x7f6ebc0cba30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:29:49.786 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.769+0000 7fe490f9b700 1 --2- 172.21.15.110:0/1617108862 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe48c0c75e0 0x7fe48c0c4c50 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fe48400b6c0 tx=0x7fe484017000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:29:49.786 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.769+0000 7f6ec0ba6700 1 -- 172.21.15.110:0/3704995055 shutdown_connections 2023-10-06T03:29:49.787 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.769+0000 7f6ec0ba6700 1 -- 172.21.15.110:0/3704995055 wait complete. 2023-10-06T03:29:49.787 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.773+0000 7fe4817fa700 1 -- 172.21.15.110:0/1617108862 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe484012b20 con 0x7fe48c0c75e0 2023-10-06T03:29:49.787 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.773+0000 7fe4817fa700 1 -- 172.21.15.110:0/1617108862 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe484012c80 con 0x7fe48c0c75e0 2023-10-06T03:29:49.787 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.773+0000 7fe4817fa700 1 -- 172.21.15.110:0/1617108862 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe48401d420 con 0x7fe48c0c75e0 2023-10-06T03:29:49.787 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.773+0000 7fe493a00700 1 -- 172.21.15.110:0/1617108862 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe48c0acbd0 con 0x7fe48c0c75e0 2023-10-06T03:29:49.787 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.773+0000 7fe493a00700 1 -- 172.21.15.110:0/1617108862 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe48c0ad190 con 0x7fe48c0c75e0 2023-10-06T03:29:49.788 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.773+0000 7f07993da700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0794067ab0 0x7f0794149d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:49.788 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.773+0000 7f07993da700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0794154d90 0x7f0794151330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:49.788 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.773+0000 7f07993da700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0794155160 0x7f0794148100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:49.788 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.769+0000 7fe95759e700 1 -- start start 2023-10-06T03:29:49.788 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.773+0000 7f07993da700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f079414a270 con 0x7f0794067ab0 2023-10-06T03:29:49.789 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.773+0000 7f07993da700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f079414f1e0 con 0x7f0794155160 2023-10-06T03:29:49.789 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.773+0000 7f07993da700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f079414f4c0 con 0x7f0794154d90 2023-10-06T03:29:49.789 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.773+0000 7f07927fc700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0794154d90 0x7f0794151330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:49.789 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.773+0000 7f07927fc700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0794154d90 0x7f0794151330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3301/0 says I am v2:172.21.15.110:44060/0 (socket says 172.21.15.110:44060) 2023-10-06T03:29:49.789 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.773+0000 7f07927fc700 1 -- 172.21.15.110:0/1348442177 learned_addr learned my addr 172.21.15.110:0/1348442177 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:29:49.789 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.773+0000 7fe95759e700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe9581550c0 0x7fe9580c7330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:49.790 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.773+0000 7f6ec0ba6700 1 Processor -- start 2023-10-06T03:29:49.790 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.773+0000 7fe95759e700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fe9581595e0 0x7fe9580c80e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:49.790 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.773+0000 7fe95759e700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe95806cbd0 con 0x7fe958154cf0 2023-10-06T03:29:49.790 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.773+0000 7fe95759e700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe958159a60 con 0x7fe9581595e0 2023-10-06T03:29:49.790 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.773+0000 7fe95759e700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9580c6ea0 con 0x7fe9581550c0 2023-10-06T03:29:49.791 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.773+0000 7f07927fc700 1 -- 172.21.15.110:0/1348442177 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0794155160 msgr2=0x7f0794148100 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-06T03:29:49.791 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.773+0000 7f07927fc700 1 --2- 172.21.15.110:0/1348442177 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0794155160 0x7f0794148100 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:49.791 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.773+0000 7f07927fc700 1 -- 172.21.15.110:0/1348442177 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0794067ab0 msgr2=0x7f0794149d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:49.791 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.773+0000 7f07927fc700 1 --2- 172.21.15.110:0/1348442177 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0794067ab0 0x7f0794149d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:49.791 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.773+0000 7f07927fc700 1 -- 172.21.15.110:0/1348442177 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f078c009a20 con 0x7f0794154d90 2023-10-06T03:29:49.791 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.773+0000 7fe955d9b700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe9581550c0 0x7fe9580c7330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:49.792 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.773+0000 7fe956d9d700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fe9581595e0 0x7fe9580c80e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:49.792 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.773+0000 7fe955d9b700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe9581550c0 0x7fe9580c7330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3301/0 says I am v2:172.21.15.110:44066/0 (socket says 172.21.15.110:44066) 2023-10-06T03:29:49.792 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.773+0000 7fe955d9b700 1 -- 172.21.15.110:0/169604864 learned_addr learned my addr 172.21.15.110:0/169604864 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:29:49.792 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.773+0000 7fe493a00700 1 -- 172.21.15.110:0/1617108862 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fe46c000fd0 con 0x7fe48c0c75e0 2023-10-06T03:29:49.792 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.773+0000 7f07927fc700 1 --2- 172.21.15.110:0/1348442177 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0794154d90 0x7f0794151330 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f078c009de0 tx=0x7f078c019000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:29:49.793 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.773+0000 7fe956d9d700 1 -- 172.21.15.110:0/169604864 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe9581550c0 msgr2=0x7fe9580c7330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:29:49.793 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.773+0000 7fe956d9d700 1 --2- 172.21.15.110:0/169604864 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe9581550c0 0x7fe9580c7330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:49.793 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.773+0000 7fe956d9d700 1 -- 172.21.15.110:0/169604864 >> v1:172.21.15.110:6789/0 conn(0x7fe958154cf0 legacy=0x7fe95806c3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:29:49.793 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.773+0000 7fe956d9d700 1 -- 172.21.15.110:0/169604864 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9580c6570 con 0x7fe9581595e0 2023-10-06T03:29:49.793 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.773+0000 7f077affd700 1 -- 172.21.15.110:0/1348442177 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f078c012ce0 con 0x7f0794154d90 2023-10-06T03:29:49.794 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.773+0000 7f077affd700 1 -- 172.21.15.110:0/1348442177 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f078c021070 con 0x7f0794154d90 2023-10-06T03:29:49.794 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.773+0000 7f077affd700 1 -- 172.21.15.110:0/1348442177 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f078c026680 con 0x7f0794154d90 2023-10-06T03:29:49.794 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.773+0000 7f07993da700 1 -- 172.21.15.110:0/1348442177 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f079414f7a0 con 0x7f0794154d90 2023-10-06T03:29:49.794 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.773+0000 7f07993da700 1 -- 172.21.15.110:0/1348442177 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f079414c940 con 0x7f0794154d90 2023-10-06T03:29:49.794 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.773+0000 7f6ec0ba6700 1 -- start start 2023-10-06T03:29:49.795 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.773+0000 7f6ec0ba6700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6ea40051c0 0x7f6ebc159090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:49.795 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.773+0000 7f6ec0ba6700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6ebc16d230 0x7f6ebc16d6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:49.795 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.773+0000 7f6ec0ba6700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6ebc0c1780 0x7f6ebc0c1bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:49.795 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.773+0000 7f6ec0ba6700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ebc1595d0 con 0x7f6ebc0c1780 2023-10-06T03:29:49.795 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.773+0000 7f6ec0ba6700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ebc16dbe0 con 0x7f6ea40051c0 2023-10-06T03:29:49.795 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.773+0000 7f6ec0ba6700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ebc0c2130 con 0x7f6ebc16d230 2023-10-06T03:29:49.795 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.773+0000 7f6ebb7fe700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6ea40051c0 0x7f6ebc159090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:49.795 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.773+0000 7f6ebaffd700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6ebc16d230 0x7f6ebc16d6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:49.795 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.773+0000 7f6ebb7fe700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6ea40051c0 0x7f6ebc159090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.110:50728/0 (socket says 172.21.15.110:50728) 2023-10-06T03:29:49.795 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.773+0000 7f6ebb7fe700 1 -- 172.21.15.110:0/1520905109 learned_addr learned my addr 172.21.15.110:0/1520905109 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:29:49.796 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.773+0000 7f6ebbfff700 1 --2- 172.21.15.110:0/1520905109 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6ebc0c1780 0x7f6ebc0c1bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:49.796 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.773+0000 7fe956d9d700 1 --2- 172.21.15.110:0/169604864 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe9581595e0 0x7fe9580c80e0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fe9480072f0 tx=0x7fe94800abc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a2b55a91d5d9087 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:29:49.796 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.773+0000 7f94f359e700 1 Processor -- start 2023-10-06T03:29:49.796 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.773+0000 7f94f359e700 1 -- start start 2023-10-06T03:29:49.796 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.773+0000 7f94f359e700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f94ec0de360 0x7f94ec04fd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:49.797 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.777+0000 7f1794fe8700 1 Processor -- start 2023-10-06T03:29:49.797 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.773+0000 7fe95559a700 1 -- 172.21.15.110:0/169604864 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe94800f070 con 0x7fe9581595e0 2023-10-06T03:29:49.797 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.773+0000 7fe95559a700 1 -- 172.21.15.110:0/169604864 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe948008bf0 con 0x7fe9581595e0 2023-10-06T03:29:49.797 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.773+0000 7fe95559a700 1 -- 172.21.15.110:0/169604864 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe94800a0d0 con 0x7fe9581595e0 2023-10-06T03:29:49.798 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.777+0000 7f6bad888700 1 Processor -- start 2023-10-06T03:29:49.798 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.773+0000 7f94f359e700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f94ec053c00 0x7f94ec050260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:49.798 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.773+0000 7f94f359e700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94ec034790 con 0x7f94ec0ddf90 2023-10-06T03:29:49.798 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.773+0000 7f94f359e700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f94ec006050 con 0x7f94ec0de360 2023-10-06T03:29:49.799 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.773+0000 7fe4817fa700 1 -- 172.21.15.110:0/1617108862 <== mon.2 v2:172.21.15.192:3301/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7fe48401d7b0 con 0x7fe48c0c75e0 2023-10-06T03:29:49.799 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.777+0000 7fe4817fa700 1 --2- 172.21.15.110:0/1617108862 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fe4700606a0 0x7fe470062b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:49.799 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.777+0000 7fe4817fa700 1 -- 172.21.15.110:0/1617108862 <== mon.2 v2:172.21.15.192:3301/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6809+0+0 (secure 0 0 0) 0x7fe48407ea00 con 0x7fe48c0c75e0 2023-10-06T03:29:49.799 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.777+0000 7fe49179c700 1 --2- 172.21.15.110:0/1617108862 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fe4700606a0 0x7fe470062b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:49.799 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.777+0000 7fe4817fa700 1 --2- 172.21.15.110:0/1617108862 >> [v2:172.21.15.110:6824/33394,v1:172.21.15.110:6825/33394] conn(0x7fe47006a370 0x7fe47006c7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:49.799 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.777+0000 7fe4817fa700 1 -- 172.21.15.110:0/1617108862 --> [v2:172.21.15.110:6824/33394,v1:172.21.15.110:6825/33394] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe47006ce70 con 0x7fe47006a370 2023-10-06T03:29:49.799 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.777+0000 7fe4817fa700 1 -- 172.21.15.110:0/1617108862 <== mon.2 v2:172.21.15.192:3301/0 6 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe48407ed30 con 0x7fe48c0c75e0 2023-10-06T03:29:49.800 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.773+0000 7f6ebbfff700 1 -- 172.21.15.110:0/1520905109 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6ebc16d230 msgr2=0x7f6ebc16d6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:49.800 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.773+0000 7f6ebbfff700 1 --2- 172.21.15.110:0/1520905109 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6ebc16d230 0x7f6ebc16d6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:49.800 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.773+0000 7f6ebbfff700 1 -- 172.21.15.110:0/1520905109 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6ea40051c0 msgr2=0x7f6ebc159090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:49.800 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.773+0000 7f6ebbfff700 1 --2- 172.21.15.110:0/1520905109 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6ea40051c0 0x7f6ebc159090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:49.800 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.773+0000 7f6ebbfff700 1 -- 172.21.15.110:0/1520905109 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ebc0bc630 con 0x7f6ebc0c1780 2023-10-06T03:29:49.800 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.773+0000 7f6ebbfff700 1 --2- 172.21.15.110:0/1520905109 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6ebc0c1780 0x7f6ebc0c1bf0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f6eb40013d0 tx=0x7f6eb4016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:29:49.800 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.777+0000 7f6ea37fe700 1 -- 172.21.15.110:0/1520905109 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6eb4001ad0 con 0x7f6ebc0c1780 2023-10-06T03:29:49.800 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.777+0000 7f6ec0ba6700 1 -- 172.21.15.110:0/1520905109 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ebc0bb920 con 0x7f6ebc0c1780 2023-10-06T03:29:49.801 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.777+0000 7f07993da700 1 -- 172.21.15.110:0/1348442177 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f0774000fd0 con 0x7f0794154d90 2023-10-06T03:29:49.801 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.777+0000 7f6ec0ba6700 1 -- 172.21.15.110:0/1520905109 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ebc16e0f0 con 0x7f6ebc0c1780 2023-10-06T03:29:49.801 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.777+0000 7f6ea37fe700 1 -- 172.21.15.110:0/1520905109 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6eb4023070 con 0x7f6ebc0c1780 2023-10-06T03:29:49.801 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.777+0000 7f6ea37fe700 1 -- 172.21.15.110:0/1520905109 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6eb4006ba0 con 0x7f6ebc0c1780 2023-10-06T03:29:49.802 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.777+0000 7f6bad888700 1 -- start start 2023-10-06T03:29:49.802 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.777+0000 7fe95759e700 1 -- 172.21.15.110:0/169604864 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe9581595e0 msgr2=0x7fe9580c80e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:49.802 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.777+0000 7fe95759e700 1 --2- 172.21.15.110:0/169604864 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe9581595e0 0x7fe9580c80e0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fe9480072f0 tx=0x7fe94800abc0 comp rx=0 tx=0).stop 2023-10-06T03:29:49.802 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.773+0000 7f94f359e700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f94ec054a40 con 0x7f94ec053c00 2023-10-06T03:29:49.803 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.777+0000 7f94f1d9b700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f94ec0de360 0x7f94ec04fd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:49.803 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.777+0000 7fe491f9d700 1 --2- 172.21.15.110:0/1617108862 >> [v2:172.21.15.110:6824/33394,v1:172.21.15.110:6825/33394] conn(0x7fe47006a370 0x7fe47006c7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:49.803 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.777+0000 7fe49179c700 1 --2- 172.21.15.110:0/1617108862 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fe4700606a0 0x7fe470062b50 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fe488009ca0 tx=0x7fe488008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:29:49.803 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.777+0000 7fe491f9d700 1 --2- 172.21.15.110:0/1617108862 >> [v2:172.21.15.110:6824/33394,v1:172.21.15.110:6825/33394] conn(0x7fe47006a370 0x7fe47006c7a0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:29:49.803 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.777+0000 7f0c77e65700 1 Processor -- start 2023-10-06T03:29:49.804 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.777+0000 7f94f1d9b700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f94ec0de360 0x7f94ec04fd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:60584/0 (socket says 172.21.15.110:60584) 2023-10-06T03:29:49.804 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.777+0000 7f94f1d9b700 1 -- 172.21.15.110:0/2160324124 learned_addr learned my addr 172.21.15.110:0/2160324124 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:29:49.804 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.777+0000 7f94f2d9d700 1 --2- 172.21.15.110:0/2160324124 >> v2:172.21.15.192:3300/0 conn(0x7f94ec053c00 0x7f94ec050260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:49.804 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.777+0000 7f94f1d9b700 1 -- 172.21.15.110:0/2160324124 >> v2:172.21.15.192:3300/0 conn(0x7f94ec053c00 msgr2=0x7f94ec050260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:29:49.804 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.777+0000 7f94f1d9b700 1 --2- 172.21.15.110:0/2160324124 >> v2:172.21.15.192:3300/0 conn(0x7f94ec053c00 0x7f94ec050260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:49.804 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.777+0000 7f94f1d9b700 1 -- 172.21.15.110:0/2160324124 >> v1:172.21.15.110:6789/0 conn(0x7f94ec0ddf90 legacy=0x7f94ec034080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:29:49.804 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.777+0000 7f94f1d9b700 1 -- 172.21.15.110:0/2160324124 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94ec0541a0 con 0x7f94ec0de360 2023-10-06T03:29:49.804 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.777+0000 7f94f1d9b700 1 --2- 172.21.15.110:0/2160324124 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f94ec0de360 0x7f94ec04fd20 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f94ec053190 tx=0x7f94e8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b0c6879930c93fbe server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:29:49.804 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.777+0000 7f94f159a700 1 -- 172.21.15.110:0/2160324124 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94e8009e70 con 0x7f94ec0de360 2023-10-06T03:29:49.804 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.777+0000 7f94f159a700 1 -- 172.21.15.110:0/2160324124 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94e800b820 con 0x7f94ec0de360 2023-10-06T03:29:49.804 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.777+0000 7f94f159a700 1 -- 172.21.15.110:0/2160324124 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94e800bbc0 con 0x7f94ec0de360 2023-10-06T03:29:49.805 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.777+0000 7f0c77e65700 1 -- start start 2023-10-06T03:29:49.805 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.777+0000 7f6ec0ba6700 1 -- 172.21.15.110:0/1520905109 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f6e9c000fd0 con 0x7f6ebc0c1780 2023-10-06T03:29:49.805 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.777+0000 7f077affd700 1 -- 172.21.15.110:0/1348442177 <== mon.2 v2:172.21.15.192:3301/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7f078c023030 con 0x7f0794154d90 2023-10-06T03:29:49.806 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.777+0000 7f1794fe8700 1 -- start start 2023-10-06T03:29:49.806 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.777+0000 7fe95759e700 1 -- 172.21.15.110:0/169604864 shutdown_connections 2023-10-06T03:29:49.806 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.777+0000 7fe95759e700 1 --2- 172.21.15.110:0/169604864 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe9581595e0 0x7fe9580c80e0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:49.806 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.777+0000 7fe95759e700 1 --2- 172.21.15.110:0/169604864 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe9581550c0 0x7fe9580c7330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:49.806 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.777+0000 7fe95759e700 1 -- 172.21.15.110:0/169604864 >> 172.21.15.110:0/169604864 conn(0x7fe95806dce0 msgr2=0x7fe95806e0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:29:49.807 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.777+0000 7f94f359e700 1 -- 172.21.15.110:0/2160324124 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f94ec0de360 msgr2=0x7f94ec04fd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:49.807 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.777+0000 7f94f359e700 1 --2- 172.21.15.110:0/2160324124 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f94ec0de360 0x7f94ec04fd20 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f94ec053190 tx=0x7f94e8009250 comp rx=0 tx=0).stop 2023-10-06T03:29:49.807 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.777+0000 7f077affd700 1 --2- 172.21.15.110:0/1348442177 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f07800605d0 0x7f0780062a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:49.807 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.777+0000 7fe4817fa700 1 -- 172.21.15.110:0/1617108862 <== osd.0 v2:172.21.15.110:6824/33394 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23999 (crc 0 0 0) 0x7fe47006ce70 con 0x7fe47006a370 2023-10-06T03:29:49.808 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.777+0000 7f077affd700 1 -- 172.21.15.110:0/1348442177 <== mon.2 v2:172.21.15.192:3301/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6809+0+0 (secure 0 0 0) 0x7f078c07f020 con 0x7f0794154d90 2023-10-06T03:29:49.808 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.777+0000 7f94f359e700 1 -- 172.21.15.110:0/2160324124 shutdown_connections 2023-10-06T03:29:49.808 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.777+0000 7f94f359e700 1 --2- 172.21.15.110:0/2160324124 >> v2:172.21.15.192:3300/0 conn(0x7f94ec053c00 0x7f94ec050260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:49.808 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.777+0000 7f94f359e700 1 --2- 172.21.15.110:0/2160324124 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f94ec0de360 0x7f94ec04fd20 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:49.808 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.777+0000 7f94f359e700 1 -- 172.21.15.110:0/2160324124 >> 172.21.15.110:0/2160324124 conn(0x7f94ec054c90 msgr2=0x7f94ec055090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:29:49.809 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.777+0000 7f077affd700 1 --2- 172.21.15.110:0/1348442177 >> [v2:172.21.15.192:6816/33683,v1:172.21.15.192:6817/33683] conn(0x7f078006a2a0 0x7f078006c6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:49.809 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.777+0000 7f077affd700 1 -- 172.21.15.110:0/1348442177 --> [v2:172.21.15.192:6816/33683,v1:172.21.15.192:6817/33683] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f078006cda0 con 0x7f078006a2a0 2023-10-06T03:29:49.809 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.777+0000 7f077affd700 1 -- 172.21.15.110:0/1348442177 <== mon.2 v2:172.21.15.192:3301/0 6 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f078c07f350 con 0x7f0794154d90 2023-10-06T03:29:49.809 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.777+0000 7f1794fe8700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f1790155b00 0x7f17900670a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:49.809 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.777+0000 7f0c77e65700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f0c701592d0 0x7f0c7006a4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:49.810 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.777+0000 7f1794fe8700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f17900675e0 0x7f1790155ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:49.810 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.777+0000 7f1794fe8700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1790156410 0x7f179006c8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:49.810 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.777+0000 7f1794fe8700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1790157820 con 0x7f17900675e0 2023-10-06T03:29:49.810 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.777+0000 7f1794fe8700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f17900adf40 con 0x7f1790155b00 2023-10-06T03:29:49.810 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.777+0000 7f1794fe8700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1790067a30 con 0x7f1790156410 2023-10-06T03:29:49.810 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.777+0000 7f0792ffd700 1 --2- 172.21.15.110:0/1348442177 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f07800605d0 0x7f0780062a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:49.811 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.777+0000 7f0c77e65700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f0c701596a0 0x7f0c700ca1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:49.811 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.777+0000 7f0c77e65700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c7006aee0 con 0x7f0c700a88d0 2023-10-06T03:29:49.811 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.777+0000 7f0c77e65700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0c7006aa10 con 0x7f0c701592d0 2023-10-06T03:29:49.811 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.777+0000 7f0c77e65700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0c700ae720 con 0x7f0c701596a0 2023-10-06T03:29:49.811 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.777+0000 7fe95759e700 1 -- 172.21.15.110:0/169604864 shutdown_connections 2023-10-06T03:29:49.812 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.777+0000 7f178ed9d700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1790156410 0x7f179006c8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:49.812 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.777+0000 7f178ed9d700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1790156410 0x7f179006c8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3301/0 says I am v2:172.21.15.110:44078/0 (socket says 172.21.15.110:44078) 2023-10-06T03:29:49.812 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.777+0000 7f178ed9d700 1 -- 172.21.15.110:0/549411836 learned_addr learned my addr 172.21.15.110:0/549411836 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:29:49.812 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.777+0000 7f6bad888700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f6ba81550c0 0x7f6ba80c7330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:49.812 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.777+0000 7f6bad888700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6ba806e7f0 0x7f6ba81595e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:49.812 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.777+0000 7f6bad888700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ba806cbd0 con 0x7f6ba8154cf0 2023-10-06T03:29:49.813 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.777+0000 7f6bad888700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ba80c6ea0 con 0x7f6ba81550c0 2023-10-06T03:29:49.813 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.777+0000 7f6bad888700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ba80c6570 con 0x7f6ba806e7f0 2023-10-06T03:29:49.813 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.781+0000 7f6bac886700 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f6ba8154cf0 0x7f6ba806c3c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:37320/0 (socket says 172.21.15.110:37320) 2023-10-06T03:29:49.813 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.781+0000 7f6ba7fff700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f6ba81550c0 0x7f6ba80c7330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:49.813 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.781+0000 7f6bac886700 1 -- 172.21.15.110:0/2003451983 learned_addr learned my addr 172.21.15.110:0/2003451983 (peer_addr_for_me v1:172.21.15.110:0/0) 2023-10-06T03:29:49.813 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.781+0000 7f6bad087700 1 --2- 172.21.15.110:0/2003451983 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6ba806e7f0 0x7f6ba81595e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:49.813 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.777+0000 7f6ea37fe700 1 -- 172.21.15.110:0/1520905109 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7f6eb4003030 con 0x7f6ebc0c1780 2023-10-06T03:29:49.814 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.777+0000 7f07937fe700 1 --2- 172.21.15.110:0/1348442177 >> [v2:172.21.15.192:6816/33683,v1:172.21.15.192:6817/33683] conn(0x7f078006a2a0 0x7f078006c6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:49.814 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.781+0000 7f0792ffd700 1 --2- 172.21.15.110:0/1348442177 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f07800605d0 0x7f0780062a80 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f07840058d0 tx=0x7f07840079d0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:29:49.814 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.781+0000 7f178dd9b700 1 --2- 172.21.15.110:0/549411836 >> v2:172.21.15.110:3300/0 conn(0x7f17900675e0 0x7f1790155ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:49.814 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.777+0000 7fe95759e700 1 -- 172.21.15.110:0/169604864 wait complete. 2023-10-06T03:29:49.814 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.777+0000 7f94f359e700 1 -- 172.21.15.110:0/2160324124 shutdown_connections 2023-10-06T03:29:49.815 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.777+0000 7f94f359e700 1 -- 172.21.15.110:0/2160324124 wait complete. 2023-10-06T03:29:49.815 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.781+0000 7f6ba7fff700 1 -- 172.21.15.110:0/2003451983 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6ba806e7f0 msgr2=0x7f6ba81595e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:29:49.815 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.781+0000 7f6ba7fff700 1 --2- 172.21.15.110:0/2003451983 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6ba806e7f0 0x7f6ba81595e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:49.815 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.781+0000 7f6ba7fff700 1 -- 172.21.15.110:0/2003451983 >> v1:172.21.15.110:6789/0 conn(0x7f6ba8154cf0 legacy=0x7f6ba806c3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:49.815 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.781+0000 7f6ba7fff700 1 -- 172.21.15.110:0/2003451983 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ba80c6850 con 0x7f6ba81550c0 2023-10-06T03:29:49.816 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.781+0000 7f07937fe700 1 --2- 172.21.15.110:0/1348442177 >> [v2:172.21.15.192:6816/33683,v1:172.21.15.192:6817/33683] conn(0x7f078006a2a0 0x7f078006c6d0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:29:49.816 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.781+0000 7f6bad087700 1 --2- 172.21.15.110:0/2003451983 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6ba806e7f0 0x7f6ba81595e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-06T03:29:49.816 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.777+0000 7f0c75c01700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f0c701592d0 0x7f0c7006a4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:49.816 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.777+0000 7f0c75c01700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f0c701592d0 0x7f0c7006a4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:60596/0 (socket says 172.21.15.110:60596) 2023-10-06T03:29:49.816 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.777+0000 7f0c75c01700 1 -- 172.21.15.110:0/1165402356 learned_addr learned my addr 172.21.15.110:0/1165402356 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:29:49.816 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.777+0000 7f0c75400700 1 --2- 172.21.15.110:0/1165402356 >> v2:172.21.15.192:3300/0 conn(0x7f0c701596a0 0x7f0c700ca1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:49.817 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.781+0000 7f0c75400700 1 -- 172.21.15.110:0/1165402356 >> v2:172.21.15.110:3300/0 conn(0x7f0c701592d0 msgr2=0x7f0c7006a4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:29:49.817 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.781+0000 7f0c75400700 1 --2- 172.21.15.110:0/1165402356 >> v2:172.21.15.110:3300/0 conn(0x7f0c701592d0 0x7f0c7006a4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:49.817 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.781+0000 7f0c75400700 1 -- 172.21.15.110:0/1165402356 >> v1:172.21.15.110:6789/0 conn(0x7f0c700a88d0 legacy=0x7f0c700c7650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:49.817 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.781+0000 7f0c75400700 1 -- 172.21.15.110:0/1165402356 --> v2:172.21.15.192:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c700c71c0 con 0x7f0c701596a0 2023-10-06T03:29:49.817 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.781+0000 7f178e59c700 1 --2- 172.21.15.110:0/549411836 >> v2:172.21.15.192:3300/0 conn(0x7f1790155b00 0x7f17900670a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:49.817 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.781+0000 7f6ea37fe700 1 --2- 172.21.15.110:0/1520905109 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f6ea8060670 0x7f6ea8062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:49.818 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.781+0000 7f6ba7fff700 1 --2- 172.21.15.110:0/2003451983 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6ba81550c0 0x7f6ba80c7330 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f6b9c0098e0 tx=0x7f6b9c0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=151c0a337353a4b1 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:29:49.818 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.781+0000 7f6ea37fe700 1 -- 172.21.15.110:0/1520905109 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6809+0+0 (secure 0 0 0) 0x7f6eb407edf0 con 0x7f6ebc0c1780 2023-10-06T03:29:49.818 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.781+0000 7f6ea37fe700 1 --2- 172.21.15.110:0/1520905109 >> [v2:172.21.15.110:6808/33395,v1:172.21.15.110:6809/33395] conn(0x7f6ea806a340 0x7f6ea806c770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:49.818 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.781+0000 7f6ea37fe700 1 -- 172.21.15.110:0/1520905109 --> [v2:172.21.15.110:6808/33395,v1:172.21.15.110:6809/33395] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6ea806ce40 con 0x7f6ea806a340 2023-10-06T03:29:49.818 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.781+0000 7f6ea37fe700 1 -- 172.21.15.110:0/1520905109 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6eb407f120 con 0x7f6ebc0c1780 2023-10-06T03:29:49.819 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.781+0000 7f178dd9b700 1 -- 172.21.15.110:0/549411836 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1790156410 msgr2=0x7f179006c8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:29:49.819 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.781+0000 7f178dd9b700 1 --2- 172.21.15.110:0/549411836 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1790156410 0x7f179006c8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:49.819 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.781+0000 7f178dd9b700 1 -- 172.21.15.110:0/549411836 >> v2:172.21.15.192:3300/0 conn(0x7f1790155b00 msgr2=0x7f17900670a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:29:49.819 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.781+0000 7f178dd9b700 1 --2- 172.21.15.110:0/549411836 >> v2:172.21.15.192:3300/0 conn(0x7f1790155b00 0x7f17900670a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:49.819 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.781+0000 7f178dd9b700 1 -- 172.21.15.110:0/549411836 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17900adc10 con 0x7f17900675e0 2023-10-06T03:29:49.820 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.781+0000 7f94f359e700 1 Processor -- start 2023-10-06T03:29:49.820 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.781+0000 7f94f359e700 1 -- start start 2023-10-06T03:29:49.820 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.781+0000 7f6ebb7fe700 1 --2- 172.21.15.110:0/1520905109 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f6ea8060670 0x7f6ea8062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:49.820 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.781+0000 7f6ebaffd700 1 --2- 172.21.15.110:0/1520905109 >> [v2:172.21.15.110:6808/33395,v1:172.21.15.110:6809/33395] conn(0x7f6ea806a340 0x7f6ea806c770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:49.820 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.781+0000 7f0c75400700 1 --2- 172.21.15.110:0/1165402356 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0c701596a0 0x7f0c700ca1e0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f0c70068cc0 tx=0x7f0c6c0091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c4d83d010dbc7fb1 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:29:49.821 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.781+0000 7f178dd9b700 1 --2- 172.21.15.110:0/549411836 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f17900675e0 0x7f1790155ed0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f179006a040 tx=0x7f17840091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=711966e6b1016e62 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:29:49.821 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.781+0000 7fe95759e700 1 Processor -- start 2023-10-06T03:29:49.821 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.781+0000 7f6ba77fe700 1 -- 172.21.15.110:0/2003451983 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b9c009e80 con 0x7f6ba81550c0 2023-10-06T03:29:49.821 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.781+0000 7f178d59a700 1 -- 172.21.15.110:0/549411836 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f178400d810 con 0x7f17900675e0 2023-10-06T03:29:49.822 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.781+0000 7f6ebb7fe700 1 --2- 172.21.15.110:0/1520905109 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f6ea8060670 0x7f6ea8062b20 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f6eac000c00 tx=0x7f6eac001bc0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:29:49.822 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.781+0000 7f077affd700 1 -- 172.21.15.110:0/1348442177 <== osd.4 v2:172.21.15.192:6816/33683 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23999 (crc 0 0 0) 0x7f078006cda0 con 0x7f078006a2a0 2023-10-06T03:29:49.822 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.781+0000 7f178d59a700 1 -- 172.21.15.110:0/549411836 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f178400d970 con 0x7f17900675e0 2023-10-06T03:29:49.822 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.781+0000 7f178d59a700 1 -- 172.21.15.110:0/549411836 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f178400dd10 con 0x7f17900675e0 2023-10-06T03:29:49.823 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.781+0000 7f6ebaffd700 1 --2- 172.21.15.110:0/1520905109 >> [v2:172.21.15.110:6808/33395,v1:172.21.15.110:6809/33395] conn(0x7f6ea806a340 0x7f6ea806c770 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:29:49.823 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.781+0000 7f6ba77fe700 1 -- 172.21.15.110:0/2003451983 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b9c00b860 con 0x7f6ba81550c0 2023-10-06T03:29:49.823 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.781+0000 7f6ba77fe700 1 -- 172.21.15.110:0/2003451983 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b9c00bbe0 con 0x7f6ba81550c0 2023-10-06T03:29:49.824 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.781+0000 7f94f359e700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f94ec053c00 0x7f94ec0dcf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:49.824 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.781+0000 7f94f359e700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f94ec0ddf90 0x7f94ec0cb470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:49.824 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.781+0000 7f94f359e700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f94ec0de360 0x7f94ec04d7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:49.824 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.781+0000 7f94f359e700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94ec0dd490 con 0x7f94ec053c00 2023-10-06T03:29:49.824 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.781+0000 7f94f359e700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94ec04eca0 con 0x7f94ec0ddf90 2023-10-06T03:29:49.824 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.781+0000 7f94f359e700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94ec04c0e0 con 0x7f94ec0de360 2023-10-06T03:29:49.824 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.781+0000 7f94f2d9d700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f94ec0de360 0x7f94ec04d7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:49.824 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.781+0000 7f94f1d9b700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f94ec0ddf90 0x7f94ec0cb470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:49.825 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.781+0000 7f6bad888700 1 -- 172.21.15.110:0/2003451983 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6ba81550c0 msgr2=0x7f6ba80c7330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:49.825 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.781+0000 7f6bad888700 1 --2- 172.21.15.110:0/2003451983 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6ba81550c0 0x7f6ba80c7330 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f6b9c0098e0 tx=0x7f6b9c0091a0 comp rx=0 tx=0).stop 2023-10-06T03:29:49.825 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.781+0000 7f1794fe8700 1 -- 172.21.15.110:0/549411836 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f17900675e0 msgr2=0x7f1790155ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:49.825 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.781+0000 7f1794fe8700 1 --2- 172.21.15.110:0/549411836 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f17900675e0 0x7f1790155ed0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f179006a040 tx=0x7f17840091a0 comp rx=0 tx=0).stop 2023-10-06T03:29:49.825 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.781+0000 7f94f1d9b700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f94ec0ddf90 0x7f94ec0cb470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.110:50764/0 (socket says 172.21.15.110:50764) 2023-10-06T03:29:49.826 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.781+0000 7f94f1d9b700 1 -- 172.21.15.110:0/3208140742 learned_addr learned my addr 172.21.15.110:0/3208140742 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:29:49.826 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.781+0000 7f1794fe8700 1 -- 172.21.15.110:0/549411836 shutdown_connections 2023-10-06T03:29:49.826 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.781+0000 7f1794fe8700 1 --2- 172.21.15.110:0/549411836 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1790156410 0x7f179006c8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:49.826 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.781+0000 7f1794fe8700 1 --2- 172.21.15.110:0/549411836 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f17900675e0 0x7f1790155ed0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:49.826 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.781+0000 7f1794fe8700 1 --2- 172.21.15.110:0/549411836 >> v2:172.21.15.192:3300/0 conn(0x7f1790155b00 0x7f17900670a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:49.826 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.781+0000 7f1794fe8700 1 -- 172.21.15.110:0/549411836 >> 172.21.15.110:0/549411836 conn(0x7f17900ccd00 msgr2=0x7f17900cd100 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:29:49.827 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.781+0000 7f94f259c700 1 --2- 172.21.15.110:0/3208140742 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f94ec053c00 0x7f94ec0dcf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:49.827 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.781+0000 7f0c74bff700 1 -- 172.21.15.110:0/1165402356 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c6c009de0 con 0x7f0c701596a0 2023-10-06T03:29:49.827 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.781+0000 7f6bad888700 1 -- 172.21.15.110:0/2003451983 shutdown_connections 2023-10-06T03:29:49.827 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.781+0000 7f6bad888700 1 --2- 172.21.15.110:0/2003451983 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6ba806e7f0 0x7f6ba81595e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:49.827 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.781+0000 7f6bad888700 1 --2- 172.21.15.110:0/2003451983 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6ba81550c0 0x7f6ba80c7330 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:49.828 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.781+0000 7f6bad888700 1 -- 172.21.15.110:0/2003451983 >> 172.21.15.110:0/2003451983 conn(0x7f6ba806dce0 msgr2=0x7f6ba806e0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:29:49.828 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.781+0000 7f0c74bff700 1 -- 172.21.15.110:0/1165402356 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c6c0128e0 con 0x7f0c701596a0 2023-10-06T03:29:49.828 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.781+0000 7f6bad888700 1 -- 172.21.15.110:0/2003451983 shutdown_connections 2023-10-06T03:29:49.828 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.781+0000 7f94f259c700 1 -- 172.21.15.110:0/3208140742 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f94ec0de360 msgr2=0x7f94ec04d7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:49.829 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.781+0000 7f94f259c700 1 --2- 172.21.15.110:0/3208140742 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f94ec0de360 0x7f94ec04d7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:49.829 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.781+0000 7f94f259c700 1 -- 172.21.15.110:0/3208140742 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f94ec0ddf90 msgr2=0x7f94ec0cb470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:49.829 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.781+0000 7f94f259c700 1 --2- 172.21.15.110:0/3208140742 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f94ec0ddf90 0x7f94ec0cb470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:49.829 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.781+0000 7f94f259c700 1 -- 172.21.15.110:0/3208140742 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94e80094f0 con 0x7f94ec053c00 2023-10-06T03:29:49.829 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.781+0000 7f94f1d9b700 1 --2- 172.21.15.110:0/3208140742 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f94ec0ddf90 0x7f94ec0cb470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-06T03:29:49.829 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.781+0000 7f6ea37fe700 1 -- 172.21.15.110:0/1520905109 <== osd.3 v2:172.21.15.110:6808/33395 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23999 (crc 0 0 0) 0x7f6ea806ce40 con 0x7f6ea806a340 2023-10-06T03:29:49.830 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.781+0000 7f94f259c700 1 --2- 172.21.15.110:0/3208140742 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f94ec053c00 0x7f94ec0dcf50 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f94f4065d30 tx=0x7f94f4065170 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:29:49.830 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.785+0000 7f1794fe8700 1 -- 172.21.15.110:0/549411836 shutdown_connections 2023-10-06T03:29:49.830 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.781+0000 7f0c74bff700 1 -- 172.21.15.110:0/1165402356 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c6c012ca0 con 0x7f0c701596a0 2023-10-06T03:29:49.830 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.781+0000 7f6bad888700 1 -- 172.21.15.110:0/2003451983 wait complete. 2023-10-06T03:29:49.831 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.785+0000 7f6bad888700 1 Processor -- start 2023-10-06T03:29:49.831 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.785+0000 7f0c77e65700 1 -- 172.21.15.110:0/1165402356 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0c701596a0 msgr2=0x7f0c700ca1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:49.831 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.785+0000 7f0c77e65700 1 --2- 172.21.15.110:0/1165402356 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0c701596a0 0x7f0c700ca1e0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f0c70068cc0 tx=0x7f0c6c0091a0 comp rx=0 tx=0).stop 2023-10-06T03:29:49.831 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.785+0000 7f6bad888700 1 -- start start 2023-10-06T03:29:49.832 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.785+0000 7f0c77e65700 1 -- 172.21.15.110:0/1165402356 shutdown_connections 2023-10-06T03:29:49.832 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.785+0000 7f0c77e65700 1 --2- 172.21.15.110:0/1165402356 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0c701596a0 0x7f0c700ca1e0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:49.832 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.785+0000 7f0c77e65700 1 --2- 172.21.15.110:0/1165402356 >> v2:172.21.15.110:3300/0 conn(0x7f0c701592d0 0x7f0c7006a4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:49.832 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.785+0000 7f0c77e65700 1 -- 172.21.15.110:0/1165402356 >> 172.21.15.110:0/1165402356 conn(0x7f0c700cc6d0 msgr2=0x7f0c700c8670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:29:49.833 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.785+0000 7f6bad888700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6ba806e7f0 0x7f6ba814af10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:49.833 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.785+0000 7f6bad888700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6ba8154cf0 0x7f6ba80b0d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:49.833 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.785+0000 7f6bad888700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6ba81550c0 0x7f6ba80b1260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:49.833 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.785+0000 7f6bad888700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ba8152250 con 0x7f6ba806e7f0 2023-10-06T03:29:49.833 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.785+0000 7f6bad888700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ba8155f00 con 0x7f6ba8154cf0 2023-10-06T03:29:49.833 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.785+0000 7f6bad888700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ba8156140 con 0x7f6ba81550c0 2023-10-06T03:29:49.833 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.785+0000 7f6bac886700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6ba806e7f0 0x7f6ba814af10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:49.834 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.785+0000 7f0c77e65700 1 -- 172.21.15.110:0/1165402356 shutdown_connections 2023-10-06T03:29:49.834 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.785+0000 7f6bac886700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6ba806e7f0 0x7f6ba814af10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:60654/0 (socket says 172.21.15.110:60654) 2023-10-06T03:29:49.834 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.785+0000 7f0c77e65700 1 -- 172.21.15.110:0/1165402356 wait complete. 2023-10-06T03:29:49.835 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.785+0000 7f6bac886700 1 -- 172.21.15.110:0/1204609507 learned_addr learned my addr 172.21.15.110:0/1204609507 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:29:49.835 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.785+0000 7f6ba7fff700 1 --2- 172.21.15.110:0/1204609507 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6ba8154cf0 0x7f6ba80b0d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:49.835 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.785+0000 7f6bad087700 1 --2- 172.21.15.110:0/1204609507 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6ba81550c0 0x7f6ba80b1260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:49.836 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.785+0000 7f6bac886700 1 -- 172.21.15.110:0/1204609507 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6ba81550c0 msgr2=0x7f6ba80b1260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:49.836 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.785+0000 7f6bac886700 1 --2- 172.21.15.110:0/1204609507 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6ba81550c0 0x7f6ba80b1260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:49.836 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.785+0000 7f6bac886700 1 -- 172.21.15.110:0/1204609507 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6ba8154cf0 msgr2=0x7f6ba80b0d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:49.836 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.785+0000 7f6bac886700 1 --2- 172.21.15.110:0/1204609507 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6ba8154cf0 0x7f6ba80b0d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:49.836 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.785+0000 7f6bac886700 1 -- 172.21.15.110:0/1204609507 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b9c009420 con 0x7f6ba806e7f0 2023-10-06T03:29:49.836 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.785+0000 7f6bad087700 1 --2- 172.21.15.110:0/1204609507 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6ba81550c0 0x7f6ba80b1260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-06T03:29:49.836 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.785+0000 7f6bac886700 1 --2- 172.21.15.110:0/1204609507 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6ba806e7f0 0x7f6ba814af10 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f6ba000cd30 tx=0x7f6ba0001ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:29:49.836 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.781+0000 7fe95759e700 1 -- start start 2023-10-06T03:29:49.836 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.781+0000 7fe95759e700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe958154cf0 0x7fe95816a160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:49.837 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.781+0000 7fe95759e700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe9581550c0 0x7fe9580af110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:49.837 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.781+0000 7fe95759e700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe9581595e0 0x7fe9580b0cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:49.837 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.781+0000 7fe95759e700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9580ac4a0 con 0x7fe958154cf0 2023-10-06T03:29:49.837 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.781+0000 7fe95759e700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9580c5770 con 0x7fe9581595e0 2023-10-06T03:29:49.837 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.781+0000 7fe95759e700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9580c5a50 con 0x7fe9581550c0 2023-10-06T03:29:49.837 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.781+0000 7fe95659c700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe958154cf0 0x7fe95816a160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:49.837 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.781+0000 7fe95659c700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe958154cf0 0x7fe95816a160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:60638/0 (socket says 172.21.15.110:60638) 2023-10-06T03:29:49.837 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.781+0000 7fe95659c700 1 -- 172.21.15.110:0/184029913 learned_addr learned my addr 172.21.15.110:0/184029913 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:29:49.837 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.781+0000 7fe955d9b700 1 --2- 172.21.15.110:0/184029913 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe9581550c0 0x7fe9580af110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:49.837 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.785+0000 7fe956d9d700 1 --2- 172.21.15.110:0/184029913 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe9581595e0 0x7fe9580b0cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:49.837 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.785+0000 7fe95659c700 1 -- 172.21.15.110:0/184029913 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe9581550c0 msgr2=0x7fe9580af110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:49.838 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.785+0000 7fe95659c700 1 --2- 172.21.15.110:0/184029913 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe9581550c0 0x7fe9580af110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:49.838 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.785+0000 7fe95659c700 1 -- 172.21.15.110:0/184029913 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe9581595e0 msgr2=0x7fe9580b0cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:49.838 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.785+0000 7fe95659c700 1 --2- 172.21.15.110:0/184029913 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe9581595e0 0x7fe9580b0cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:49.838 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.785+0000 7fe95659c700 1 -- 172.21.15.110:0/184029913 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9580ac130 con 0x7fe958154cf0 2023-10-06T03:29:49.838 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.785+0000 7fe95659c700 1 --2- 172.21.15.110:0/184029913 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe958154cf0 0x7fe95816a160 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fe94c00c5d0 tx=0x7fe94c007f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:29:49.839 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.789+0000 7f94e27fc700 1 -- 172.21.15.110:0/3208140742 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94f406b040 con 0x7f94ec053c00 2023-10-06T03:29:49.839 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.789+0000 7f94e27fc700 1 -- 172.21.15.110:0/3208140742 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94f406fd40 con 0x7f94ec053c00 2023-10-06T03:29:49.839 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.789+0000 7f94e27fc700 1 -- 172.21.15.110:0/3208140742 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94f405dd50 con 0x7f94ec053c00 2023-10-06T03:29:49.839 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.789+0000 7f94f359e700 1 -- 172.21.15.110:0/3208140742 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94ec04e650 con 0x7f94ec053c00 2023-10-06T03:29:49.839 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.789+0000 7f94f359e700 1 -- 172.21.15.110:0/3208140742 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94ec037d00 con 0x7f94ec053c00 2023-10-06T03:29:49.839 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.789+0000 7f0c77e65700 1 Processor -- start 2023-10-06T03:29:49.840 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.789+0000 7fe9467fc700 1 -- 172.21.15.110:0/184029913 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe94c0335a0 con 0x7fe958154cf0 2023-10-06T03:29:49.840 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.789+0000 7fe9467fc700 1 -- 172.21.15.110:0/184029913 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe94c033720 con 0x7fe958154cf0 2023-10-06T03:29:49.840 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.789+0000 7fe9467fc700 1 -- 172.21.15.110:0/184029913 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe94c033ae0 con 0x7fe958154cf0 2023-10-06T03:29:49.840 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.789+0000 7fe95759e700 1 -- 172.21.15.110:0/184029913 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9580c5d30 con 0x7fe958154cf0 2023-10-06T03:29:49.840 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.789+0000 7f1794fe8700 1 -- 172.21.15.110:0/549411836 wait complete. 2023-10-06T03:29:49.841 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.789+0000 7f1794fe8700 1 Processor -- start 2023-10-06T03:29:49.841 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.789+0000 7fe95759e700 1 -- 172.21.15.110:0/184029913 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe958150f70 con 0x7fe958154cf0 2023-10-06T03:29:49.841 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.789+0000 7fe95759e700 1 -- 172.21.15.110:0/184029913 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fe958070ff0 con 0x7fe958154cf0 2023-10-06T03:29:49.841 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.789+0000 7f94e27fc700 1 -- 172.21.15.110:0/3208140742 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7f94f4064160 con 0x7f94ec053c00 2023-10-06T03:29:49.842 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.789+0000 7f1794fe8700 1 -- start start 2023-10-06T03:29:49.842 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.789+0000 7f1794fe8700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f17900675e0 0x7f179016df40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:49.842 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.789+0000 7f1794fe8700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1790155b00 0x7f179016e870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:49.842 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.789+0000 7f1794fe8700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1790156410 0x7f17900b1960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:49.842 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.789+0000 7f1794fe8700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17900b0720 con 0x7f17900675e0 2023-10-06T03:29:49.842 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.789+0000 7f1794fe8700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17900afb50 con 0x7f1790156410 2023-10-06T03:29:49.842 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.789+0000 7f1794fe8700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17900afdf0 con 0x7f1790155b00 2023-10-06T03:29:49.843 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.789+0000 7f0c77e65700 1 -- start start 2023-10-06T03:29:49.843 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.789+0000 7f0c77e65700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0c700a88d0 0x7f0c700c4a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:49.843 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.789+0000 7f0c77e65700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0c701592d0 0x7f0c700c4f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:49.843 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.789+0000 7f6ba4ff9700 1 -- 172.21.15.110:0/1204609507 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ba000ce30 con 0x7f6ba806e7f0 2023-10-06T03:29:49.843 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.789+0000 7f0c77e65700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0c701596a0 0x7f0c70157f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:49.844 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.789+0000 7f0c77e65700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c700c2c10 con 0x7f0c701596a0 2023-10-06T03:29:49.844 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.789+0000 7f0c77e65700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c700c3a70 con 0x7f0c700a88d0 2023-10-06T03:29:49.844 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.789+0000 7f0c77e65700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c700c3f00 con 0x7f0c701592d0 2023-10-06T03:29:49.844 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.789+0000 7f6ba4ff9700 1 -- 172.21.15.110:0/1204609507 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ba0004bb0 con 0x7f6ba806e7f0 2023-10-06T03:29:49.844 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.789+0000 7f6ba4ff9700 1 -- 172.21.15.110:0/1204609507 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ba000ab10 con 0x7f6ba806e7f0 2023-10-06T03:29:49.845 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.789+0000 7f178e59c700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f17900675e0 0x7f179016df40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:49.845 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.789+0000 7f178e59c700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f17900675e0 0x7f179016df40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:60660/0 (socket says 172.21.15.110:60660) 2023-10-06T03:29:49.845 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.789+0000 7f178e59c700 1 -- 172.21.15.110:0/1675188097 learned_addr learned my addr 172.21.15.110:0/1675188097 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:29:49.845 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.789+0000 7f178dd9b700 1 --2- 172.21.15.110:0/1675188097 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1790155b00 0x7f179016e870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:49.845 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.789+0000 7f178ed9d700 1 --2- 172.21.15.110:0/1675188097 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1790156410 0x7f17900b1960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:49.845 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.789+0000 7f0c76402700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0c701596a0 0x7f0c70157f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:49.845 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.789+0000 7f0c75400700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0c701592d0 0x7f0c700c4f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:49.846 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.789+0000 7f0c76402700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0c701596a0 0x7f0c70157f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:60666/0 (socket says 172.21.15.110:60666) 2023-10-06T03:29:49.846 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.789+0000 7f0c76402700 1 -- 172.21.15.110:0/3012905740 learned_addr learned my addr 172.21.15.110:0/3012905740 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:29:49.846 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.789+0000 7f6bad888700 1 -- 172.21.15.110:0/1204609507 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ba8156420 con 0x7f6ba806e7f0 2023-10-06T03:29:49.846 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.789+0000 7f6bad888700 1 -- 172.21.15.110:0/1204609507 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ba81569e0 con 0x7f6ba806e7f0 2023-10-06T03:29:49.846 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.793+0000 7f0c75400700 1 -- 172.21.15.110:0/3012905740 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0c700a88d0 msgr2=0x7f0c700c4a10 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2023-10-06T03:29:49.846 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.793+0000 7f0c75400700 1 --2- 172.21.15.110:0/3012905740 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0c700a88d0 0x7f0c700c4a10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:49.847 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.793+0000 7f0c75400700 1 -- 172.21.15.110:0/3012905740 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0c701596a0 msgr2=0x7f0c70157f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:49.847 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.793+0000 7f0c75400700 1 --2- 172.21.15.110:0/3012905740 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0c701596a0 0x7f0c70157f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:49.847 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.793+0000 7f0c75400700 1 -- 172.21.15.110:0/3012905740 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c6c013a10 con 0x7f0c701592d0 2023-10-06T03:29:49.847 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.793+0000 7f0c75400700 1 --2- 172.21.15.110:0/3012905740 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0c701592d0 0x7f0c700c4f50 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f0c70068cc0 tx=0x7f0c6c01a000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:29:49.847 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.789+0000 7f94e27fc700 1 --2- 172.21.15.110:0/3208140742 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f94d4060620 0x7f94d4062ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:49.847 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.793+0000 7f94f1d9b700 1 --2- 172.21.15.110:0/3208140742 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f94d4060620 0x7f94d4062ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:49.847 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.793+0000 7f94e27fc700 1 -- 172.21.15.110:0/3208140742 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6809+0+0 (secure 0 0 0) 0x7f94f40f2c40 con 0x7f94ec053c00 2023-10-06T03:29:49.848 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.793+0000 7f178dd9b700 1 -- 172.21.15.110:0/1675188097 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1790156410 msgr2=0x7f17900b1960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:49.848 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.793+0000 7f178dd9b700 1 --2- 172.21.15.110:0/1675188097 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1790156410 0x7f17900b1960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:49.848 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.793+0000 7f178dd9b700 1 -- 172.21.15.110:0/1675188097 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f17900675e0 msgr2=0x7f179016df40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:49.848 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.793+0000 7f6bad888700 1 -- 172.21.15.110:0/1204609507 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f6ba8070ff0 con 0x7f6ba806e7f0 2023-10-06T03:29:49.848 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.793+0000 7f94f1d9b700 1 --2- 172.21.15.110:0/3208140742 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f94d4060620 0x7f94d4062ad0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f94ec04b1e0 tx=0x7f94e8016000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:29:49.849 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.793+0000 7f0c65ffb700 1 -- 172.21.15.110:0/3012905740 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c6c0123e0 con 0x7f0c701592d0 2023-10-06T03:29:49.849 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.793+0000 7f0c65ffb700 1 -- 172.21.15.110:0/3012905740 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c6c012560 con 0x7f0c701592d0 2023-10-06T03:29:49.849 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.793+0000 7f0c65ffb700 1 -- 172.21.15.110:0/3012905740 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c6c01b420 con 0x7f0c701592d0 2023-10-06T03:29:49.849 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.793+0000 7f178dd9b700 1 --2- 172.21.15.110:0/1675188097 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f17900675e0 0x7f179016df40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:49.849 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.793+0000 7f178dd9b700 1 -- 172.21.15.110:0/1675188097 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1784009420 con 0x7f1790155b00 2023-10-06T03:29:49.850 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.793+0000 7fe9467fc700 1 -- 172.21.15.110:0/184029913 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7fe94c03e460 con 0x7fe958154cf0 2023-10-06T03:29:49.850 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.793+0000 7f0c77e65700 1 -- 172.21.15.110:0/3012905740 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c70155430 con 0x7f0c701592d0 2023-10-06T03:29:49.850 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.793+0000 7f0c77e65700 1 -- 172.21.15.110:0/3012905740 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c700c3140 con 0x7f0c701592d0 2023-10-06T03:29:49.850 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.793+0000 7fe9467fc700 1 --2- 172.21.15.110:0/184029913 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fe938060620 0x7fe938062ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:49.850 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.793+0000 7f178ed9d700 1 --2- 172.21.15.110:0/1675188097 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1790156410 0x7f17900b1960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-06T03:29:49.851 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.793+0000 7f178e59c700 1 --2- 172.21.15.110:0/1675188097 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f17900675e0 0x7f179016df40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-06T03:29:49.851 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.793+0000 7f178dd9b700 1 --2- 172.21.15.110:0/1675188097 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1790155b00 0x7f179016e870 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f1784018000 tx=0x7f1784019000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:29:49.851 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.793+0000 7f6ba4ff9700 1 -- 172.21.15.110:0/1204609507 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7f6ba00070f0 con 0x7f6ba806e7f0 2023-10-06T03:29:49.851 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.793+0000 7fe955d9b700 1 --2- 172.21.15.110:0/184029913 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fe938060620 0x7fe938062ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:49.851 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.793+0000 7fe9467fc700 1 -- 172.21.15.110:0/184029913 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6809+0+0 (secure 0 0 0) 0x7fe94c09f3b0 con 0x7fe958154cf0 2023-10-06T03:29:49.852 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.793+0000 7f177e7fc700 1 -- 172.21.15.110:0/1675188097 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f178400c070 con 0x7f1790155b00 2023-10-06T03:29:49.852 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.793+0000 7f177e7fc700 1 -- 172.21.15.110:0/1675188097 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f178400d160 con 0x7f1790155b00 2023-10-06T03:29:49.852 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.793+0000 7f177e7fc700 1 -- 172.21.15.110:0/1675188097 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f178400d810 con 0x7f1790155b00 2023-10-06T03:29:49.852 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.793+0000 7fe955d9b700 1 --2- 172.21.15.110:0/184029913 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fe938060620 0x7fe938062ad0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fe9580c92c0 tx=0x7fe95000b040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:29:49.853 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.793+0000 7f1794fe8700 1 -- 172.21.15.110:0/1675188097 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17900b00d0 con 0x7f1790155b00 2023-10-06T03:29:49.853 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.793+0000 7f1794fe8700 1 -- 172.21.15.110:0/1675188097 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f179016e480 con 0x7f1790155b00 2023-10-06T03:29:49.853 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.793+0000 7fe9467fc700 1 --2- 172.21.15.110:0/184029913 >> [v2:172.21.15.192:6824/33686,v1:172.21.15.192:6825/33686] conn(0x7fe93806a2f0 0x7fe93806c720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:49.853 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.793+0000 7fe9467fc700 1 -- 172.21.15.110:0/184029913 --> [v2:172.21.15.192:6824/33686,v1:172.21.15.192:6825/33686] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe93806cdf0 con 0x7fe93806a2f0 2023-10-06T03:29:49.853 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.793+0000 7fe9467fc700 1 -- 172.21.15.110:0/184029913 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe94c09f780 con 0x7fe958154cf0 2023-10-06T03:29:49.854 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.793+0000 7f6ba4ff9700 1 --2- 172.21.15.110:0/1204609507 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f6b94060620 0x7f6b94062ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:49.854 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.793+0000 7f6ba4ff9700 1 -- 172.21.15.110:0/1204609507 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6809+0+0 (secure 0 0 0) 0x7f6ba007d090 con 0x7f6ba806e7f0 2023-10-06T03:29:49.854 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.793+0000 7fe956d9d700 1 --2- 172.21.15.110:0/184029913 >> [v2:172.21.15.192:6824/33686,v1:172.21.15.192:6825/33686] conn(0x7fe93806a2f0 0x7fe93806c720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:49.855 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.793+0000 7f6ba4ff9700 1 --2- 172.21.15.110:0/1204609507 >> [v2:172.21.15.110:6800/33396,v1:172.21.15.110:6801/33396] conn(0x7f6b9406a2f0 0x7f6b9406c720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:49.855 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.797+0000 7f6ba4ff9700 1 -- 172.21.15.110:0/1204609507 --> [v2:172.21.15.110:6800/33396,v1:172.21.15.110:6801/33396] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6b9406cdf0 con 0x7f6b9406a2f0 2023-10-06T03:29:49.855 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.797+0000 7f6ba4ff9700 1 -- 172.21.15.110:0/1204609507 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6ba007d4b0 con 0x7f6ba806e7f0 2023-10-06T03:29:49.855 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.797+0000 7f6bad087700 1 --2- 172.21.15.110:0/1204609507 >> [v2:172.21.15.110:6800/33396,v1:172.21.15.110:6801/33396] conn(0x7f6b9406a2f0 0x7f6b9406c720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:49.857 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.797+0000 7f94f359e700 1 --2- 172.21.15.110:0/3208140742 >> [v2:172.21.15.192:6808/33685,v1:172.21.15.192:6809/33685] conn(0x7f94ec0cb9b0 0x7f94ec003140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:49.857 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.797+0000 7f94f359e700 1 -- 172.21.15.110:0/3208140742 --> [v2:172.21.15.192:6808/33685,v1:172.21.15.192:6809/33685] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f94ec054480 con 0x7f94ec0cb9b0 2023-10-06T03:29:49.857 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.797+0000 7f0c65ffb700 1 -- 172.21.15.110:0/3012905740 <== mon.2 v2:172.21.15.192:3301/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7f0c6c01b850 con 0x7f0c701592d0 2023-10-06T03:29:49.858 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.797+0000 7f94f2d9d700 1 --2- 172.21.15.110:0/3208140742 >> [v2:172.21.15.192:6808/33685,v1:172.21.15.192:6809/33685] conn(0x7f94ec0cb9b0 0x7f94ec003140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:49.858 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.797+0000 7f6ba7fff700 1 --2- 172.21.15.110:0/1204609507 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f6b94060620 0x7f6b94062ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:49.858 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.797+0000 7f6bad087700 1 --2- 172.21.15.110:0/1204609507 >> [v2:172.21.15.110:6800/33396,v1:172.21.15.110:6801/33396] conn(0x7f6b9406a2f0 0x7f6b9406c720 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:29:49.858 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.797+0000 7f6ba7fff700 1 --2- 172.21.15.110:0/1204609507 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f6b94060620 0x7f6b94062ad0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f6ba81554f0 tx=0x7f6b9c017000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:29:49.858 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.797+0000 7fe956d9d700 1 --2- 172.21.15.110:0/184029913 >> [v2:172.21.15.192:6824/33686,v1:172.21.15.192:6825/33686] conn(0x7fe93806a2f0 0x7fe93806c720 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:29:49.859 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.797+0000 7f0c65ffb700 1 --2- 172.21.15.110:0/3012905740 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f0c54060620 0x7f0c54062ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:49.859 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.797+0000 7f0c65ffb700 1 -- 172.21.15.110:0/3012905740 <== mon.2 v2:172.21.15.192:3301/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6809+0+0 (secure 0 0 0) 0x7f0c6c07e2f0 con 0x7f0c701592d0 2023-10-06T03:29:49.859 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.797+0000 7f0c75c01700 1 --2- 172.21.15.110:0/3012905740 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f0c54060620 0x7f0c54062ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:49.859 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.797+0000 7f0c75c01700 1 --2- 172.21.15.110:0/3012905740 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f0c54060620 0x7f0c54062ad0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f0c6800e2f0 tx=0x7f0c6800b040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:29:49.859 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.797+0000 7f177e7fc700 1 -- 172.21.15.110:0/1675188097 <== mon.2 v2:172.21.15.192:3301/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7f1784010680 con 0x7f1790155b00 2023-10-06T03:29:49.860 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.797+0000 7fe9467fc700 1 -- 172.21.15.110:0/184029913 <== osd.6 v2:172.21.15.192:6824/33686 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23999 (crc 0 0 0) 0x7fe93806cdf0 con 0x7fe93806a2f0 2023-10-06T03:29:49.860 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.797+0000 7f177e7fc700 1 --2- 172.21.15.110:0/1675188097 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f17780606c0 0x7f1778062b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:49.860 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.797+0000 7f177e7fc700 1 -- 172.21.15.110:0/1675188097 <== mon.2 v2:172.21.15.192:3301/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6809+0+0 (secure 0 0 0) 0x7f1784014070 con 0x7f1790155b00 2023-10-06T03:29:49.860 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.797+0000 7f178e59c700 1 --2- 172.21.15.110:0/1675188097 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f17780606c0 0x7f1778062b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:49.860 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.797+0000 7f6ba4ff9700 1 -- 172.21.15.110:0/1204609507 <== osd.1 v2:172.21.15.110:6800/33396 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23999 (crc 0 0 0) 0x7f6b9406cdf0 con 0x7f6b9406a2f0 2023-10-06T03:29:49.861 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.797+0000 7f178e59c700 1 --2- 172.21.15.110:0/1675188097 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f17780606c0 0x7f1778062b70 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f1780006fd0 tx=0x7f1780008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:29:49.861 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.805+0000 7f0c5b7fe700 1 --2- 172.21.15.110:0/3012905740 >> [v2:172.21.15.110:6816/33397,v1:172.21.15.110:6817/33397] conn(0x7f0c701584b0 0x7f0c700c3330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:49.861 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.805+0000 7f0c5b7fe700 1 -- 172.21.15.110:0/3012905740 --> [v2:172.21.15.110:6816/33397,v1:172.21.15.110:6817/33397] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0c700c3870 con 0x7f0c701584b0 2023-10-06T03:29:49.861 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.797+0000 7f94f2d9d700 1 --2- 172.21.15.110:0/3208140742 >> [v2:172.21.15.192:6808/33685,v1:172.21.15.192:6809/33685] conn(0x7f94ec0cb9b0 0x7f94ec003140 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:29:49.862 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.797+0000 7f94e27fc700 1 -- 172.21.15.110:0/3208140742 <== osd.7 v2:172.21.15.192:6808/33685 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23999 (crc 0 0 0) 0x7f94ec054480 con 0x7f94ec0cb9b0 2023-10-06T03:29:49.862 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.805+0000 7fe493a00700 1 -- 172.21.15.110:0/1617108862 --> [v2:172.21.15.110:6824/33394,v1:172.21.15.110:6825/33394] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fe46c002d30 con 0x7fe47006a370 2023-10-06T03:29:49.862 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.805+0000 7f0c76402700 1 --2- 172.21.15.110:0/3012905740 >> [v2:172.21.15.110:6816/33397,v1:172.21.15.110:6817/33397] conn(0x7f0c701584b0 0x7f0c700c3330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:49.862 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.805+0000 7f1794fe8700 1 --2- 172.21.15.110:0/1675188097 >> [v2:172.21.15.192:6800/33684,v1:172.21.15.192:6801/33684] conn(0x7f17740015f0 0x7f1774003aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:49.862 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.805+0000 7f1794fe8700 1 -- 172.21.15.110:0/1675188097 --> [v2:172.21.15.192:6800/33684,v1:172.21.15.192:6801/33684] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1774006b90 con 0x7f17740015f0 2023-10-06T03:29:49.863 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.805+0000 7fe4817fa700 1 -- 172.21.15.110:0/1617108862 <== osd.0 v2:172.21.15.110:6824/33394 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fe46c002d30 con 0x7fe47006a370 2023-10-06T03:29:49.863 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.805+0000 7f178ed9d700 1 --2- 172.21.15.110:0/1675188097 >> [v2:172.21.15.192:6800/33684,v1:172.21.15.192:6801/33684] conn(0x7f17740015f0 0x7f1774003aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:49.863 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.805+0000 7f0c76402700 1 --2- 172.21.15.110:0/3012905740 >> [v2:172.21.15.110:6816/33397,v1:172.21.15.110:6817/33397] conn(0x7f0c701584b0 0x7f0c700c3330 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:29:49.864 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.805+0000 7f178ed9d700 1 --2- 172.21.15.110:0/1675188097 >> [v2:172.21.15.192:6800/33684,v1:172.21.15.192:6801/33684] conn(0x7f17740015f0 0x7f1774003aa0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:29:49.864 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.805+0000 7fe493a00700 1 -- 172.21.15.110:0/1617108862 >> [v2:172.21.15.110:6824/33394,v1:172.21.15.110:6825/33394] conn(0x7fe47006a370 msgr2=0x7fe47006c7a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:49.864 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.805+0000 7fe493a00700 1 --2- 172.21.15.110:0/1617108862 >> [v2:172.21.15.110:6824/33394,v1:172.21.15.110:6825/33394] conn(0x7fe47006a370 0x7fe47006c7a0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:49.864 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.805+0000 7f07993da700 1 -- 172.21.15.110:0/1348442177 --> [v2:172.21.15.192:6816/33683,v1:172.21.15.192:6817/33683] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f0774002d30 con 0x7f078006a2a0 2023-10-06T03:29:49.865 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.805+0000 7f0c65ffb700 1 -- 172.21.15.110:0/3012905740 <== osd.2 v2:172.21.15.110:6816/33397 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23999 (crc 0 0 0) 0x7f0c700c3870 con 0x7f0c701584b0 2023-10-06T03:29:49.865 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.805+0000 7fe493a00700 1 -- 172.21.15.110:0/1617108862 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fe4700606a0 msgr2=0x7fe470062b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:49.865 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.805+0000 7fe493a00700 1 --2- 172.21.15.110:0/1617108862 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fe4700606a0 0x7fe470062b50 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fe488009ca0 tx=0x7fe488008040 comp rx=0 tx=0).stop 2023-10-06T03:29:49.866 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.805+0000 7fe493a00700 1 -- 172.21.15.110:0/1617108862 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe48c0c75e0 msgr2=0x7fe48c0c4c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:49.866 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.805+0000 7fe493a00700 1 --2- 172.21.15.110:0/1617108862 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe48c0c75e0 0x7fe48c0c4c50 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fe48400b6c0 tx=0x7fe484017000 comp rx=0 tx=0).stop 2023-10-06T03:29:49.866 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.805+0000 7fe491f9d700 1 -- 172.21.15.110:0/1617108862 reap_dead start 2023-10-06T03:29:49.866 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.805+0000 7fe493a00700 1 -- 172.21.15.110:0/1617108862 shutdown_connections 2023-10-06T03:29:49.866 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.805+0000 7fe493a00700 1 -- 172.21.15.110:0/1617108862 >> 172.21.15.110:0/1617108862 conn(0x7fe48c0cc560 msgr2=0x7fe48c069c80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:29:49.866 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.805+0000 7fe493a00700 1 -- 172.21.15.110:0/1617108862 shutdown_connections 2023-10-06T03:29:49.866 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.805+0000 7fe493a00700 1 -- 172.21.15.110:0/1617108862 wait complete. 2023-10-06T03:29:49.866 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.805+0000 7f077affd700 1 -- 172.21.15.110:0/1348442177 <== osd.4 v2:172.21.15.192:6816/33683 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f0774002d30 con 0x7f078006a2a0 2023-10-06T03:29:49.867 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.805+0000 7f177e7fc700 1 -- 172.21.15.110:0/1675188097 <== osd.5 v2:172.21.15.192:6800/33684 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23999 (crc 0 0 0) 0x7f1774006b90 con 0x7f17740015f0 2023-10-06T03:29:49.867 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.813+0000 7f6ec0ba6700 1 -- 172.21.15.110:0/1520905109 --> [v2:172.21.15.110:6808/33395,v1:172.21.15.110:6809/33395] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f6e9c002c90 con 0x7f6ea806a340 2023-10-06T03:29:49.868 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.813+0000 7f0778ff9700 1 -- 172.21.15.110:0/1348442177 >> [v2:172.21.15.192:6816/33683,v1:172.21.15.192:6817/33683] conn(0x7f078006a2a0 msgr2=0x7f078006c6d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:49.868 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.813+0000 7f0778ff9700 1 --2- 172.21.15.110:0/1348442177 >> [v2:172.21.15.192:6816/33683,v1:172.21.15.192:6817/33683] conn(0x7f078006a2a0 0x7f078006c6d0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:49.868 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.813+0000 7f0778ff9700 1 -- 172.21.15.110:0/1348442177 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f07800605d0 msgr2=0x7f0780062a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:49.868 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.813+0000 7f0778ff9700 1 --2- 172.21.15.110:0/1348442177 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f07800605d0 0x7f0780062a80 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f07840058d0 tx=0x7f07840079d0 comp rx=0 tx=0).stop 2023-10-06T03:29:49.868 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.813+0000 7f0778ff9700 1 -- 172.21.15.110:0/1348442177 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0794154d90 msgr2=0x7f0794151330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:49.868 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.813+0000 7f0778ff9700 1 --2- 172.21.15.110:0/1348442177 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0794154d90 0x7f0794151330 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f078c009de0 tx=0x7f078c019000 comp rx=0 tx=0).stop 2023-10-06T03:29:49.868 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.813+0000 7f6ea37fe700 1 -- 172.21.15.110:0/1520905109 <== osd.3 v2:172.21.15.110:6808/33395 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f6e9c002c90 con 0x7f6ea806a340 2023-10-06T03:29:49.869 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.813+0000 7f07937fe700 1 -- 172.21.15.110:0/1348442177 reap_dead start 2023-10-06T03:29:49.869 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.813+0000 7f0778ff9700 1 -- 172.21.15.110:0/1348442177 shutdown_connections 2023-10-06T03:29:49.869 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.813+0000 7f0778ff9700 1 -- 172.21.15.110:0/1348442177 >> 172.21.15.110:0/1348442177 conn(0x7f07940cc560 msgr2=0x7f07940addf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:29:49.870 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.813+0000 7f0778ff9700 1 -- 172.21.15.110:0/1348442177 shutdown_connections 2023-10-06T03:29:49.870 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.813+0000 7f0778ff9700 1 -- 172.21.15.110:0/1348442177 wait complete. 2023-10-06T03:29:49.870 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.817+0000 7f6ec0ba6700 1 -- 172.21.15.110:0/1520905109 >> [v2:172.21.15.110:6808/33395,v1:172.21.15.110:6809/33395] conn(0x7f6ea806a340 msgr2=0x7f6ea806c770 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:49.870 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.817+0000 7f6ec0ba6700 1 --2- 172.21.15.110:0/1520905109 >> [v2:172.21.15.110:6808/33395,v1:172.21.15.110:6809/33395] conn(0x7f6ea806a340 0x7f6ea806c770 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:49.870 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.817+0000 7f6bad888700 1 -- 172.21.15.110:0/1204609507 --> [v2:172.21.15.110:6800/33396,v1:172.21.15.110:6801/33396] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f6ba8156d80 con 0x7f6b9406a2f0 2023-10-06T03:29:49.871 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.817+0000 7f6ba4ff9700 1 -- 172.21.15.110:0/1204609507 <== osd.1 v2:172.21.15.110:6800/33396 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f6ba8156d80 con 0x7f6b9406a2f0 2023-10-06T03:29:49.871 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.817+0000 7f6bad888700 1 -- 172.21.15.110:0/1204609507 >> [v2:172.21.15.110:6800/33396,v1:172.21.15.110:6801/33396] conn(0x7f6b9406a2f0 msgr2=0x7f6b9406c720 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:49.872 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.817+0000 7f6bad888700 1 --2- 172.21.15.110:0/1204609507 >> [v2:172.21.15.110:6800/33396,v1:172.21.15.110:6801/33396] conn(0x7f6b9406a2f0 0x7f6b9406c720 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:49.872 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.817+0000 7f6bad888700 1 -- 172.21.15.110:0/1204609507 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f6b94060620 msgr2=0x7f6b94062ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:49.872 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.817+0000 7f6bad888700 1 --2- 172.21.15.110:0/1204609507 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f6b94060620 0x7f6b94062ad0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f6ba81554f0 tx=0x7f6b9c017000 comp rx=0 tx=0).stop 2023-10-06T03:29:49.872 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.817+0000 7f6bad888700 1 -- 172.21.15.110:0/1204609507 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6ba806e7f0 msgr2=0x7f6ba814af10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:49.872 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.817+0000 7f6bad888700 1 --2- 172.21.15.110:0/1204609507 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6ba806e7f0 0x7f6ba814af10 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f6ba000cd30 tx=0x7f6ba0001ad0 comp rx=0 tx=0).stop 2023-10-06T03:29:49.872 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.817+0000 7f6bad087700 1 -- 172.21.15.110:0/1204609507 reap_dead start 2023-10-06T03:29:49.872 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.821+0000 7f6bad888700 1 -- 172.21.15.110:0/1204609507 shutdown_connections 2023-10-06T03:29:49.872 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.821+0000 7f6bad888700 1 -- 172.21.15.110:0/1204609507 >> 172.21.15.110:0/1204609507 conn(0x7f6ba806dce0 msgr2=0x7f6ba80adc70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:29:49.872 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.821+0000 7f6bad888700 1 -- 172.21.15.110:0/1204609507 shutdown_connections 2023-10-06T03:29:49.873 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.821+0000 7f6ec0ba6700 1 -- 172.21.15.110:0/1520905109 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f6ea8060670 msgr2=0x7f6ea8062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:49.873 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.821+0000 7f6ec0ba6700 1 --2- 172.21.15.110:0/1520905109 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f6ea8060670 0x7f6ea8062b20 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f6eac000c00 tx=0x7f6eac001bc0 comp rx=0 tx=0).stop 2023-10-06T03:29:49.873 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.821+0000 7f6ec0ba6700 1 -- 172.21.15.110:0/1520905109 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6ebc0c1780 msgr2=0x7f6ebc0c1bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:49.873 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.821+0000 7f6ec0ba6700 1 --2- 172.21.15.110:0/1520905109 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6ebc0c1780 0x7f6ebc0c1bf0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f6eb40013d0 tx=0x7f6eb4016000 comp rx=0 tx=0).stop 2023-10-06T03:29:49.873 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.821+0000 7f6ebbfff700 1 -- 172.21.15.110:0/1520905109 reap_dead start 2023-10-06T03:29:49.874 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.821+0000 7f6bad888700 1 -- 172.21.15.110:0/1204609507 wait complete. 2023-10-06T03:29:49.874 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.821+0000 7f6ec0ba6700 1 -- 172.21.15.110:0/1520905109 shutdown_connections 2023-10-06T03:29:49.874 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.821+0000 7f6ec0ba6700 1 -- 172.21.15.110:0/1520905109 >> 172.21.15.110:0/1520905109 conn(0x7f6ebc06e760 msgr2=0x7f6ebc0b08f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:29:49.875 INFO:teuthology.orchestra.run.smithi110.stdout:64424509450 2023-10-06T03:29:49.875 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-10-06T03:29:49.875 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.821+0000 7f6ec0ba6700 1 -- 172.21.15.110:0/1520905109 shutdown_connections 2023-10-06T03:29:49.876 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.821+0000 7f6ec0ba6700 1 -- 172.21.15.110:0/1520905109 wait complete. 2023-10-06T03:29:49.876 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.825+0000 7fe95759e700 1 -- 172.21.15.110:0/184029913 --> [v2:172.21.15.192:6824/33686,v1:172.21.15.192:6825/33686] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fe9580ac2c0 con 0x7fe93806a2f0 2023-10-06T03:29:49.876 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.825+0000 7fe9467fc700 1 -- 172.21.15.110:0/184029913 <== osd.6 v2:172.21.15.192:6824/33686 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fe9580ac2c0 con 0x7fe93806a2f0 2023-10-06T03:29:49.877 INFO:teuthology.orchestra.run.smithi110.stdout:68719476745 2023-10-06T03:29:49.877 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2023-10-06T03:29:49.878 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.829+0000 7f1794fe8700 1 -- 172.21.15.110:0/1675188097 --> [v2:172.21.15.192:6800/33684,v1:172.21.15.192:6801/33684] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f1774005c80 con 0x7f17740015f0 2023-10-06T03:29:49.878 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.829+0000 7fe95759e700 1 -- 172.21.15.110:0/184029913 >> [v2:172.21.15.192:6824/33686,v1:172.21.15.192:6825/33686] conn(0x7fe93806a2f0 msgr2=0x7fe93806c720 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:49.879 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.829+0000 7fe95759e700 1 --2- 172.21.15.110:0/184029913 >> [v2:172.21.15.192:6824/33686,v1:172.21.15.192:6825/33686] conn(0x7fe93806a2f0 0x7fe93806c720 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:49.879 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.829+0000 7fe95759e700 1 -- 172.21.15.110:0/184029913 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fe938060620 msgr2=0x7fe938062ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:49.879 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.829+0000 7fe95759e700 1 --2- 172.21.15.110:0/184029913 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fe938060620 0x7fe938062ad0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fe9580c92c0 tx=0x7fe95000b040 comp rx=0 tx=0).stop 2023-10-06T03:29:49.879 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.829+0000 7fe95759e700 1 -- 172.21.15.110:0/184029913 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe958154cf0 msgr2=0x7fe95816a160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:49.879 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.829+0000 7fe95759e700 1 --2- 172.21.15.110:0/184029913 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe958154cf0 0x7fe95816a160 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fe94c00c5d0 tx=0x7fe94c007f60 comp rx=0 tx=0).stop 2023-10-06T03:29:49.879 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.829+0000 7fe956d9d700 1 -- 172.21.15.110:0/184029913 reap_dead start 2023-10-06T03:29:49.879 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.829+0000 7fe95759e700 1 -- 172.21.15.110:0/184029913 shutdown_connections 2023-10-06T03:29:49.879 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.829+0000 7fe95759e700 1 -- 172.21.15.110:0/184029913 >> 172.21.15.110:0/184029913 conn(0x7fe95806dce0 msgr2=0x7fe958151ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:29:49.879 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.829+0000 7fe95759e700 1 -- 172.21.15.110:0/184029913 shutdown_connections 2023-10-06T03:29:49.879 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.829+0000 7fe95759e700 1 -- 172.21.15.110:0/184029913 wait complete. 2023-10-06T03:29:49.880 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.829+0000 7f177e7fc700 1 -- 172.21.15.110:0/1675188097 <== osd.5 v2:172.21.15.192:6800/33684 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f1774005c80 con 0x7f17740015f0 2023-10-06T03:29:49.880 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.829+0000 7f1794fe8700 1 -- 172.21.15.110:0/1675188097 >> [v2:172.21.15.192:6800/33684,v1:172.21.15.192:6801/33684] conn(0x7f17740015f0 msgr2=0x7f1774003aa0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:49.881 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.829+0000 7f1794fe8700 1 --2- 172.21.15.110:0/1675188097 >> [v2:172.21.15.192:6800/33684,v1:172.21.15.192:6801/33684] conn(0x7f17740015f0 0x7f1774003aa0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:49.881 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.829+0000 7f1794fe8700 1 -- 172.21.15.110:0/1675188097 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f17780606c0 msgr2=0x7f1778062b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:49.881 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.829+0000 7f1794fe8700 1 --2- 172.21.15.110:0/1675188097 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f17780606c0 0x7f1778062b70 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f1780006fd0 tx=0x7f1780008040 comp rx=0 tx=0).stop 2023-10-06T03:29:49.881 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.829+0000 7f1794fe8700 1 -- 172.21.15.110:0/1675188097 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1790155b00 msgr2=0x7f179016e870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:49.881 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.829+0000 7f1794fe8700 1 --2- 172.21.15.110:0/1675188097 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1790155b00 0x7f179016e870 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f1784018000 tx=0x7f1784019000 comp rx=0 tx=0).stop 2023-10-06T03:29:49.881 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.829+0000 7f178ed9d700 1 -- 172.21.15.110:0/1675188097 reap_dead start 2023-10-06T03:29:49.881 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.829+0000 7f1794fe8700 1 -- 172.21.15.110:0/1675188097 shutdown_connections 2023-10-06T03:29:49.881 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.829+0000 7f1794fe8700 1 -- 172.21.15.110:0/1675188097 >> 172.21.15.110:0/1675188097 conn(0x7f17900ccd00 msgr2=0x7f17901513c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:29:49.881 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.829+0000 7f1794fe8700 1 -- 172.21.15.110:0/1675188097 shutdown_connections 2023-10-06T03:29:49.881 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.829+0000 7f1794fe8700 1 -- 172.21.15.110:0/1675188097 wait complete. 2023-10-06T03:29:49.882 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.829+0000 7f94f359e700 1 -- 172.21.15.110:0/3208140742 --> [v2:172.21.15.192:6808/33685,v1:172.21.15.192:6809/33685] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f94ec052ee0 con 0x7f94ec0cb9b0 2023-10-06T03:29:49.882 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.829+0000 7f94e27fc700 1 -- 172.21.15.110:0/3208140742 <== osd.7 v2:172.21.15.192:6808/33685 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f94ec052ee0 con 0x7f94ec0cb9b0 2023-10-06T03:29:49.883 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.829+0000 7f94f359e700 1 -- 172.21.15.110:0/3208140742 >> [v2:172.21.15.192:6808/33685,v1:172.21.15.192:6809/33685] conn(0x7f94ec0cb9b0 msgr2=0x7f94ec003140 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:49.883 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.829+0000 7f94f359e700 1 --2- 172.21.15.110:0/3208140742 >> [v2:172.21.15.192:6808/33685,v1:172.21.15.192:6809/33685] conn(0x7f94ec0cb9b0 0x7f94ec003140 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:49.883 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.829+0000 7f94f359e700 1 -- 172.21.15.110:0/3208140742 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f94d4060620 msgr2=0x7f94d4062ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:49.883 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.829+0000 7f94f359e700 1 --2- 172.21.15.110:0/3208140742 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f94d4060620 0x7f94d4062ad0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f94ec04b1e0 tx=0x7f94e8016000 comp rx=0 tx=0).stop 2023-10-06T03:29:49.883 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.829+0000 7f94f359e700 1 -- 172.21.15.110:0/3208140742 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f94ec053c00 msgr2=0x7f94ec0dcf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:49.883 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.829+0000 7f94f359e700 1 --2- 172.21.15.110:0/3208140742 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f94ec053c00 0x7f94ec0dcf50 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f94f4065d30 tx=0x7f94f4065170 comp rx=0 tx=0).stop 2023-10-06T03:29:49.883 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.833+0000 7f94f2d9d700 1 -- 172.21.15.110:0/3208140742 reap_dead start 2023-10-06T03:29:49.883 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.833+0000 7f94f359e700 1 -- 172.21.15.110:0/3208140742 shutdown_connections 2023-10-06T03:29:49.884 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.833+0000 7f94f359e700 1 -- 172.21.15.110:0/3208140742 >> 172.21.15.110:0/3208140742 conn(0x7f94ec054c90 msgr2=0x7f94ec0d2160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:29:49.884 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.833+0000 7f94f359e700 1 -- 172.21.15.110:0/3208140742 shutdown_connections 2023-10-06T03:29:49.884 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.833+0000 7f94f359e700 1 -- 172.21.15.110:0/3208140742 wait complete. 2023-10-06T03:29:49.884 INFO:teuthology.orchestra.run.smithi110.stdout:60129542153 2023-10-06T03:29:49.885 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2023-10-06T03:29:49.885 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.833+0000 7f0c5b7fe700 1 -- 172.21.15.110:0/3012905740 --> [v2:172.21.15.110:6816/33397,v1:172.21.15.110:6817/33397] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f0c7006d8d0 con 0x7f0c701584b0 2023-10-06T03:29:49.885 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.837+0000 7f0c65ffb700 1 -- 172.21.15.110:0/3012905740 <== osd.2 v2:172.21.15.110:6816/33397 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f0c7006d8d0 con 0x7f0c701584b0 2023-10-06T03:29:49.886 INFO:teuthology.orchestra.run.smithi110.stdout:60129542152 2023-10-06T03:29:49.886 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-10-06T03:29:49.887 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.837+0000 7f0c5b7fe700 1 -- 172.21.15.110:0/3012905740 >> [v2:172.21.15.110:6816/33397,v1:172.21.15.110:6817/33397] conn(0x7f0c701584b0 msgr2=0x7f0c700c3330 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:49.888 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.837+0000 7f0c5b7fe700 1 --2- 172.21.15.110:0/3012905740 >> [v2:172.21.15.110:6816/33397,v1:172.21.15.110:6817/33397] conn(0x7f0c701584b0 0x7f0c700c3330 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:49.888 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.837+0000 7f0c5b7fe700 1 -- 172.21.15.110:0/3012905740 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f0c54060620 msgr2=0x7f0c54062ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:49.888 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.837+0000 7f0c5b7fe700 1 --2- 172.21.15.110:0/3012905740 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f0c54060620 0x7f0c54062ad0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f0c6800e2f0 tx=0x7f0c6800b040 comp rx=0 tx=0).stop 2023-10-06T03:29:49.888 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.837+0000 7f0c5b7fe700 1 -- 172.21.15.110:0/3012905740 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0c701592d0 msgr2=0x7f0c700c4f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:49.888 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.837+0000 7f0c5b7fe700 1 --2- 172.21.15.110:0/3012905740 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0c701592d0 0x7f0c700c4f50 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f0c70068cc0 tx=0x7f0c6c01a000 comp rx=0 tx=0).stop 2023-10-06T03:29:49.888 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.837+0000 7f0c76402700 1 -- 172.21.15.110:0/3012905740 reap_dead start 2023-10-06T03:29:49.888 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.837+0000 7f0c5b7fe700 1 -- 172.21.15.110:0/3012905740 shutdown_connections 2023-10-06T03:29:49.888 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.837+0000 7f0c5b7fe700 1 -- 172.21.15.110:0/3012905740 >> 172.21.15.110:0/3012905740 conn(0x7f0c700cc6d0 msgr2=0x7f0c70069e30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:29:49.888 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.837+0000 7f0c5b7fe700 1 -- 172.21.15.110:0/3012905740 shutdown_connections 2023-10-06T03:29:49.888 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:49.837+0000 7f0c5b7fe700 1 -- 172.21.15.110:0/3012905740 wait complete. 2023-10-06T03:29:49.889 INFO:teuthology.orchestra.run.smithi110.stdout:68719476746 2023-10-06T03:29:49.889 INFO:teuthology.orchestra.run.smithi110.stdout:68719476746 2023-10-06T03:29:49.889 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2023-10-06T03:29:49.890 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2023-10-06T03:29:49.891 INFO:teuthology.orchestra.run.smithi110.stdout:64424509449 2023-10-06T03:29:49.891 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2023-10-06T03:29:49.892 INFO:teuthology.orchestra.run.smithi110.stdout:60129542152 2023-10-06T03:29:49.892 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-10-06T03:29:50.069 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.065+0000 7f276745d700 1 Processor -- start 2023-10-06T03:29:50.070 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.065+0000 7f276745d700 1 -- start start 2023-10-06T03:29:50.070 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.065+0000 7fb9eaca7700 1 Processor -- start 2023-10-06T03:29:50.070 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.065+0000 7fb9eaca7700 1 -- start start 2023-10-06T03:29:50.071 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.065+0000 7eff66410700 1 Processor -- start 2023-10-06T03:29:50.071 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.065+0000 7f276745d700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f2760156f90 0x7f276006cb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:50.071 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.065+0000 7f276745d700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f2760157360 0x7f276006a640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:50.071 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.065+0000 7f276745d700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f276006c6b0 con 0x7f276006ab80 2023-10-06T03:29:50.072 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.065+0000 7f276745d700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f276006afc0 con 0x7f2760157360 2023-10-06T03:29:50.072 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.065+0000 7f276745d700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f276006b7c0 con 0x7f2760156f90 2023-10-06T03:29:50.072 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.065+0000 7fb9eaca7700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fb9e415cc80 0x7fb9e40c6420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:50.072 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.065+0000 7fb9eaca7700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb9e41590a0 0x7fb9e40694a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:50.072 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.065+0000 7fb9eaca7700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9e40699e0 con 0x7fb9e415d050 2023-10-06T03:29:50.072 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.065+0000 7fb9eaca7700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb9e406ace0 con 0x7fb9e415cc80 2023-10-06T03:29:50.072 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.065+0000 7fb9eaca7700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9e4069c40 con 0x7fb9e41590a0 2023-10-06T03:29:50.073 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.065+0000 7f27651f9700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f2760156f90 0x7f276006cb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:50.073 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.065+0000 7f27651f9700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f2760156f90 0x7f276006cb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.110:50820/0 (socket says 172.21.15.110:50820) 2023-10-06T03:29:50.073 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.065+0000 7f27651f9700 1 -- 172.21.15.110:0/62284582 learned_addr learned my addr 172.21.15.110:0/62284582 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:29:50.074 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.065+0000 7eff66410700 1 -- start start 2023-10-06T03:29:50.074 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.069+0000 7fb9e94a4700 1 --1- >> v1:172.21.15.192:6789/0 conn(0x7fb9e415d050 0x7fb9e4158990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.110:52100/0 (socket says 172.21.15.110:52100) 2023-10-06T03:29:50.074 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.069+0000 7fb9e94a4700 1 -- 172.21.15.110:0/1769261396 learned_addr learned my addr 172.21.15.110:0/1769261396 (peer_addr_for_me v1:172.21.15.110:0/0) 2023-10-06T03:29:50.075 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.069+0000 7f27649f8700 1 --2- 172.21.15.110:0/62284582 >> v2:172.21.15.110:3300/0 conn(0x7f2760157360 0x7f276006a640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:50.075 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.069+0000 7f2757fff700 1 -- 172.21.15.110:0/62284582 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1943323409 0 0) 0x7f276006c6b0 con 0x7f276006ab80 2023-10-06T03:29:50.075 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.069+0000 7f2757fff700 1 -- 172.21.15.110:0/62284582 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f274c003570 con 0x7f276006ab80 2023-10-06T03:29:50.075 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.069+0000 7fb9ea4a6700 1 --2- 172.21.15.110:0/1769261396 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb9e41590a0 0x7fb9e40694a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:50.076 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.069+0000 7f27649f8700 1 -- 172.21.15.110:0/62284582 >> v2:172.21.15.192:3300/0 conn(0x7f2760156f90 msgr2=0x7f276006cb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:29:50.076 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.069+0000 7f27649f8700 1 --2- 172.21.15.110:0/62284582 >> v2:172.21.15.192:3300/0 conn(0x7f2760156f90 0x7f276006cb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:50.076 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.069+0000 7f27649f8700 1 -- 172.21.15.110:0/62284582 >> v1:172.21.15.110:6789/0 conn(0x7f276006ab80 legacy=0x7f27600695e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:50.076 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.069+0000 7f27649f8700 1 -- 172.21.15.110:0/62284582 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f276006baa0 con 0x7f2760157360 2023-10-06T03:29:50.076 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.069+0000 7fb9e8ca3700 1 -- 172.21.15.110:0/1769261396 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3378763239 0 0) 0x7fb9e40699e0 con 0x7fb9e415d050 2023-10-06T03:29:50.077 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.069+0000 7fb9e8ca3700 1 -- 172.21.15.110:0/1769261396 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb9d0003570 con 0x7fb9e415d050 2023-10-06T03:29:50.077 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.069+0000 7f27649f8700 1 --2- 172.21.15.110:0/62284582 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2760157360 0x7f276006a640 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f27600c8f10 tx=0x7f27500091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d8a9e9d193ce265 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:29:50.078 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.069+0000 7eff66410700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7eff600c7330 0x7eff601595e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:50.078 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.069+0000 7eff66410700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7eff600c80e0 0x7eff600c8500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:50.078 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.069+0000 7eff66410700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff600c7780 con 0x7eff6006e7f0 2023-10-06T03:29:50.078 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.069+0000 7eff66410700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7eff601553e0 con 0x7eff600c80e0 2023-10-06T03:29:50.078 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.069+0000 7eff66410700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7eff600c6ea0 con 0x7eff600c7330 2023-10-06T03:29:50.078 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.069+0000 7fb9ea4a6700 1 -- 172.21.15.110:0/1769261396 >> v2:172.21.15.192:3300/0 conn(0x7fb9e415cc80 msgr2=0x7fb9e40c6420 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-06T03:29:50.079 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.069+0000 7fb9ea4a6700 1 --2- 172.21.15.110:0/1769261396 >> v2:172.21.15.192:3300/0 conn(0x7fb9e415cc80 0x7fb9e40c6420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:50.079 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.069+0000 7fb9ea4a6700 1 -- 172.21.15.110:0/1769261396 >> v1:172.21.15.192:6789/0 conn(0x7fb9e415d050 legacy=0x7fb9e4158990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:50.079 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.069+0000 7fb9ea4a6700 1 -- 172.21.15.110:0/1769261396 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9e4069f20 con 0x7fb9e41590a0 2023-10-06T03:29:50.079 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.069+0000 7fb9ea4a6700 1 --2- 172.21.15.110:0/1769261396 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb9e41590a0 0x7fb9e40694a0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fb9dc009e60 tx=0x7fb9dc009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2d2b54613ab4b8ab server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:29:50.079 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.069+0000 7eff65c0f700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7eff600c80e0 0x7eff600c8500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:50.079 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.069+0000 7eff65c0f700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7eff600c80e0 0x7eff600c8500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:60694/0 (socket says 172.21.15.110:60694) 2023-10-06T03:29:50.080 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.069+0000 7f2757fff700 1 -- 172.21.15.110:0/62284582 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2750011630 con 0x7f2760157360 2023-10-06T03:29:50.080 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.069+0000 7f2757fff700 1 -- 172.21.15.110:0/62284582 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27500117b0 con 0x7f2760157360 2023-10-06T03:29:50.080 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.069+0000 7f2757fff700 1 -- 172.21.15.110:0/62284582 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2750011b70 con 0x7f2760157360 2023-10-06T03:29:50.080 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.069+0000 7eff65c0f700 1 -- 172.21.15.110:0/3896863682 learned_addr learned my addr 172.21.15.110:0/3896863682 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:29:50.080 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.069+0000 7fb9e8ca3700 1 -- 172.21.15.110:0/1769261396 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9dc00e070 con 0x7fb9e41590a0 2023-10-06T03:29:50.081 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.069+0000 7f276745d700 1 -- 172.21.15.110:0/62284582 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2760157360 msgr2=0x7f276006a640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:50.081 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.069+0000 7f276745d700 1 --2- 172.21.15.110:0/62284582 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2760157360 0x7f276006a640 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f27600c8f10 tx=0x7f27500091a0 comp rx=0 tx=0).stop 2023-10-06T03:29:50.081 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.069+0000 7f276745d700 1 -- 172.21.15.110:0/62284582 shutdown_connections 2023-10-06T03:29:50.081 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.069+0000 7f276745d700 1 --2- 172.21.15.110:0/62284582 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2760157360 0x7f276006a640 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:50.081 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.069+0000 7f276745d700 1 --2- 172.21.15.110:0/62284582 >> v2:172.21.15.192:3300/0 conn(0x7f2760156f90 0x7f276006cb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:50.081 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.069+0000 7f276745d700 1 -- 172.21.15.110:0/62284582 >> 172.21.15.110:0/62284582 conn(0x7f27600cc6d0 msgr2=0x7f27600c8670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:29:50.081 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.069+0000 7f276745d700 1 -- 172.21.15.110:0/62284582 shutdown_connections 2023-10-06T03:29:50.081 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.069+0000 7eff57fff700 1 -- 172.21.15.110:0/3896863682 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1527428254 0 0) 0x7eff600c7780 con 0x7eff6006e7f0 2023-10-06T03:29:50.082 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.069+0000 7eff57fff700 1 -- 172.21.15.110:0/3896863682 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7eff4c003570 con 0x7eff6006e7f0 2023-10-06T03:29:50.082 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.069+0000 7eff57fff700 1 -- 172.21.15.110:0/3896863682 <== mon.0 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 12202989 0 0) 0x7eff4c003570 con 0x7eff6006e7f0 2023-10-06T03:29:50.082 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.069+0000 7eff57fff700 1 -- 172.21.15.110:0/3896863682 >> v2:172.21.15.192:3300/0 conn(0x7eff600c7330 msgr2=0x7eff601595e0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-10-06T03:29:50.082 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.069+0000 7eff57fff700 1 --2- 172.21.15.110:0/3896863682 >> v2:172.21.15.192:3300/0 conn(0x7eff600c7330 0x7eff601595e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:50.082 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.069+0000 7eff57fff700 1 -- 172.21.15.110:0/3896863682 >> v2:172.21.15.110:3300/0 conn(0x7eff600c80e0 msgr2=0x7eff600c8500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:29:50.082 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.069+0000 7eff57fff700 1 --2- 172.21.15.110:0/3896863682 >> v2:172.21.15.110:3300/0 conn(0x7eff600c80e0 0x7eff600c8500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:50.082 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.069+0000 7eff57fff700 1 -- 172.21.15.110:0/3896863682 --> v1:172.21.15.110:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff600c6570 con 0x7eff6006e7f0 2023-10-06T03:29:50.082 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.069+0000 7eff57fff700 1 -- 172.21.15.110:0/3896863682 <== mon.0 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 118603703 0 0) 0x7eff5c001470 con 0x7eff6006e7f0 2023-10-06T03:29:50.083 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.069+0000 7fb9e8ca3700 1 -- 172.21.15.110:0/1769261396 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9dc00bb60 con 0x7fb9e41590a0 2023-10-06T03:29:50.083 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.069+0000 7fb9e8ca3700 1 -- 172.21.15.110:0/1769261396 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9dc012430 con 0x7fb9e41590a0 2023-10-06T03:29:50.083 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.069+0000 7fb9eaca7700 1 -- 172.21.15.110:0/1769261396 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb9e41590a0 msgr2=0x7fb9e40694a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:50.083 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.069+0000 7fb9eaca7700 1 --2- 172.21.15.110:0/1769261396 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb9e41590a0 0x7fb9e40694a0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fb9dc009e60 tx=0x7fb9dc009780 comp rx=0 tx=0).stop 2023-10-06T03:29:50.083 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.069+0000 7f276745d700 1 -- 172.21.15.110:0/62284582 wait complete. 2023-10-06T03:29:50.084 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.069+0000 7eff57fff700 1 -- 172.21.15.110:0/3896863682 >> v1:172.21.15.110:6789/0 conn(0x7eff6006e7f0 legacy=0x7eff6006c3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:50.084 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.069+0000 7eff57fff700 1 --2- 172.21.15.110:0/3896863682 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7eff4c003a60 0x7eff4c003e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:50.084 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.069+0000 7eff57fff700 1 --2- 172.21.15.110:0/3896863682 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7eff4c004810 0x7eff4c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:50.084 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.069+0000 7eff57fff700 1 --2- 172.21.15.110:0/3896863682 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7eff4c0052e0 0x7eff4c00ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:50.084 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.069+0000 7fb9eaca7700 1 -- 172.21.15.110:0/1769261396 shutdown_connections 2023-10-06T03:29:50.085 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.069+0000 7fb9eaca7700 1 --2- 172.21.15.110:0/1769261396 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb9e41590a0 0x7fb9e40694a0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:50.085 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.069+0000 7fb9eaca7700 1 --2- 172.21.15.110:0/1769261396 >> v2:172.21.15.192:3300/0 conn(0x7fb9e415cc80 0x7fb9e40c6420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:50.085 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.069+0000 7fb9eaca7700 1 -- 172.21.15.110:0/1769261396 >> 172.21.15.110:0/1769261396 conn(0x7fb9e406e760 msgr2=0x7fb9e40cba30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:29:50.085 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.069+0000 7eff57fff700 1 -- 172.21.15.110:0/3896863682 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff601553e0 con 0x7eff4c004810 2023-10-06T03:29:50.085 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.069+0000 7eff64c0d700 1 --2- 172.21.15.110:0/3896863682 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7eff4c004810 0x7eff4c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:50.085 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.069+0000 7eff6540e700 1 --2- 172.21.15.110:0/3896863682 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7eff4c003a60 0x7eff4c003e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:50.086 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.069+0000 7eff65c0f700 1 --2- 172.21.15.110:0/3896863682 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7eff4c0052e0 0x7eff4c00ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:50.086 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.069+0000 7eff57fff700 1 -- 172.21.15.110:0/3896863682 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff600c6ea0 con 0x7eff4c0052e0 2023-10-06T03:29:50.086 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.069+0000 7eff57fff700 1 -- 172.21.15.110:0/3896863682 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff4c00c2b0 con 0x7eff4c003a60 2023-10-06T03:29:50.086 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.069+0000 7fb9eaca7700 1 -- 172.21.15.110:0/1769261396 shutdown_connections 2023-10-06T03:29:50.086 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.069+0000 7fb9eaca7700 1 -- 172.21.15.110:0/1769261396 wait complete. 2023-10-06T03:29:50.086 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.073+0000 7f00b3d9e700 1 Processor -- start 2023-10-06T03:29:50.087 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.073+0000 7eff6540e700 1 -- 172.21.15.110:0/3896863682 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7eff4c0052e0 msgr2=0x7eff4c00ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:29:50.087 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.073+0000 7eff6540e700 1 --2- 172.21.15.110:0/3896863682 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7eff4c0052e0 0x7eff4c00ba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:50.087 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.073+0000 7eff6540e700 1 -- 172.21.15.110:0/3896863682 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7eff4c004810 msgr2=0x7eff4c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:29:50.087 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.073+0000 7eff6540e700 1 --2- 172.21.15.110:0/3896863682 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7eff4c004810 0x7eff4c004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:50.087 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.073+0000 7eff6540e700 1 -- 172.21.15.110:0/3896863682 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff4c003570 con 0x7eff4c003a60 2023-10-06T03:29:50.087 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.073+0000 7f276745d700 1 Processor -- start 2023-10-06T03:29:50.088 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.073+0000 7f276745d700 1 -- start start 2023-10-06T03:29:50.088 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.073+0000 7ff845b0f700 1 Processor -- start 2023-10-06T03:29:50.088 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.073+0000 7fb9eaca7700 1 Processor -- start 2023-10-06T03:29:50.088 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.073+0000 7f00b3d9e700 1 -- start start 2023-10-06T03:29:50.089 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.073+0000 7f276745d700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f276006ab80 0x7f2760151d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:50.089 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.073+0000 7f00b3d9e700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f00ac06b960 0x7f00ac06bd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:50.089 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.073+0000 7fa351afa700 1 Processor -- start 2023-10-06T03:29:50.090 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.073+0000 7eff65c0f700 1 --2- 172.21.15.110:0/3896863682 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7eff4c0052e0 0x7eff4c00ba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-06T03:29:50.090 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.073+0000 7eff6540e700 1 --2- 172.21.15.110:0/3896863682 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7eff4c003a60 0x7eff4c003e30 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7eff5c002740 tx=0x7eff5c008be0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8b693e92277326bf server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:29:50.090 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.073+0000 7eff65c0f700 1 -- 172.21.15.110:0/3896863682 reap_dead start 2023-10-06T03:29:50.090 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.073+0000 7f00b3d9e700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f00ac152ff0 0x7f00ac155b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:50.090 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.073+0000 7f00b3d9e700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00ac1590b0 con 0x7f00ac06ec20 2023-10-06T03:29:50.090 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.073+0000 7f00b3d9e700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f00ac1568b0 con 0x7f00ac06b960 2023-10-06T03:29:50.091 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.073+0000 7f00b3d9e700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00ac159390 con 0x7f00ac152ff0 2023-10-06T03:29:50.091 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.073+0000 7ff845b0f700 1 -- start start 2023-10-06T03:29:50.091 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.073+0000 7fb9eaca7700 1 -- start start 2023-10-06T03:29:50.092 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.073+0000 7eff57fff700 1 -- 172.21.15.110:0/3896863682 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff5c0123d0 con 0x7eff4c003a60 2023-10-06T03:29:50.092 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.073+0000 7f00b233b700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f00ac152ff0 0x7f00ac155b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:50.092 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.073+0000 7f00b233b700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f00ac152ff0 0x7f00ac155b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3301/0 says I am v2:172.21.15.110:44184/0 (socket says 172.21.15.110:44184) 2023-10-06T03:29:50.092 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.073+0000 7f00b233b700 1 -- 172.21.15.110:0/669385157 learned_addr learned my addr 172.21.15.110:0/669385157 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:29:50.092 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.073+0000 7eff57fff700 1 -- 172.21.15.110:0/3896863682 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff5c012530 con 0x7eff4c003a60 2023-10-06T03:29:50.093 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.073+0000 7f00b1b3a700 1 --2- 172.21.15.110:0/669385157 >> v2:172.21.15.110:3300/0 conn(0x7f00ac06b960 0x7f00ac06bd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:50.093 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.073+0000 7ff845b0f700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7ff8400c9750 0x7ff8400c75e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:50.093 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.073+0000 7ff845b0f700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7ff8400c20d0 0x7ff8400ca3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:50.093 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.073+0000 7ff845b0f700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8400c9b80 con 0x7ff8400c8c10 2023-10-06T03:29:50.094 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.073+0000 7f00b1b3a700 1 -- 172.21.15.110:0/669385157 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f00ac152ff0 msgr2=0x7f00ac155b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:29:50.094 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.073+0000 7f00b1b3a700 1 --2- 172.21.15.110:0/669385157 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f00ac152ff0 0x7f00ac155b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:50.094 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.073+0000 7f00b1b3a700 1 -- 172.21.15.110:0/669385157 >> v1:172.21.15.192:6789/0 conn(0x7f00ac06ec20 legacy=0x7f00ac1528e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:29:50.094 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.073+0000 7f00b1b3a700 1 -- 172.21.15.110:0/669385157 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00ac159670 con 0x7f00ac06b960 2023-10-06T03:29:50.095 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.073+0000 7ff845b0f700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff8400c7b20 con 0x7ff8400c20d0 2023-10-06T03:29:50.095 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.073+0000 7f00b1b3a700 1 --2- 172.21.15.110:0/669385157 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f00ac06b960 0x7f00ac06bd30 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f009c0098e0 tx=0x7f009c0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a9b5e659a3771da4 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:29:50.096 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.073+0000 7ff845b0f700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff8400c7d10 con 0x7ff8400c9750 2023-10-06T03:29:50.096 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.073+0000 7fb9eaca7700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb9e41590a0 0x7fb9e40c8320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:50.096 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.073+0000 7fb9eaca7700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb9e415cc80 0x7fb9e414ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:50.096 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.073+0000 7fb9eaca7700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb9e415d050 0x7fb9e414e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:50.097 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.073+0000 7fb9eaca7700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9e40c4b50 con 0x7fb9e415cc80 2023-10-06T03:29:50.097 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.073+0000 7fb9eaca7700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9e40c46c0 con 0x7fb9e41590a0 2023-10-06T03:29:50.097 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.073+0000 7fb9eaca7700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9e40c3ab0 con 0x7fb9e415d050 2023-10-06T03:29:50.097 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.073+0000 7ff83effd700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7ff8400c9750 0x7ff8400c75e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:50.098 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.073+0000 7f276745d700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2760156f90 0x7f27600afd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:50.098 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.073+0000 7f276745d700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2760157360 0x7f276014f730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:50.098 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.073+0000 7f276745d700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2760150240 con 0x7f276006ab80 2023-10-06T03:29:50.098 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.073+0000 7f276745d700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2760150610 con 0x7f2760157360 2023-10-06T03:29:50.099 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.073+0000 7f276745d700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27600b02a0 con 0x7f2760156f90 2023-10-06T03:29:50.099 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.073+0000 7f27649f8700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2760156f90 0x7f27600afd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:50.099 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.073+0000 7f27649f8700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2760156f90 0x7f27600afd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3301/0 says I am v2:172.21.15.110:44196/0 (socket says 172.21.15.110:44196) 2023-10-06T03:29:50.099 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.073+0000 7f27649f8700 1 -- 172.21.15.110:0/768454148 learned_addr learned my addr 172.21.15.110:0/768454148 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:29:50.099 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.073+0000 7f27659fa700 1 --2- 172.21.15.110:0/768454148 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2760157360 0x7f276014f730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:50.100 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.073+0000 7f27651f9700 1 --2- 172.21.15.110:0/768454148 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f276006ab80 0x7f2760151d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:50.100 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.073+0000 7f27651f9700 1 -- 172.21.15.110:0/768454148 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2760156f90 msgr2=0x7f27600afd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:50.100 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.073+0000 7f27651f9700 1 --2- 172.21.15.110:0/768454148 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2760156f90 0x7f27600afd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:50.100 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.073+0000 7f27651f9700 1 -- 172.21.15.110:0/768454148 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2760157360 msgr2=0x7f276014f730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:50.100 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.073+0000 7f27651f9700 1 --2- 172.21.15.110:0/768454148 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2760157360 0x7f276014f730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:50.100 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.073+0000 7f27651f9700 1 -- 172.21.15.110:0/768454148 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2750009420 con 0x7f276006ab80 2023-10-06T03:29:50.100 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.073+0000 7f27659fa700 1 --2- 172.21.15.110:0/768454148 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2760157360 0x7f276014f730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-06T03:29:50.101 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.073+0000 7eff66410700 1 -- 172.21.15.110:0/3896863682 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7eff4c003a60 msgr2=0x7eff4c003e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:50.101 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.073+0000 7eff66410700 1 --2- 172.21.15.110:0/3896863682 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7eff4c003a60 0x7eff4c003e30 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7eff5c002740 tx=0x7eff5c008be0 comp rx=0 tx=0).stop 2023-10-06T03:29:50.101 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.073+0000 7fb9e9ca5700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb9e41590a0 0x7fb9e40c8320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:50.102 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.073+0000 7fb9e9ca5700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb9e41590a0 0x7fb9e40c8320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.110:50856/0 (socket says 172.21.15.110:50856) 2023-10-06T03:29:50.102 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.073+0000 7fb9e9ca5700 1 -- 172.21.15.110:0/3455238065 learned_addr learned my addr 172.21.15.110:0/3455238065 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:29:50.102 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.073+0000 7ff83effd700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7ff8400c9750 0x7ff8400c75e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.110:50846/0 (socket says 172.21.15.110:50846) 2023-10-06T03:29:50.102 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.073+0000 7ff83effd700 1 -- 172.21.15.110:0/17861001 learned_addr learned my addr 172.21.15.110:0/17861001 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:29:50.102 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.073+0000 7ff83ffff700 1 --2- 172.21.15.110:0/17861001 >> v2:172.21.15.110:3300/0 conn(0x7ff8400c20d0 0x7ff8400ca3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:50.103 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.077+0000 7ff40c8dd700 1 Processor -- start 2023-10-06T03:29:50.103 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.073+0000 7fa351afa700 1 -- start start 2023-10-06T03:29:50.103 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.077+0000 7f00b0b38700 1 -- 172.21.15.110:0/669385157 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f009c009e80 con 0x7f00ac06b960 2023-10-06T03:29:50.104 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.077+0000 7f27651f9700 1 --2- 172.21.15.110:0/768454148 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f276006ab80 0x7f2760151d60 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f2758009ef0 tx=0x7f275800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:29:50.104 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.077+0000 7ff40c8dd700 1 -- start start 2023-10-06T03:29:50.105 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.077+0000 7eff66410700 1 -- 172.21.15.110:0/3896863682 shutdown_connections 2023-10-06T03:29:50.105 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.077+0000 7eff66410700 1 --2- 172.21.15.110:0/3896863682 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7eff4c003a60 0x7eff4c003e30 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:50.105 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.077+0000 7eff66410700 1 -- 172.21.15.110:0/3896863682 >> 172.21.15.110:0/3896863682 conn(0x7eff6006dce0 msgr2=0x7eff6006e0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:29:50.106 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.077+0000 7ff83effd700 1 -- 172.21.15.110:0/17861001 >> v2:172.21.15.110:3300/0 conn(0x7ff8400c20d0 msgr2=0x7ff8400ca3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:29:50.106 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.077+0000 7ff83effd700 1 --2- 172.21.15.110:0/17861001 >> v2:172.21.15.110:3300/0 conn(0x7ff8400c20d0 0x7ff8400ca3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:50.106 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.077+0000 7ff83effd700 1 -- 172.21.15.110:0/17861001 >> v1:172.21.15.110:6789/0 conn(0x7ff8400c8c10 legacy=0x7ff840154d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:50.106 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.077+0000 7ff83effd700 1 -- 172.21.15.110:0/17861001 --> v2:172.21.15.192:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8400c15f0 con 0x7ff8400c9750 2023-10-06T03:29:50.107 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.077+0000 7fb9e9ca5700 1 -- 172.21.15.110:0/3455238065 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb9e415d050 msgr2=0x7fb9e414e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:50.107 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.077+0000 7fb9e9ca5700 1 --2- 172.21.15.110:0/3455238065 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb9e415d050 0x7fb9e414e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:50.107 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.077+0000 7fb9e9ca5700 1 -- 172.21.15.110:0/3455238065 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb9e415cc80 msgr2=0x7fb9e414ddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:50.108 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.077+0000 7fb9e9ca5700 1 --2- 172.21.15.110:0/3455238065 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb9e415cc80 0x7fb9e414ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:50.108 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.077+0000 7fb9e9ca5700 1 -- 172.21.15.110:0/3455238065 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9dc009a20 con 0x7fb9e41590a0 2023-10-06T03:29:50.108 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.077+0000 7f00b0b38700 1 -- 172.21.15.110:0/669385157 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f009c00b820 con 0x7f00ac06b960 2023-10-06T03:29:50.108 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.077+0000 7f00b0b38700 1 -- 172.21.15.110:0/669385157 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f009c00bbc0 con 0x7f00ac06b960 2023-10-06T03:29:50.109 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.077+0000 7eff66410700 1 -- 172.21.15.110:0/3896863682 shutdown_connections 2023-10-06T03:29:50.109 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.077+0000 7ff83effd700 1 --2- 172.21.15.110:0/17861001 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff8400c9750 0x7ff8400c75e0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7ff8400cba60 tx=0x7ff834009250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=32f71c94af0ef800 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:29:50.110 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.077+0000 7fb9e9ca5700 1 --2- 172.21.15.110:0/3455238065 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb9e41590a0 0x7fb9e40c8320 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fb9e000b600 tx=0x7fb9e00091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:29:50.110 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.077+0000 7ff83e7fc700 1 -- 172.21.15.110:0/17861001 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff834009e70 con 0x7ff8400c9750 2023-10-06T03:29:50.111 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.077+0000 7fa351afa700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fa34c069640 0x7fa34c06a880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:50.111 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.077+0000 7fa351afa700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa34c06c9e0 0x7fa34c069a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:50.111 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.077+0000 7fa351afa700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa34c0c7440 con 0x7fa34c06b7c0 2023-10-06T03:29:50.111 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.077+0000 7fa351afa700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa34c069f50 con 0x7fa34c069640 2023-10-06T03:29:50.111 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.077+0000 7fa351afa700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa34c0bf3b0 con 0x7fa34c06c9e0 2023-10-06T03:29:50.112 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.077+0000 7ff83e7fc700 1 -- 172.21.15.110:0/17861001 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff83400b760 con 0x7ff8400c9750 2023-10-06T03:29:50.112 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.077+0000 7fb9d9ffb700 1 -- 172.21.15.110:0/3455238065 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9e000f510 con 0x7fb9e41590a0 2023-10-06T03:29:50.112 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.077+0000 7fb9d9ffb700 1 -- 172.21.15.110:0/3455238065 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9e000f6b0 con 0x7fb9e41590a0 2023-10-06T03:29:50.112 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.077+0000 7fb9eaca7700 1 -- 172.21.15.110:0/3455238065 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9e40c3d90 con 0x7fb9e41590a0 2023-10-06T03:29:50.112 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.077+0000 7fb9d9ffb700 1 -- 172.21.15.110:0/3455238065 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9e000fa70 con 0x7fb9e41590a0 2023-10-06T03:29:50.113 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.077+0000 7fa3512f9700 1 --1- >> v1:172.21.15.192:6789/0 conn(0x7fa34c06b7c0 0x7fa34c06bc20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.110:52120/0 (socket says 172.21.15.110:52120) 2023-10-06T03:29:50.113 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.077+0000 7fa3512f9700 1 -- 172.21.15.110:0/3879314431 learned_addr learned my addr 172.21.15.110:0/3879314431 (peer_addr_for_me v1:172.21.15.110:0/0) 2023-10-06T03:29:50.113 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.077+0000 7ff40c8dd700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7ff408156f90 0x7ff40806cb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:50.113 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.077+0000 7ff40c8dd700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff40806c6b0 con 0x7ff4080695e0 2023-10-06T03:29:50.113 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.077+0000 7ff40c8dd700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff40806b7c0 con 0x7ff408157360 2023-10-06T03:29:50.113 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.077+0000 7ff40c8dd700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff40806baa0 con 0x7ff408156f90 2023-10-06T03:29:50.114 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.077+0000 7f00b3d9e700 1 -- 172.21.15.110:0/669385157 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f00ac06b960 msgr2=0x7f00ac06bd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:50.114 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.077+0000 7f00b3d9e700 1 --2- 172.21.15.110:0/669385157 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f00ac06b960 0x7f00ac06bd30 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f009c0098e0 tx=0x7f009c0091a0 comp rx=0 tx=0).stop 2023-10-06T03:29:50.114 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.077+0000 7ff845b0f700 1 -- 172.21.15.110:0/17861001 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff8400c9750 msgr2=0x7ff8400c75e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:50.114 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.077+0000 7ff845b0f700 1 --2- 172.21.15.110:0/17861001 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff8400c9750 0x7ff8400c75e0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7ff8400cba60 tx=0x7ff834009250 comp rx=0 tx=0).stop 2023-10-06T03:29:50.115 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.077+0000 7ff406d9d700 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7ff4080695e0 0x7ff4080abd80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:37354/0 (socket says 172.21.15.110:37354) 2023-10-06T03:29:50.115 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.077+0000 7ff845b0f700 1 -- 172.21.15.110:0/17861001 shutdown_connections 2023-10-06T03:29:50.115 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.077+0000 7ff845b0f700 1 --2- 172.21.15.110:0/17861001 >> v2:172.21.15.110:3300/0 conn(0x7ff8400c20d0 0x7ff8400ca3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:50.115 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.077+0000 7ff845b0f700 1 --2- 172.21.15.110:0/17861001 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff8400c9750 0x7ff8400c75e0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:50.115 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.077+0000 7ff845b0f700 1 -- 172.21.15.110:0/17861001 >> 172.21.15.110:0/17861001 conn(0x7ff8400cc560 msgr2=0x7ff8400c8500 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:29:50.116 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.077+0000 7eff66410700 1 -- 172.21.15.110:0/3896863682 wait complete. 2023-10-06T03:29:50.116 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.077+0000 7eff66410700 1 Processor -- start 2023-10-06T03:29:50.116 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.077+0000 7fa350af8700 1 --2- 172.21.15.110:0/3879314431 >> v2:172.21.15.110:3300/0 conn(0x7fa34c069640 0x7fa34c06a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:50.117 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.077+0000 7f27557fa700 1 -- 172.21.15.110:0/768454148 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27580092e0 con 0x7f276006ab80 2023-10-06T03:29:50.117 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.077+0000 7f27557fa700 1 -- 172.21.15.110:0/768454148 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f275800f040 con 0x7f276006ab80 2023-10-06T03:29:50.117 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.077+0000 7f27557fa700 1 -- 172.21.15.110:0/768454148 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f275800cb70 con 0x7f276006ab80 2023-10-06T03:29:50.117 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.077+0000 7f276745d700 1 -- 172.21.15.110:0/768454148 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f276014ebe0 con 0x7f276006ab80 2023-10-06T03:29:50.117 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.077+0000 7fb9eaca7700 1 -- 172.21.15.110:0/3455238065 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9e40c4350 con 0x7fb9e41590a0 2023-10-06T03:29:50.117 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.077+0000 7ff406d9d700 1 -- 172.21.15.110:0/4163094067 learned_addr learned my addr 172.21.15.110:0/4163094067 (peer_addr_for_me v1:172.21.15.110:0/0) 2023-10-06T03:29:50.118 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.077+0000 7f276745d700 1 -- 172.21.15.110:0/768454148 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27600b0630 con 0x7f276006ab80 2023-10-06T03:29:50.118 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.077+0000 7fa34bfff700 1 --2- 172.21.15.110:0/3879314431 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa34c06c9e0 0x7fa34c069a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:50.119 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.077+0000 7f00b3d9e700 1 -- 172.21.15.110:0/669385157 shutdown_connections 2023-10-06T03:29:50.119 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.077+0000 7f00b3d9e700 1 --2- 172.21.15.110:0/669385157 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f00ac152ff0 0x7f00ac155b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:50.119 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.077+0000 7f00b3d9e700 1 --2- 172.21.15.110:0/669385157 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f00ac06b960 0x7f00ac06bd30 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:50.119 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.077+0000 7f00b3d9e700 1 -- 172.21.15.110:0/669385157 >> 172.21.15.110:0/669385157 conn(0x7f00ac0cc6d0 msgr2=0x7f00ac0c8670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:29:50.119 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.077+0000 7eff66410700 1 -- start start 2023-10-06T03:29:50.120 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.081+0000 7eff66410700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7eff4c003a60 0x7eff6016e940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:50.120 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.081+0000 7eff66410700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7eff6014f630 0x7eff6014faa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:50.120 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.081+0000 7eff66410700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7eff601578b0 0x7eff60157d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:50.120 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.081+0000 7eff66410700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff60158260 con 0x7eff6014f630 2023-10-06T03:29:50.120 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.081+0000 7eff66410700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff6016e740 con 0x7eff4c003a60 2023-10-06T03:29:50.120 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.081+0000 7eff66410700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff60155f80 con 0x7eff601578b0 2023-10-06T03:29:50.120 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.081+0000 7eff6540e700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7eff4c003a60 0x7eff6016e940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:50.121 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.077+0000 7ff845b0f700 1 -- 172.21.15.110:0/17861001 shutdown_connections 2023-10-06T03:29:50.121 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.081+0000 7ff845b0f700 1 -- 172.21.15.110:0/17861001 wait complete. 2023-10-06T03:29:50.121 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.077+0000 7fa34b7fe700 1 -- 172.21.15.110:0/3879314431 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3813038000 0 0) 0x7fa34c0c7440 con 0x7fa34c06b7c0 2023-10-06T03:29:50.121 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.077+0000 7fa34b7fe700 1 -- 172.21.15.110:0/3879314431 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa334003570 con 0x7fa34c06b7c0 2023-10-06T03:29:50.122 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.081+0000 7eff6540e700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7eff4c003a60 0x7eff6016e940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.110:50862/0 (socket says 172.21.15.110:50862) 2023-10-06T03:29:50.122 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.081+0000 7eff6540e700 1 -- 172.21.15.110:0/1403710584 learned_addr learned my addr 172.21.15.110:0/1403710584 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:29:50.122 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.081+0000 7f276745d700 1 -- 172.21.15.110:0/768454148 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f27600b0490 con 0x7f276006ab80 2023-10-06T03:29:50.122 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.081+0000 7f27557fa700 1 -- 172.21.15.110:0/768454148 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7f2758014480 con 0x7f276006ab80 2023-10-06T03:29:50.122 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.081+0000 7eff65c0f700 1 --2- 172.21.15.110:0/1403710584 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7eff601578b0 0x7eff60157d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:50.123 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.081+0000 7eff6540e700 1 -- 172.21.15.110:0/1403710584 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7eff601578b0 msgr2=0x7eff60157d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:50.123 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.081+0000 7eff6540e700 1 --2- 172.21.15.110:0/1403710584 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7eff601578b0 0x7eff60157d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:50.123 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.081+0000 7eff6540e700 1 -- 172.21.15.110:0/1403710584 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7eff6014f630 msgr2=0x7eff6014faa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:50.123 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.081+0000 7eff6540e700 1 --2- 172.21.15.110:0/1403710584 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7eff6014f630 0x7eff6014faa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:50.123 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.081+0000 7eff6540e700 1 -- 172.21.15.110:0/1403710584 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff60155650 con 0x7eff4c003a60 2023-10-06T03:29:50.123 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.081+0000 7fb9d9ffb700 1 -- 172.21.15.110:0/3455238065 <== mon.1 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7fb9e002f920 con 0x7fb9e41590a0 2023-10-06T03:29:50.123 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.081+0000 7fb9eaca7700 1 -- 172.21.15.110:0/3455238065 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb9e4070ff0 con 0x7fb9e41590a0 2023-10-06T03:29:50.124 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.081+0000 7fb9d9ffb700 1 --2- 172.21.15.110:0/3455238065 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fb9c8060620 0x7fb9c8062ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:50.124 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.081+0000 7fb9d9ffb700 1 -- 172.21.15.110:0/3455238065 <== mon.1 v2:172.21.15.192:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6809+0+0 (secure 0 0 0) 0x7fb9e007bf30 con 0x7fb9e41590a0 2023-10-06T03:29:50.124 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.081+0000 7fb9e94a4700 1 --2- 172.21.15.110:0/3455238065 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fb9c8060620 0x7fb9c8062ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:50.124 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.081+0000 7ff40659c700 1 --2- 172.21.15.110:0/4163094067 >> v2:172.21.15.192:3300/0 conn(0x7ff408156f90 0x7ff40806cb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:50.124 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.081+0000 7ff40659c700 1 -- 172.21.15.110:0/4163094067 >> v1:172.21.15.192:6789/0 conn(0x7ff408157360 legacy=0x7ff40806a640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:50.124 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.081+0000 7ff40659c700 1 -- 172.21.15.110:0/4163094067 >> v1:172.21.15.110:6789/0 conn(0x7ff4080695e0 legacy=0x7ff4080abd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:29:50.124 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.081+0000 7ff40659c700 1 -- 172.21.15.110:0/4163094067 --> v2:172.21.15.192:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff40806bd80 con 0x7ff408156f90 2023-10-06T03:29:50.124 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.081+0000 7ff40659c700 1 --2- 172.21.15.110:0/4163094067 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff408156f90 0x7ff40806cb40 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7ff3f80098e0 tx=0x7ff3f80091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7a402bd82cfe3bba server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:29:50.125 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.081+0000 7fa34b7fe700 1 -- 172.21.15.110:0/3879314431 <== mon.1 v1:172.21.15.192:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1282108216 0 0) 0x7fa334003570 con 0x7fa34c06b7c0 2023-10-06T03:29:50.125 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.081+0000 7fa350af8700 1 -- 172.21.15.110:0/3879314431 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa34c06c9e0 msgr2=0x7fa34c069a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:29:50.125 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.081+0000 7fa350af8700 1 --2- 172.21.15.110:0/3879314431 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa34c06c9e0 0x7fa34c069a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:50.125 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.081+0000 7fa350af8700 1 -- 172.21.15.110:0/3879314431 >> v1:172.21.15.192:6789/0 conn(0x7fa34c06b7c0 legacy=0x7fa34c06bc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:50.125 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.081+0000 7fa350af8700 1 -- 172.21.15.110:0/3879314431 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa34c0ab990 con 0x7fa34c069640 2023-10-06T03:29:50.125 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.081+0000 7fa34bfff700 1 --2- 172.21.15.110:0/3879314431 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa34c06c9e0 0x7fa34c069a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-06T03:29:50.125 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.081+0000 7eff6540e700 1 --2- 172.21.15.110:0/1403710584 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7eff4c003a60 0x7eff6016e940 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7eff5c002d30 tx=0x7eff5c017000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:29:50.126 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.081+0000 7fb9e94a4700 1 --2- 172.21.15.110:0/3455238065 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fb9c8060620 0x7fb9c8062ad0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fb9d4001200 tx=0x7fb9d40079d0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:29:50.126 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.081+0000 7fa350af8700 1 --2- 172.21.15.110:0/3879314431 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa34c069640 0x7fa34c06a880 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fa3400098e0 tx=0x7fa3400091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d09f155e35eeada7 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:29:50.126 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.081+0000 7f00b3d9e700 1 -- 172.21.15.110:0/669385157 shutdown_connections 2023-10-06T03:29:50.127 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.081+0000 7f27557fa700 1 --2- 172.21.15.110:0/768454148 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f2744060620 0x7f2744062ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:50.127 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.081+0000 7f00b3d9e700 1 -- 172.21.15.110:0/669385157 wait complete. 2023-10-06T03:29:50.128 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.081+0000 7ff845b0f700 1 Processor -- start 2023-10-06T03:29:50.128 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.081+0000 7ff845b0f700 1 -- start start 2023-10-06T03:29:50.128 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.081+0000 7ff845b0f700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff8400c20d0 0x7ff8400ad2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:50.128 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.081+0000 7ff845b0f700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff8400c8c10 0x7ff840141f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:50.128 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.081+0000 7ff845b0f700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff8400c9750 0x7ff840142460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:50.129 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.081+0000 7ff845b0f700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8400adf00 con 0x7ff8400c8c10 2023-10-06T03:29:50.129 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.081+0000 7f27557fa700 1 -- 172.21.15.110:0/768454148 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6809+0+0 (secure 0 0 0) 0x7f275807d150 con 0x7f276006ab80 2023-10-06T03:29:50.129 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.081+0000 7f27649f8700 1 --2- 172.21.15.110:0/768454148 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f2744060620 0x7f2744062ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:50.129 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.081+0000 7f27649f8700 1 --2- 172.21.15.110:0/768454148 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f2744060620 0x7f2744062ad0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f27600c71a0 tx=0x7f2750017000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:29:50.129 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.081+0000 7f00b3d9e700 1 Processor -- start 2023-10-06T03:29:50.130 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.081+0000 7ff406d9d700 1 --1- 172.21.15.110:0/4163094067 >> v1:172.21.15.110:6789/0 conn(0x7ff4080695e0 0x7ff4080abd80 :-1 s=CLOSED pgs=0 cs=0 l=0).handle_server_banner_and_identify state changed while learned_addr, mark_down or replacing must be happened just now 2023-10-06T03:29:50.130 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.081+0000 7ff845b0f700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff840155d10 con 0x7ff8400c9750 2023-10-06T03:29:50.130 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.081+0000 7ff845b0f700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8400adc20 con 0x7ff8400c20d0 2023-10-06T03:29:50.130 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.081+0000 7ff83effd700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff8400c8c10 0x7ff840141f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:50.130 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.085+0000 7f00b3d9e700 1 -- start start 2023-10-06T03:29:50.131 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.081+0000 7ff83effd700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff8400c8c10 0x7ff840141f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:60750/0 (socket says 172.21.15.110:60750) 2023-10-06T03:29:50.131 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.081+0000 7ff83effd700 1 -- 172.21.15.110:0/2650449895 learned_addr learned my addr 172.21.15.110:0/2650449895 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:29:50.131 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.081+0000 7ff83f7fe700 1 --2- 172.21.15.110:0/2650449895 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff8400c20d0 0x7ff8400ad2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:50.131 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.081+0000 7ff83ffff700 1 --2- 172.21.15.110:0/2650449895 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff8400c9750 0x7ff840142460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:50.131 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.085+0000 7ff83effd700 1 -- 172.21.15.110:0/2650449895 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff8400c20d0 msgr2=0x7ff8400ad2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:50.132 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.085+0000 7ff83effd700 1 --2- 172.21.15.110:0/2650449895 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff8400c20d0 0x7ff8400ad2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:50.132 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.085+0000 7ff83effd700 1 -- 172.21.15.110:0/2650449895 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff8400c9750 msgr2=0x7ff840142460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:50.132 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.085+0000 7ff83effd700 1 --2- 172.21.15.110:0/2650449895 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff8400c9750 0x7ff840142460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:50.132 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.085+0000 7ff83effd700 1 -- 172.21.15.110:0/2650449895 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8340094f0 con 0x7ff8400c8c10 2023-10-06T03:29:50.132 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.085+0000 7ff83f7fe700 1 --2- 172.21.15.110:0/2650449895 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff8400c20d0 0x7ff8400ad2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-06T03:29:50.132 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.085+0000 7ff83effd700 1 --2- 172.21.15.110:0/2650449895 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff8400c8c10 0x7ff840141f20 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7ff8400c9d50 tx=0x7ff834016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:29:50.134 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.085+0000 7f00b3d9e700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f00ac06b960 0x7f00ac151ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:50.134 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.085+0000 7f00b3d9e700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f00ac06ec20 0x7f00ac0c5220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:50.134 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.085+0000 7f00b3d9e700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f00ac152ff0 0x7f00ac1394c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:50.134 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.085+0000 7f00b3d9e700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00ac152430 con 0x7f00ac06ec20 2023-10-06T03:29:50.134 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.085+0000 7f00b3d9e700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00ac16d5b0 con 0x7f00ac06b960 2023-10-06T03:29:50.134 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.085+0000 7f00b3d9e700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00ac16ccc0 con 0x7f00ac152ff0 2023-10-06T03:29:50.134 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.085+0000 7f00b1339700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f00ac06ec20 0x7f00ac0c5220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:50.134 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.085+0000 7f00b1339700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f00ac06ec20 0x7f00ac0c5220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:60754/0 (socket says 172.21.15.110:60754) 2023-10-06T03:29:50.134 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.085+0000 7f00b1339700 1 -- 172.21.15.110:0/3589528979 learned_addr learned my addr 172.21.15.110:0/3589528979 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:29:50.135 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.085+0000 7f00b1b3a700 1 --2- 172.21.15.110:0/3589528979 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f00ac06b960 0x7f00ac151ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:50.135 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.085+0000 7f00b233b700 1 --2- 172.21.15.110:0/3589528979 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f00ac152ff0 0x7f00ac1394c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:50.135 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.085+0000 7f00b1339700 1 -- 172.21.15.110:0/3589528979 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f00ac152ff0 msgr2=0x7f00ac1394c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:50.135 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.085+0000 7f00b1339700 1 --2- 172.21.15.110:0/3589528979 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f00ac152ff0 0x7f00ac1394c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:50.135 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.085+0000 7f00b1339700 1 -- 172.21.15.110:0/3589528979 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f00ac06b960 msgr2=0x7f00ac151ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:50.135 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.085+0000 7f00b1339700 1 --2- 172.21.15.110:0/3589528979 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f00ac06b960 0x7f00ac151ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:50.135 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.085+0000 7f00b1339700 1 -- 172.21.15.110:0/3589528979 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f009c009420 con 0x7f00ac06ec20 2023-10-06T03:29:50.135 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.085+0000 7f00b1339700 1 --2- 172.21.15.110:0/3589528979 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f00ac06ec20 0x7f00ac0c5220 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f00a800b5e0 tx=0x7f00a80090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:29:50.136 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.089+0000 7ff34b59e700 1 Processor -- start 2023-10-06T03:29:50.136 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.089+0000 7ff34b59e700 1 -- start start 2023-10-06T03:29:50.137 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.089+0000 7ff34b59e700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7ff34c06a970 0x7ff34c069a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:50.137 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.089+0000 7ff34b59e700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7ff34c06bed0 0x7ff34c0acd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:50.137 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.089+0000 7ff34b59e700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff34c0ac8a0 con 0x7ff34c069640 2023-10-06T03:29:50.137 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.089+0000 7ff34b59e700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff34c0ae280 con 0x7ff34c06a970 2023-10-06T03:29:50.137 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.089+0000 7ff34b59e700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff34c06d170 con 0x7ff34c06bed0 2023-10-06T03:29:50.137 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.089+0000 7ff349d9b700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7ff34c06a970 0x7ff34c069a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:50.137 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.089+0000 7ff34a59c700 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7ff34c069640 0x7ff34c06b7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:37356/0 (socket says 172.21.15.110:37356) 2023-10-06T03:29:50.137 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.089+0000 7ff34a59c700 1 -- 172.21.15.110:0/2373213730 learned_addr learned my addr 172.21.15.110:0/2373213730 (peer_addr_for_me v1:172.21.15.110:0/0) 2023-10-06T03:29:50.138 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.089+0000 7ff34ad9d700 1 --2- 172.21.15.110:0/2373213730 >> v2:172.21.15.192:3300/0 conn(0x7ff34c06bed0 0x7ff34c0acd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:50.138 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.089+0000 7ff8277fe700 1 -- 172.21.15.110:0/2650449895 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff83400b8d0 con 0x7ff8400c8c10 2023-10-06T03:29:50.138 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.089+0000 7fa34b7fe700 1 -- 172.21.15.110:0/3879314431 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa33c001490 con 0x7fa34c069640 2023-10-06T03:29:50.139 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.089+0000 7ff349d9b700 1 -- 172.21.15.110:0/2373213730 >> v2:172.21.15.192:3300/0 conn(0x7ff34c06bed0 msgr2=0x7ff34c0acd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:29:50.139 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.089+0000 7ff349d9b700 1 --2- 172.21.15.110:0/2373213730 >> v2:172.21.15.192:3300/0 conn(0x7ff34c06bed0 0x7ff34c0acd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:50.139 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.089+0000 7f00a1ffb700 1 -- 172.21.15.110:0/3589528979 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00a800f510 con 0x7f00ac06ec20 2023-10-06T03:29:50.140 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.089+0000 7ff349d9b700 1 -- 172.21.15.110:0/2373213730 >> v1:172.21.15.110:6789/0 conn(0x7ff34c069640 legacy=0x7ff34c06b7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:50.140 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.089+0000 7ff349d9b700 1 -- 172.21.15.110:0/2373213730 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff34c0c7760 con 0x7ff34c06a970 2023-10-06T03:29:50.140 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.089+0000 7fa34b7fe700 1 -- 172.21.15.110:0/3879314431 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa340011420 con 0x7fa34c069640 2023-10-06T03:29:50.140 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.089+0000 7fa34b7fe700 1 -- 172.21.15.110:0/3879314431 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3400117a0 con 0x7fa34c069640 2023-10-06T03:29:50.140 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.089+0000 7ff40559a700 1 -- 172.21.15.110:0/4163094067 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3f8009e80 con 0x7ff408156f90 2023-10-06T03:29:50.141 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.089+0000 7ff40559a700 1 -- 172.21.15.110:0/4163094067 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3f800b820 con 0x7ff408156f90 2023-10-06T03:29:50.141 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.089+0000 7ff40559a700 1 -- 172.21.15.110:0/4163094067 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3f800bba0 con 0x7ff408156f90 2023-10-06T03:29:50.141 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.089+0000 7ff349d9b700 1 --2- 172.21.15.110:0/2373213730 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff34c06a970 0x7ff34c069a10 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7ff34c0ae6a0 tx=0x7ff344009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be2b3eca19a4964b server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:29:50.141 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.089+0000 7f27557fa700 1 -- 172.21.15.110:0/768454148 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f27600b0490 con 0x7f276006ab80 2023-10-06T03:29:50.142 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.089+0000 7f00a1ffb700 1 -- 172.21.15.110:0/3589528979 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f00a800f6b0 con 0x7f00ac06ec20 2023-10-06T03:29:50.142 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.089+0000 7ff845b0f700 1 -- 172.21.15.110:0/2650449895 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8401556c0 con 0x7ff8400c8c10 2023-10-06T03:29:50.142 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.089+0000 7ff845b0f700 1 -- 172.21.15.110:0/2650449895 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8400ad820 con 0x7ff8400c8c10 2023-10-06T03:29:50.142 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.089+0000 7ff8277fe700 1 -- 172.21.15.110:0/2650449895 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff83401c460 con 0x7ff8400c8c10 2023-10-06T03:29:50.143 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.089+0000 7ff8277fe700 1 -- 172.21.15.110:0/2650449895 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff83401c5d0 con 0x7ff8400c8c10 2023-10-06T03:29:50.143 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.089+0000 7fa351afa700 1 -- 172.21.15.110:0/3879314431 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa34c069640 msgr2=0x7fa34c06a880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:50.143 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.089+0000 7fa351afa700 1 --2- 172.21.15.110:0/3879314431 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa34c069640 0x7fa34c06a880 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fa3400098e0 tx=0x7fa3400091a0 comp rx=0 tx=0).stop 2023-10-06T03:29:50.143 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.089+0000 7ff40c8dd700 1 -- 172.21.15.110:0/4163094067 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff408156f90 msgr2=0x7ff40806cb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:50.143 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.089+0000 7ff40c8dd700 1 --2- 172.21.15.110:0/4163094067 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff408156f90 0x7ff40806cb40 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7ff3f80098e0 tx=0x7ff3f80091a0 comp rx=0 tx=0).stop 2023-10-06T03:29:50.144 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.089+0000 7fb9d9ffb700 1 -- 172.21.15.110:0/3455238065 <== mon.1 v2:172.21.15.192:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fb9e00abbe0 con 0x7fb9e41590a0 2023-10-06T03:29:50.144 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.089+0000 7f00b3d9e700 1 -- 172.21.15.110:0/3589528979 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00ac16cf60 con 0x7f00ac06ec20 2023-10-06T03:29:50.144 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.089+0000 7f00b3d9e700 1 -- 172.21.15.110:0/3589528979 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00ac0c5ce0 con 0x7f00ac06ec20 2023-10-06T03:29:50.144 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.093+0000 7f00a1ffb700 1 -- 172.21.15.110:0/3589528979 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00a800fa70 con 0x7f00ac06ec20 2023-10-06T03:29:50.145 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.093+0000 7eff557fa700 1 -- 172.21.15.110:0/1403710584 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff5c001760 con 0x7eff4c003a60 2023-10-06T03:29:50.145 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.093+0000 7eff557fa700 1 -- 172.21.15.110:0/1403710584 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff5c0018c0 con 0x7eff4c003a60 2023-10-06T03:29:50.145 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.089+0000 7ff34959a700 1 -- 172.21.15.110:0/2373213730 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff34400bb60 con 0x7ff34c06a970 2023-10-06T03:29:50.146 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.093+0000 7ff40c8dd700 1 -- 172.21.15.110:0/4163094067 shutdown_connections 2023-10-06T03:29:50.146 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.093+0000 7ff40c8dd700 1 --2- 172.21.15.110:0/4163094067 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff408156f90 0x7ff40806cb40 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:50.146 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.093+0000 7ff40c8dd700 1 -- 172.21.15.110:0/4163094067 >> 172.21.15.110:0/4163094067 conn(0x7ff4080cc6d0 msgr2=0x7ff4080c8670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:29:50.147 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.089+0000 7ff34959a700 1 -- 172.21.15.110:0/2373213730 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff34400bcc0 con 0x7ff34c06a970 2023-10-06T03:29:50.147 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.093+0000 7fa351afa700 1 -- 172.21.15.110:0/3879314431 shutdown_connections 2023-10-06T03:29:50.147 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.093+0000 7fa351afa700 1 --2- 172.21.15.110:0/3879314431 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa34c06c9e0 0x7fa34c069a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:50.147 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.093+0000 7fa351afa700 1 --2- 172.21.15.110:0/3879314431 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa34c069640 0x7fa34c06a880 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:50.147 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.093+0000 7fa351afa700 1 -- 172.21.15.110:0/3879314431 >> 172.21.15.110:0/3879314431 conn(0x7fa34c06e760 msgr2=0x7fa34c0cba30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:29:50.148 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.093+0000 7eff557fa700 1 -- 172.21.15.110:0/1403710584 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff5c013470 con 0x7eff4c003a60 2023-10-06T03:29:50.148 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.093+0000 7ff40c8dd700 1 -- 172.21.15.110:0/4163094067 shutdown_connections 2023-10-06T03:29:50.148 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.093+0000 7ff40c8dd700 1 -- 172.21.15.110:0/4163094067 wait complete. 2023-10-06T03:29:50.149 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.093+0000 7ff34b59e700 1 -- 172.21.15.110:0/2373213730 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff34c06a970 msgr2=0x7ff34c069a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:50.149 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.093+0000 7ff34959a700 1 -- 172.21.15.110:0/2373213730 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff34400bb60 con 0x7ff34c06a970 2023-10-06T03:29:50.150 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.093+0000 7ff34b59e700 1 --2- 172.21.15.110:0/2373213730 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff34c06a970 0x7ff34c069a10 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7ff34c0ae6a0 tx=0x7ff344009780 comp rx=0 tx=0).stop 2023-10-06T03:29:50.150 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.093+0000 7ff34b59e700 1 -- 172.21.15.110:0/2373213730 shutdown_connections 2023-10-06T03:29:50.150 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.093+0000 7ff34b59e700 1 --2- 172.21.15.110:0/2373213730 >> v2:172.21.15.192:3300/0 conn(0x7ff34c06bed0 0x7ff34c0acd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:50.150 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.093+0000 7ff34b59e700 1 --2- 172.21.15.110:0/2373213730 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff34c06a970 0x7ff34c069a10 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:50.150 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.093+0000 7ff34b59e700 1 -- 172.21.15.110:0/2373213730 >> 172.21.15.110:0/2373213730 conn(0x7ff34c06dce0 msgr2=0x7ff34c06e0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:29:50.150 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.093+0000 7ff34b59e700 1 -- 172.21.15.110:0/2373213730 shutdown_connections 2023-10-06T03:29:50.150 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.093+0000 7ff34b59e700 1 -- 172.21.15.110:0/2373213730 wait complete. 2023-10-06T03:29:50.150 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.093+0000 7ff40c8dd700 1 Processor -- start 2023-10-06T03:29:50.151 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.093+0000 7f00b3d9e700 1 -- 172.21.15.110:0/3589528979 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f00ac0c5f60 con 0x7f00ac06ec20 2023-10-06T03:29:50.151 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.093+0000 7eff66410700 1 -- 172.21.15.110:0/1403710584 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff60155930 con 0x7eff4c003a60 2023-10-06T03:29:50.151 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.093+0000 7eff66410700 1 -- 172.21.15.110:0/1403710584 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff6016ee80 con 0x7eff4c003a60 2023-10-06T03:29:50.151 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.093+0000 7eff66410700 1 -- 172.21.15.110:0/1403710584 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff60070ff0 con 0x7eff4c003a60 2023-10-06T03:29:50.152 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.093+0000 7eff557fa700 1 -- 172.21.15.110:0/1403710584 <== mon.1 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7eff5c042a20 con 0x7eff4c003a60 2023-10-06T03:29:50.152 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.093+0000 7f00a1ffb700 1 -- 172.21.15.110:0/3589528979 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7f00a801a400 con 0x7f00ac06ec20 2023-10-06T03:29:50.152 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.093+0000 7f00a1ffb700 1 --2- 172.21.15.110:0/3589528979 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f0090060550 0x7f0090062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:50.152 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.097+0000 7f00a1ffb700 1 -- 172.21.15.110:0/3589528979 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6809+0+0 (secure 0 0 0) 0x7f00a807c710 con 0x7f00ac06ec20 2023-10-06T03:29:50.152 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.097+0000 7f00b1b3a700 1 --2- 172.21.15.110:0/3589528979 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f0090060550 0x7f0090062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:50.152 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.097+0000 7ff40c8dd700 1 -- start start 2023-10-06T03:29:50.153 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.097+0000 7ff34b59e700 1 Processor -- start 2023-10-06T03:29:50.153 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.097+0000 7ff34b59e700 1 -- start start 2023-10-06T03:29:50.153 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.097+0000 7eff557fa700 1 --2- 172.21.15.110:0/1403710584 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7eff4c062100 0x7eff4c0645b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:50.153 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.097+0000 7eff557fa700 1 -- 172.21.15.110:0/1403710584 <== mon.1 v2:172.21.15.192:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6809+0+0 (secure 0 0 0) 0x7eff5c07da80 con 0x7eff4c003a60 2023-10-06T03:29:50.154 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.097+0000 7eff64c0d700 1 --2- 172.21.15.110:0/1403710584 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7eff4c062100 0x7eff4c0645b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:50.154 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.097+0000 7f00b1b3a700 1 --2- 172.21.15.110:0/3589528979 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f0090060550 0x7f0090062a00 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f009c0098e0 tx=0x7f009c016000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:29:50.154 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.097+0000 7eff64c0d700 1 --2- 172.21.15.110:0/1403710584 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7eff4c062100 0x7eff4c0645b0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7eff50006fd0 tx=0x7eff50008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:29:50.155 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.097+0000 7ff40c8dd700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff4080695e0 0x7ff4081578e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:50.155 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.097+0000 7ff40c8dd700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff408156f90 0x7ff4080b1380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:50.155 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.097+0000 7ff40c8dd700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff408157360 0x7ff408159160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:50.155 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.097+0000 7ff40c8dd700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4080b18c0 con 0x7ff408156f90 2023-10-06T03:29:50.155 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.097+0000 7ff40c8dd700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4080aa350 con 0x7ff4080695e0 2023-10-06T03:29:50.155 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.097+0000 7ff40c8dd700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4080aa630 con 0x7ff408157360 2023-10-06T03:29:50.156 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.097+0000 7ff34b59e700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff34c069640 0x7ff34c0c34c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:50.156 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.097+0000 7ff34b59e700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff34c06a970 0x7ff34c152420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:50.156 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.097+0000 7ff34b59e700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff34c06bed0 0x7ff34c1595e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:50.156 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.097+0000 7ff34b59e700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff34c0a5860 con 0x7ff34c06bed0 2023-10-06T03:29:50.156 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.097+0000 7ff34b59e700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff34c0c48f0 con 0x7ff34c06a970 2023-10-06T03:29:50.156 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.097+0000 7ff34b59e700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff34c0c4bd0 con 0x7ff34c069640 2023-10-06T03:29:50.157 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.097+0000 7ff40659c700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff4080695e0 0x7ff4081578e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:50.157 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.097+0000 7ff406d9d700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff408157360 0x7ff408159160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:50.157 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.097+0000 7ff40659c700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff4080695e0 0x7ff4081578e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.110:50884/0 (socket says 172.21.15.110:50884) 2023-10-06T03:29:50.157 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.097+0000 7ff40659c700 1 -- 172.21.15.110:0/1147892266 learned_addr learned my addr 172.21.15.110:0/1147892266 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:29:50.157 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.097+0000 7ff406d9d700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff408157360 0x7ff408159160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3301/0 says I am v2:172.21.15.110:44250/0 (socket says 172.21.15.110:44250) 2023-10-06T03:29:50.157 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.097+0000 7ff405d9b700 1 --2- 172.21.15.110:0/1147892266 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff408156f90 0x7ff4080b1380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:50.158 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.097+0000 7ff349d9b700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff34c06a970 0x7ff34c152420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:50.158 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.097+0000 7ff34a59c700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff34c069640 0x7ff34c0c34c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:50.158 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.097+0000 7ff34ad9d700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff34c06bed0 0x7ff34c1595e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:50.158 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.097+0000 7ff34ad9d700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff34c06bed0 0x7ff34c1595e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:60780/0 (socket says 172.21.15.110:60780) 2023-10-06T03:29:50.158 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.097+0000 7ff34a59c700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff34c069640 0x7ff34c0c34c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3301/0 says I am v2:172.21.15.110:44266/0 (socket says 172.21.15.110:44266) 2023-10-06T03:29:50.158 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.097+0000 7ff34ad9d700 1 -- 172.21.15.110:0/1155933271 learned_addr learned my addr 172.21.15.110:0/1155933271 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:29:50.158 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.097+0000 7ff349d9b700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff34c06a970 0x7ff34c152420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.110:50890/0 (socket says 172.21.15.110:50890) 2023-10-06T03:29:50.158 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.097+0000 7ff34ad9d700 1 -- 172.21.15.110:0/1155933271 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff34c069640 msgr2=0x7ff34c0c34c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:50.158 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.097+0000 7ff34ad9d700 1 --2- 172.21.15.110:0/1155933271 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff34c069640 0x7ff34c0c34c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:50.159 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.097+0000 7ff34ad9d700 1 -- 172.21.15.110:0/1155933271 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff34c06a970 msgr2=0x7ff34c152420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:50.159 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.097+0000 7ff34ad9d700 1 --2- 172.21.15.110:0/1155933271 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff34c06a970 0x7ff34c152420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:50.159 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.097+0000 7ff34ad9d700 1 -- 172.21.15.110:0/1155933271 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff344009a20 con 0x7ff34c06bed0 2023-10-06T03:29:50.159 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.097+0000 7ff34a59c700 1 --2- 172.21.15.110:0/1155933271 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff34c069640 0x7ff34c0c34c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-06T03:29:50.159 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.097+0000 7ff40659c700 1 -- 172.21.15.110:0/1147892266 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff408157360 msgr2=0x7ff408159160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:50.159 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.097+0000 7ff40659c700 1 --2- 172.21.15.110:0/1147892266 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff408157360 0x7ff408159160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:50.159 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.097+0000 7ff40659c700 1 -- 172.21.15.110:0/1147892266 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff408156f90 msgr2=0x7ff4080b1380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:50.160 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.097+0000 7ff40659c700 1 --2- 172.21.15.110:0/1147892266 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff408156f90 0x7ff4080b1380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:50.160 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.097+0000 7ff40659c700 1 -- 172.21.15.110:0/1147892266 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3f8009420 con 0x7ff4080695e0 2023-10-06T03:29:50.160 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.097+0000 7ff406d9d700 1 --2- 172.21.15.110:0/1147892266 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff408157360 0x7ff408159160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-06T03:29:50.160 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.097+0000 7ff34ad9d700 1 --2- 172.21.15.110:0/1155933271 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff34c06bed0 0x7ff34c1595e0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7ff34000ea50 tx=0x7ff3400091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:29:50.160 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.097+0000 7ff40659c700 1 --2- 172.21.15.110:0/1147892266 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff4080695e0 0x7ff4081578e0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7ff408068ee0 tx=0x7ff3f8016000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:29:50.161 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.093+0000 7ff845b0f700 1 -- 172.21.15.110:0/2650449895 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff8200052b0 con 0x7ff8400c8c10 2023-10-06T03:29:50.161 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.093+0000 7ff8277fe700 1 -- 172.21.15.110:0/2650449895 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7ff83401c730 con 0x7ff8400c8c10 2023-10-06T03:29:50.161 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.093+0000 7ff8277fe700 1 --2- 172.21.15.110:0/2650449895 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7ff82c060550 0x7ff82c062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:50.161 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.093+0000 7ff8277fe700 1 -- 172.21.15.110:0/2650449895 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6809+0+0 (secure 0 0 0) 0x7ff83407ee40 con 0x7ff8400c8c10 2023-10-06T03:29:50.161 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.097+0000 7ff83f7fe700 1 --2- 172.21.15.110:0/2650449895 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7ff82c060550 0x7ff82c062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:50.161 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.097+0000 7ff83f7fe700 1 --2- 172.21.15.110:0/2650449895 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7ff82c060550 0x7ff82c062a00 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7ff830001a20 tx=0x7ff830009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:29:50.162 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.101+0000 7fa351afa700 1 -- 172.21.15.110:0/3879314431 shutdown_connections 2023-10-06T03:29:50.162 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.101+0000 7fa351afa700 1 -- 172.21.15.110:0/3879314431 wait complete. 2023-10-06T03:29:50.163 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.101+0000 7ff3f67fc700 1 -- 172.21.15.110:0/1147892266 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3f8009e80 con 0x7ff4080695e0 2023-10-06T03:29:50.163 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.101+0000 7ff40c8dd700 1 -- 172.21.15.110:0/1147892266 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4080aa910 con 0x7ff4080695e0 2023-10-06T03:29:50.163 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.101+0000 7ff40c8dd700 1 -- 172.21.15.110:0/1147892266 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4080aaed0 con 0x7ff4080695e0 2023-10-06T03:29:50.163 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.101+0000 7eff557fa700 1 -- 172.21.15.110:0/1403710584 <== mon.1 v2:172.21.15.192:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7eff5c04dc00 con 0x7eff4c003a60 2023-10-06T03:29:50.163 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.101+0000 7fa351afa700 1 Processor -- start 2023-10-06T03:29:50.164 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.101+0000 7fa351afa700 1 -- start start 2023-10-06T03:29:50.164 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.101+0000 7ff3f67fc700 1 -- 172.21.15.110:0/1147892266 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3f800bba0 con 0x7ff4080695e0 2023-10-06T03:29:50.164 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.101+0000 7ff3f67fc700 1 -- 172.21.15.110:0/1147892266 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3f801a690 con 0x7ff4080695e0 2023-10-06T03:29:50.164 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.101+0000 7ff3f67fc700 1 -- 172.21.15.110:0/1147892266 <== mon.1 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7ff3f801aa80 con 0x7ff4080695e0 2023-10-06T03:29:50.164 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.101+0000 7ff3f67fc700 1 --2- 172.21.15.110:0/1147892266 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7ff3e8060550 0x7ff3e8062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:50.164 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.101+0000 7ff405d9b700 1 --2- 172.21.15.110:0/1147892266 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7ff3e8060550 0x7ff3e8062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:50.165 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.101+0000 7fa351afa700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa34c069640 0x7fa34c161ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:50.165 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.101+0000 7fa351afa700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa34c06b7c0 0x7fa34c15e390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:50.165 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.101+0000 7fa351afa700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa34c06c9e0 0x7fa34c0c6110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:50.166 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.101+0000 7fa351afa700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa34c0c3d30 con 0x7fa34c06c9e0 2023-10-06T03:29:50.166 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.101+0000 7fa351afa700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa34c15e8d0 con 0x7fa34c069640 2023-10-06T03:29:50.166 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.101+0000 7fa351afa700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa34c160f50 con 0x7fa34c06b7c0 2023-10-06T03:29:50.166 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.105+0000 7fa3512f9700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa34c06c9e0 0x7fa34c0c6110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:50.166 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.105+0000 7fa3512f9700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa34c06c9e0 0x7fa34c0c6110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:60792/0 (socket says 172.21.15.110:60792) 2023-10-06T03:29:50.166 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.105+0000 7fa3512f9700 1 -- 172.21.15.110:0/2046814958 learned_addr learned my addr 172.21.15.110:0/2046814958 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:29:50.166 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.105+0000 7fa34bfff700 1 --2- 172.21.15.110:0/2046814958 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa34c06b7c0 0x7fa34c15e390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:50.166 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.105+0000 7fa350af8700 1 --2- 172.21.15.110:0/2046814958 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa34c069640 0x7fa34c161ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:50.167 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.105+0000 7ff3f67fc700 1 -- 172.21.15.110:0/1147892266 <== mon.1 v2:172.21.15.192:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6809+0+0 (secure 0 0 0) 0x7ff3f807e9e0 con 0x7ff4080695e0 2023-10-06T03:29:50.167 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.105+0000 7fa3512f9700 1 -- 172.21.15.110:0/2046814958 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa34c06b7c0 msgr2=0x7fa34c15e390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:50.167 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.105+0000 7fa3512f9700 1 --2- 172.21.15.110:0/2046814958 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa34c06b7c0 0x7fa34c15e390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:50.167 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.105+0000 7fa3512f9700 1 -- 172.21.15.110:0/2046814958 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa34c069640 msgr2=0x7fa34c161ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:50.167 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.105+0000 7fa3512f9700 1 --2- 172.21.15.110:0/2046814958 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa34c069640 0x7fa34c161ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:50.167 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.105+0000 7fa3512f9700 1 -- 172.21.15.110:0/2046814958 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa344009a20 con 0x7fa34c06c9e0 2023-10-06T03:29:50.168 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.105+0000 7ff405d9b700 1 --2- 172.21.15.110:0/1147892266 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7ff3e8060550 0x7ff3e8062a00 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7ff40806f100 tx=0x7ff400001650 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:29:50.168 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.109+0000 7ff3effff700 1 -- 172.21.15.110:0/1147892266 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3e40052b0 con 0x7ff4080695e0 2023-10-06T03:29:50.168 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.105+0000 7ff8277fe700 1 -- 172.21.15.110:0/2650449895 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7ff83404efe0 con 0x7ff8400c8c10 2023-10-06T03:29:50.168 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.105+0000 7fa3512f9700 1 --2- 172.21.15.110:0/2046814958 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa34c06c9e0 0x7fa34c0c6110 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fa33c00cb90 tx=0x7fa33c001ac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:29:50.169 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.113+0000 7f00a1ffb700 1 -- 172.21.15.110:0/3589528979 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f00a804c820 con 0x7f00ac06ec20 2023-10-06T03:29:50.169 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.113+0000 7ff33a7fc700 1 -- 172.21.15.110:0/1155933271 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff34000ee90 con 0x7ff34c06bed0 2023-10-06T03:29:50.170 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.113+0000 7ff34b59e700 1 -- 172.21.15.110:0/1155933271 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff34c0c4eb0 con 0x7ff34c06bed0 2023-10-06T03:29:50.170 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.113+0000 7ff34b59e700 1 -- 172.21.15.110:0/1155933271 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff34c0c5470 con 0x7ff34c06bed0 2023-10-06T03:29:50.170 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.113+0000 7ff33a7fc700 1 -- 172.21.15.110:0/1155933271 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff34000c850 con 0x7ff34c06bed0 2023-10-06T03:29:50.170 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.113+0000 7ff33a7fc700 1 -- 172.21.15.110:0/1155933271 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff34000cc10 con 0x7ff34c06bed0 2023-10-06T03:29:50.170 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.113+0000 7fa348ff9700 1 -- 172.21.15.110:0/2046814958 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa33c007150 con 0x7fa34c06c9e0 2023-10-06T03:29:50.170 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.113+0000 7fa351afa700 1 -- 172.21.15.110:0/2046814958 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa340009420 con 0x7fa34c06c9e0 2023-10-06T03:29:50.171 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.113+0000 7fa351afa700 1 -- 172.21.15.110:0/2046814958 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa34c161510 con 0x7fa34c06c9e0 2023-10-06T03:29:50.171 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.117+0000 7ff33a7fc700 1 -- 172.21.15.110:0/1155933271 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7ff340013470 con 0x7ff34c06bed0 2023-10-06T03:29:50.171 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.117+0000 7ff33a7fc700 1 --2- 172.21.15.110:0/1155933271 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7ff32c0605d0 0x7ff32c062a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:50.171 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.117+0000 7ff33a7fc700 1 -- 172.21.15.110:0/1155933271 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6809+0+0 (secure 0 0 0) 0x7ff34007ce30 con 0x7ff34c06bed0 2023-10-06T03:29:50.171 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.117+0000 7ff34a59c700 1 --2- 172.21.15.110:0/1155933271 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7ff32c0605d0 0x7ff32c062a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:50.171 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.117+0000 7ff34a59c700 1 --2- 172.21.15.110:0/1155933271 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7ff32c0605d0 0x7ff32c062a80 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7ff33c007110 tx=0x7ff33c007f10 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:29:50.172 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.117+0000 7fa348ff9700 1 -- 172.21.15.110:0/2046814958 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa33c0072b0 con 0x7fa34c06c9e0 2023-10-06T03:29:50.172 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.117+0000 7fa348ff9700 1 -- 172.21.15.110:0/2046814958 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa33c007670 con 0x7fa34c06c9e0 2023-10-06T03:29:50.172 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.117+0000 7fa348ff9700 1 -- 172.21.15.110:0/2046814958 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7fa33c004a40 con 0x7fa34c06c9e0 2023-10-06T03:29:50.172 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.117+0000 7ff3f67fc700 1 -- 172.21.15.110:0/1147892266 <== mon.1 v2:172.21.15.192:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7ff3f804eb80 con 0x7ff4080695e0 2023-10-06T03:29:50.173 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.117+0000 7fa348ff9700 1 --2- 172.21.15.110:0/2046814958 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fa338060580 0x7fa338062a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:50.173 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.117+0000 7fa348ff9700 1 -- 172.21.15.110:0/2046814958 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6809+0+0 (secure 0 0 0) 0x7fa33c07c240 con 0x7fa34c06c9e0 2023-10-06T03:29:50.173 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.117+0000 7fa350af8700 1 --2- 172.21.15.110:0/2046814958 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fa338060580 0x7fa338062a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:50.173 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.117+0000 7fa350af8700 1 --2- 172.21.15.110:0/2046814958 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fa338060580 0x7fa338062a30 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fa3400091e0 tx=0x7fa340016000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:29:50.174 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.121+0000 7ff34b59e700 1 -- 172.21.15.110:0/1155933271 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3300052b0 con 0x7ff34c06bed0 2023-10-06T03:29:50.174 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.129+0000 7ff33a7fc700 1 -- 172.21.15.110:0/1155933271 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7ff34004cf30 con 0x7ff34c06bed0 2023-10-06T03:29:50.174 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.121+0000 7fa351afa700 1 -- 172.21.15.110:0/2046814958 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa32c0052b0 con 0x7fa34c06c9e0 2023-10-06T03:29:50.174 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.137+0000 7fa348ff9700 1 -- 172.21.15.110:0/2046814958 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fa33c018080 con 0x7fa34c06c9e0 2023-10-06T03:29:50.404 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.401+0000 7fb9eaca7700 1 -- 172.21.15.110:0/3455238065 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7fb9e40cc700 con 0x7fb9e41590a0 2023-10-06T03:29:50.404 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.401+0000 7fb9d9ffb700 1 -- 172.21.15.110:0/3455238065 <== mon.1 v2:172.21.15.192:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fb9e004fcc0 con 0x7fb9e41590a0 2023-10-06T03:29:50.406 INFO:teuthology.orchestra.run.smithi110.stdout:60129542151 2023-10-06T03:29:50.408 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.405+0000 7fb9cf7fe700 1 -- 172.21.15.110:0/3455238065 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fb9c8060620 msgr2=0x7fb9c8062ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:50.408 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.405+0000 7fb9cf7fe700 1 --2- 172.21.15.110:0/3455238065 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fb9c8060620 0x7fb9c8062ad0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fb9d4001200 tx=0x7fb9d40079d0 comp rx=0 tx=0).stop 2023-10-06T03:29:50.408 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.405+0000 7fb9cf7fe700 1 -- 172.21.15.110:0/3455238065 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb9e41590a0 msgr2=0x7fb9e40c8320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:50.408 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.405+0000 7fb9cf7fe700 1 --2- 172.21.15.110:0/3455238065 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb9e41590a0 0x7fb9e40c8320 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fb9e000b600 tx=0x7fb9e00091a0 comp rx=0 tx=0).stop 2023-10-06T03:29:50.408 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.405+0000 7fb9cf7fe700 1 -- 172.21.15.110:0/3455238065 shutdown_connections 2023-10-06T03:29:50.408 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.405+0000 7fb9cf7fe700 1 --2- 172.21.15.110:0/3455238065 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fb9c8060620 0x7fb9c8062ad0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:50.408 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.405+0000 7fb9cf7fe700 1 --2- 172.21.15.110:0/3455238065 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb9e415d050 0x7fb9e414e2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:50.408 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.405+0000 7fb9cf7fe700 1 --2- 172.21.15.110:0/3455238065 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb9e415cc80 0x7fb9e414ddb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:50.409 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.405+0000 7fb9cf7fe700 1 --2- 172.21.15.110:0/3455238065 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb9e41590a0 0x7fb9e40c8320 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:50.409 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.405+0000 7fb9cf7fe700 1 -- 172.21.15.110:0/3455238065 >> 172.21.15.110:0/3455238065 conn(0x7fb9e406e760 msgr2=0x7fb9e4157f50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:29:50.409 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.405+0000 7fb9cf7fe700 1 -- 172.21.15.110:0/3455238065 shutdown_connections 2023-10-06T03:29:50.409 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.405+0000 7fb9cf7fe700 1 -- 172.21.15.110:0/3455238065 wait complete. 2023-10-06T03:29:50.416 INFO:teuthology.orchestra.run.smithi110.stdout:64424509449 2023-10-06T03:29:50.416 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.405+0000 7f276745d700 1 -- 172.21.15.110:0/768454148 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f27600b0490 con 0x7f276006ab80 2023-10-06T03:29:50.417 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.409+0000 7f27557fa700 1 -- 172.21.15.110:0/768454148 <== mon.0 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f27600b0490 con 0x7f276006ab80 2023-10-06T03:29:50.417 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.413+0000 7f274affd700 1 -- 172.21.15.110:0/768454148 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f2744060620 msgr2=0x7f2744062ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:50.417 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.413+0000 7f274affd700 1 --2- 172.21.15.110:0/768454148 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f2744060620 0x7f2744062ad0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f27600c71a0 tx=0x7f2750017000 comp rx=0 tx=0).stop 2023-10-06T03:29:50.417 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.413+0000 7f274affd700 1 -- 172.21.15.110:0/768454148 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f276006ab80 msgr2=0x7f2760151d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:50.417 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.413+0000 7f274affd700 1 --2- 172.21.15.110:0/768454148 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f276006ab80 0x7f2760151d60 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f2758009ef0 tx=0x7f275800b040 comp rx=0 tx=0).stop 2023-10-06T03:29:50.417 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.413+0000 7f274affd700 1 -- 172.21.15.110:0/768454148 shutdown_connections 2023-10-06T03:29:50.417 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.413+0000 7f274affd700 1 --2- 172.21.15.110:0/768454148 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f2744060620 0x7f2744062ad0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:50.417 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.413+0000 7f274affd700 1 --2- 172.21.15.110:0/768454148 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2760157360 0x7f276014f730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:50.417 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.413+0000 7f274affd700 1 --2- 172.21.15.110:0/768454148 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2760156f90 0x7f27600afd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:50.417 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.413+0000 7f274affd700 1 --2- 172.21.15.110:0/768454148 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f276006ab80 0x7f2760151d60 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:50.417 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.413+0000 7f274affd700 1 -- 172.21.15.110:0/768454148 >> 172.21.15.110:0/768454148 conn(0x7f27600cc6d0 msgr2=0x7f27600ca700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:29:50.417 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.413+0000 7f274affd700 1 -- 172.21.15.110:0/768454148 shutdown_connections 2023-10-06T03:29:50.418 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.413+0000 7f274affd700 1 -- 172.21.15.110:0/768454148 wait complete. 2023-10-06T03:29:50.418 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.413+0000 7ff845b0f700 1 -- 172.21.15.110:0/2650449895 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7ff820005830 con 0x7ff8400c8c10 2023-10-06T03:29:50.418 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.413+0000 7ff8277fe700 1 -- 172.21.15.110:0/2650449895 <== mon.0 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7ff834052600 con 0x7ff8400c8c10 2023-10-06T03:29:50.418 INFO:teuthology.orchestra.run.smithi110.stdout:68719476745 2023-10-06T03:29:50.422 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542152 got 60129542151 for osd.1 2023-10-06T03:29:50.423 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.421+0000 7ff8257fa700 1 -- 172.21.15.110:0/2650449895 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7ff82c060550 msgr2=0x7ff82c062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:50.423 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.421+0000 7ff8257fa700 1 --2- 172.21.15.110:0/2650449895 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7ff82c060550 0x7ff82c062a00 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7ff830001a20 tx=0x7ff830009040 comp rx=0 tx=0).stop 2023-10-06T03:29:50.423 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.421+0000 7ff8257fa700 1 -- 172.21.15.110:0/2650449895 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff8400c8c10 msgr2=0x7ff840141f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:50.423 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.421+0000 7ff8257fa700 1 --2- 172.21.15.110:0/2650449895 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff8400c8c10 0x7ff840141f20 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7ff8400c9d50 tx=0x7ff834016000 comp rx=0 tx=0).stop 2023-10-06T03:29:50.424 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.421+0000 7ff8257fa700 1 -- 172.21.15.110:0/2650449895 shutdown_connections 2023-10-06T03:29:50.424 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.421+0000 7ff8257fa700 1 --2- 172.21.15.110:0/2650449895 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7ff82c060550 0x7ff82c062a00 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:50.424 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.421+0000 7ff8257fa700 1 --2- 172.21.15.110:0/2650449895 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff8400c9750 0x7ff840142460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:50.424 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.421+0000 7ff8257fa700 1 --2- 172.21.15.110:0/2650449895 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff8400c8c10 0x7ff840141f20 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:50.424 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.421+0000 7ff8257fa700 1 --2- 172.21.15.110:0/2650449895 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff8400c20d0 0x7ff8400ad2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:50.424 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.421+0000 7ff8257fa700 1 -- 172.21.15.110:0/2650449895 >> 172.21.15.110:0/2650449895 conn(0x7ff8400cc560 msgr2=0x7ff8400ca690 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:29:50.424 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.421+0000 7ff8257fa700 1 -- 172.21.15.110:0/2650449895 shutdown_connections 2023-10-06T03:29:50.425 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.421+0000 7ff8257fa700 1 -- 172.21.15.110:0/2650449895 wait complete. 2023-10-06T03:29:50.431 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509450 got 64424509449 for osd.0 2023-10-06T03:29:50.433 INFO:tasks.ceph.ceph_manager.ceph:need seq 68719476746 got 68719476745 for osd.6 2023-10-06T03:29:50.436 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.433+0000 7ff34b59e700 1 -- 172.21.15.110:0/1155933271 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7ff330005560 con 0x7ff34c06bed0 2023-10-06T03:29:50.437 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.433+0000 7ff33a7fc700 1 -- 172.21.15.110:0/1155933271 <== mon.0 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7ff340021020 con 0x7ff34c06bed0 2023-10-06T03:29:50.437 INFO:teuthology.orchestra.run.smithi110.stdout:64424509448 2023-10-06T03:29:50.439 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.433+0000 7ff34b59e700 1 -- 172.21.15.110:0/1155933271 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7ff32c0605d0 msgr2=0x7ff32c062a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:50.439 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.433+0000 7ff34b59e700 1 --2- 172.21.15.110:0/1155933271 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7ff32c0605d0 0x7ff32c062a80 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7ff33c007110 tx=0x7ff33c007f10 comp rx=0 tx=0).stop 2023-10-06T03:29:50.439 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.433+0000 7ff34b59e700 1 -- 172.21.15.110:0/1155933271 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff34c06bed0 msgr2=0x7ff34c1595e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:50.439 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.433+0000 7ff34b59e700 1 --2- 172.21.15.110:0/1155933271 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff34c06bed0 0x7ff34c1595e0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7ff34000ea50 tx=0x7ff3400091a0 comp rx=0 tx=0).stop 2023-10-06T03:29:50.439 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.433+0000 7ff34b59e700 1 -- 172.21.15.110:0/1155933271 shutdown_connections 2023-10-06T03:29:50.439 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.433+0000 7ff34b59e700 1 --2- 172.21.15.110:0/1155933271 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7ff32c0605d0 0x7ff32c062a80 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:50.439 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.433+0000 7ff34b59e700 1 --2- 172.21.15.110:0/1155933271 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff34c06bed0 0x7ff34c1595e0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:50.439 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.433+0000 7ff34b59e700 1 --2- 172.21.15.110:0/1155933271 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff34c06a970 0x7ff34c152420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:50.439 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.433+0000 7ff34b59e700 1 --2- 172.21.15.110:0/1155933271 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff34c069640 0x7ff34c0c34c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:50.439 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.433+0000 7ff34b59e700 1 -- 172.21.15.110:0/1155933271 >> 172.21.15.110:0/1155933271 conn(0x7ff34c06dce0 msgr2=0x7ff34c0b08f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:29:50.439 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.433+0000 7ff34b59e700 1 -- 172.21.15.110:0/1155933271 shutdown_connections 2023-10-06T03:29:50.439 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.433+0000 7ff34b59e700 1 -- 172.21.15.110:0/1155933271 wait complete. 2023-10-06T03:29:50.448 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509449 got 64424509448 for osd.7 2023-10-06T03:29:50.449 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.445+0000 7ff3effff700 1 -- 172.21.15.110:0/1147892266 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7ff3e4005830 con 0x7ff4080695e0 2023-10-06T03:29:50.450 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.445+0000 7ff3f67fc700 1 -- 172.21.15.110:0/1147892266 <== mon.1 v2:172.21.15.192:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7ff3f8029020 con 0x7ff4080695e0 2023-10-06T03:29:50.450 INFO:teuthology.orchestra.run.smithi110.stdout:68719476745 2023-10-06T03:29:50.453 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.449+0000 7ff3effff700 1 -- 172.21.15.110:0/1147892266 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7ff3e8060550 msgr2=0x7ff3e8062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:50.453 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.449+0000 7ff3effff700 1 --2- 172.21.15.110:0/1147892266 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7ff3e8060550 0x7ff3e8062a00 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7ff40806f100 tx=0x7ff400001650 comp rx=0 tx=0).stop 2023-10-06T03:29:50.453 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.449+0000 7ff3effff700 1 -- 172.21.15.110:0/1147892266 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff4080695e0 msgr2=0x7ff4081578e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:50.453 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.449+0000 7ff3effff700 1 --2- 172.21.15.110:0/1147892266 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff4080695e0 0x7ff4081578e0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7ff408068ee0 tx=0x7ff3f8016000 comp rx=0 tx=0).stop 2023-10-06T03:29:50.454 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.449+0000 7ff3effff700 1 -- 172.21.15.110:0/1147892266 shutdown_connections 2023-10-06T03:29:50.454 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.449+0000 7ff3effff700 1 --2- 172.21.15.110:0/1147892266 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7ff3e8060550 0x7ff3e8062a00 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:50.454 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.449+0000 7ff3effff700 1 --2- 172.21.15.110:0/1147892266 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff408157360 0x7ff408159160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:50.454 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.449+0000 7ff3effff700 1 --2- 172.21.15.110:0/1147892266 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff408156f90 0x7ff4080b1380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:50.454 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.449+0000 7ff3effff700 1 --2- 172.21.15.110:0/1147892266 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff4080695e0 0x7ff4081578e0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:50.454 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.449+0000 7ff3effff700 1 -- 172.21.15.110:0/1147892266 >> 172.21.15.110:0/1147892266 conn(0x7ff4080cc6d0 msgr2=0x7ff4080c1460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:29:50.454 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.449+0000 7ff3effff700 1 -- 172.21.15.110:0/1147892266 shutdown_connections 2023-10-06T03:29:50.454 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.449+0000 7ff3effff700 1 -- 172.21.15.110:0/1147892266 wait complete. 2023-10-06T03:29:50.466 INFO:tasks.ceph.ceph_manager.ceph:need seq 68719476746 got 68719476745 for osd.5 2023-10-06T03:29:50.472 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.465+0000 7f00b3d9e700 1 -- 172.21.15.110:0/3589528979 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7f00ac06ff60 con 0x7f00ac06ec20 2023-10-06T03:29:50.472 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.469+0000 7f00a1ffb700 1 -- 172.21.15.110:0/3589528979 <== mon.0 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f00a80091f0 con 0x7f00ac06ec20 2023-10-06T03:29:50.472 INFO:teuthology.orchestra.run.smithi110.stdout:68719476744 2023-10-06T03:29:50.474 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.469+0000 7f00b3d9e700 1 -- 172.21.15.110:0/3589528979 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f0090060550 msgr2=0x7f0090062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:50.474 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.469+0000 7f00b3d9e700 1 --2- 172.21.15.110:0/3589528979 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f0090060550 0x7f0090062a00 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f009c0098e0 tx=0x7f009c016000 comp rx=0 tx=0).stop 2023-10-06T03:29:50.474 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.469+0000 7f00b3d9e700 1 -- 172.21.15.110:0/3589528979 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f00ac06ec20 msgr2=0x7f00ac0c5220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:50.474 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.469+0000 7f00b3d9e700 1 --2- 172.21.15.110:0/3589528979 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f00ac06ec20 0x7f00ac0c5220 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f00a800b5e0 tx=0x7f00a80090f0 comp rx=0 tx=0).stop 2023-10-06T03:29:50.474 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.469+0000 7f00b3d9e700 1 -- 172.21.15.110:0/3589528979 shutdown_connections 2023-10-06T03:29:50.474 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.469+0000 7f00b3d9e700 1 --2- 172.21.15.110:0/3589528979 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f0090060550 0x7f0090062a00 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:50.474 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.469+0000 7f00b3d9e700 1 --2- 172.21.15.110:0/3589528979 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f00ac152ff0 0x7f00ac1394c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:50.474 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.469+0000 7f00b3d9e700 1 --2- 172.21.15.110:0/3589528979 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f00ac06ec20 0x7f00ac0c5220 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:50.474 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.469+0000 7f00b3d9e700 1 --2- 172.21.15.110:0/3589528979 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f00ac06b960 0x7f00ac151ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:50.475 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.469+0000 7f00b3d9e700 1 -- 172.21.15.110:0/3589528979 >> 172.21.15.110:0/3589528979 conn(0x7f00ac0cc6d0 msgr2=0x7f00ac0c7940 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:29:50.475 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.469+0000 7f00b3d9e700 1 -- 172.21.15.110:0/3589528979 shutdown_connections 2023-10-06T03:29:50.475 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.469+0000 7f00b3d9e700 1 -- 172.21.15.110:0/3589528979 wait complete. 2023-10-06T03:29:50.483 INFO:tasks.ceph.ceph_manager.ceph:need seq 68719476745 got 68719476744 for osd.4 2023-10-06T03:29:50.489 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.485+0000 7eff66410700 1 -- 172.21.15.110:0/1403710584 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7eff600c6850 con 0x7eff4c003a60 2023-10-06T03:29:50.489 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.485+0000 7eff557fa700 1 -- 172.21.15.110:0/1403710584 <== mon.1 v2:172.21.15.192:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7eff5c019090 con 0x7eff4c003a60 2023-10-06T03:29:50.490 INFO:teuthology.orchestra.run.smithi110.stdout:60129542152 2023-10-06T03:29:50.493 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.489+0000 7eff66410700 1 -- 172.21.15.110:0/1403710584 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7eff4c062100 msgr2=0x7eff4c0645b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:50.493 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.489+0000 7eff66410700 1 --2- 172.21.15.110:0/1403710584 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7eff4c062100 0x7eff4c0645b0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7eff50006fd0 tx=0x7eff50008040 comp rx=0 tx=0).stop 2023-10-06T03:29:50.493 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.489+0000 7eff66410700 1 -- 172.21.15.110:0/1403710584 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7eff4c003a60 msgr2=0x7eff6016e940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:50.493 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.489+0000 7eff66410700 1 --2- 172.21.15.110:0/1403710584 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7eff4c003a60 0x7eff6016e940 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7eff5c002d30 tx=0x7eff5c017000 comp rx=0 tx=0).stop 2023-10-06T03:29:50.494 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.489+0000 7eff66410700 1 -- 172.21.15.110:0/1403710584 shutdown_connections 2023-10-06T03:29:50.494 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.489+0000 7eff66410700 1 --2- 172.21.15.110:0/1403710584 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7eff4c062100 0x7eff4c0645b0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:50.494 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.489+0000 7eff66410700 1 --2- 172.21.15.110:0/1403710584 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7eff601578b0 0x7eff60157d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:50.494 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.489+0000 7eff66410700 1 --2- 172.21.15.110:0/1403710584 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7eff6014f630 0x7eff6014faa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:50.494 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.489+0000 7eff66410700 1 --2- 172.21.15.110:0/1403710584 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7eff4c003a60 0x7eff6016e940 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:50.494 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.489+0000 7eff66410700 1 -- 172.21.15.110:0/1403710584 >> 172.21.15.110:0/1403710584 conn(0x7eff6006dce0 msgr2=0x7eff6006e0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:29:50.494 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.489+0000 7eff66410700 1 -- 172.21.15.110:0/1403710584 shutdown_connections 2023-10-06T03:29:50.494 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.489+0000 7eff66410700 1 -- 172.21.15.110:0/1403710584 wait complete. 2023-10-06T03:29:50.495 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.489+0000 7fa351afa700 1 -- 172.21.15.110:0/2046814958 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7fa32c005830 con 0x7fa34c06c9e0 2023-10-06T03:29:50.495 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.489+0000 7fa348ff9700 1 -- 172.21.15.110:0/2046814958 <== mon.0 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fa33c052070 con 0x7fa34c06c9e0 2023-10-06T03:29:50.495 INFO:teuthology.orchestra.run.smithi110.stdout:60129542151 2023-10-06T03:29:50.496 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.493+0000 7fa351afa700 1 -- 172.21.15.110:0/2046814958 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fa338060580 msgr2=0x7fa338062a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:50.497 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.493+0000 7fa351afa700 1 --2- 172.21.15.110:0/2046814958 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fa338060580 0x7fa338062a30 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fa3400091e0 tx=0x7fa340016000 comp rx=0 tx=0).stop 2023-10-06T03:29:50.497 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.493+0000 7fa351afa700 1 -- 172.21.15.110:0/2046814958 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa34c06c9e0 msgr2=0x7fa34c0c6110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:50.497 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.493+0000 7fa351afa700 1 --2- 172.21.15.110:0/2046814958 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa34c06c9e0 0x7fa34c0c6110 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fa33c00cb90 tx=0x7fa33c001ac0 comp rx=0 tx=0).stop 2023-10-06T03:29:50.497 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.493+0000 7fa351afa700 1 -- 172.21.15.110:0/2046814958 shutdown_connections 2023-10-06T03:29:50.497 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.493+0000 7fa351afa700 1 --2- 172.21.15.110:0/2046814958 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fa338060580 0x7fa338062a30 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:50.497 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.493+0000 7fa351afa700 1 --2- 172.21.15.110:0/2046814958 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa34c06c9e0 0x7fa34c0c6110 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:50.497 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.493+0000 7fa351afa700 1 --2- 172.21.15.110:0/2046814958 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa34c06b7c0 0x7fa34c15e390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:50.497 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.493+0000 7fa351afa700 1 --2- 172.21.15.110:0/2046814958 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa34c069640 0x7fa34c161ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:50.498 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.493+0000 7fa351afa700 1 -- 172.21.15.110:0/2046814958 >> 172.21.15.110:0/2046814958 conn(0x7fa34c06e760 msgr2=0x7fa34c0abd10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:29:50.498 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.493+0000 7fa351afa700 1 -- 172.21.15.110:0/2046814958 shutdown_connections 2023-10-06T03:29:50.498 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:50.493+0000 7fa351afa700 1 -- 172.21.15.110:0/2046814958 wait complete. 2023-10-06T03:29:50.506 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542153 got 60129542152 for osd.3 2023-10-06T03:29:50.507 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542152 got 60129542151 for osd.2 2023-10-06T03:29:51.424 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-10-06T03:29:51.432 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-10-06T03:29:51.434 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2023-10-06T03:29:51.449 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2023-10-06T03:29:51.467 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2023-10-06T03:29:51.484 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2023-10-06T03:29:51.506 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2023-10-06T03:29:51.507 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-10-06T03:29:51.554 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.545+0000 7f91f5a50700 1 Processor -- start 2023-10-06T03:29:51.555 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.545+0000 7f91f5a50700 1 -- start start 2023-10-06T03:29:51.555 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.549+0000 7f91f5a50700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f91f006b7c0 0x7f91f006ccf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:51.555 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.549+0000 7f91f5a50700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f91f01692a0 0x7f91f00c0fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:51.555 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.549+0000 7f91f5a50700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91f006c860 con 0x7f91f0168ed0 2023-10-06T03:29:51.555 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.549+0000 7f91f5a50700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f91f00cab90 con 0x7f91f006b7c0 2023-10-06T03:29:51.555 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.549+0000 7f91f5a50700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91f006bc50 con 0x7f91f01692a0 2023-10-06T03:29:51.555 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.549+0000 7f91eeffd700 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f91f0168ed0 0x7f91f00ca480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:40098/0 (socket says 172.21.15.110:40098) 2023-10-06T03:29:51.555 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.549+0000 7f91eeffd700 1 -- 172.21.15.110:0/3746850449 learned_addr learned my addr 172.21.15.110:0/3746850449 (peer_addr_for_me v1:172.21.15.110:0/0) 2023-10-06T03:29:51.555 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.549+0000 7f91ef7fe700 1 --2- 172.21.15.110:0/3746850449 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f91f01692a0 0x7f91f00c0fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:51.556 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.549+0000 7f91ee7fc700 1 --2- 172.21.15.110:0/3746850449 >> v2:172.21.15.192:3300/0 conn(0x7f91f006b7c0 0x7f91f006ccf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:51.556 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.549+0000 7f91edffb700 1 -- 172.21.15.110:0/3746850449 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 503576541 0 0) 0x7f91f006c860 con 0x7f91f0168ed0 2023-10-06T03:29:51.556 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.549+0000 7f91edffb700 1 -- 172.21.15.110:0/3746850449 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f91d8007980 con 0x7f91f0168ed0 2023-10-06T03:29:51.556 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.549+0000 7f91edffb700 1 -- 172.21.15.110:0/3746850449 <== mon.0 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3364688557 0 0) 0x7f91d8007980 con 0x7f91f0168ed0 2023-10-06T03:29:51.556 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.549+0000 7f91edffb700 1 -- 172.21.15.110:0/3746850449 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f91f01692a0 msgr2=0x7f91f00c0fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:29:51.556 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.549+0000 7f91edffb700 1 --2- 172.21.15.110:0/3746850449 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f91f01692a0 0x7f91f00c0fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:51.556 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.549+0000 7f91edffb700 1 -- 172.21.15.110:0/3746850449 >> v2:172.21.15.192:3300/0 conn(0x7f91f006b7c0 msgr2=0x7f91f006ccf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:29:51.556 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.549+0000 7f91edffb700 1 --2- 172.21.15.110:0/3746850449 >> v2:172.21.15.192:3300/0 conn(0x7f91f006b7c0 0x7f91f006ccf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:51.556 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.549+0000 7f91edffb700 1 -- 172.21.15.110:0/3746850449 --> v1:172.21.15.110:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91f006bf30 con 0x7f91f0168ed0 2023-10-06T03:29:51.556 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.549+0000 7f91edffb700 1 -- 172.21.15.110:0/3746850449 <== mon.0 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 118603703 0 0) 0x7f91e0001470 con 0x7f91f0168ed0 2023-10-06T03:29:51.556 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.549+0000 7f91edffb700 1 -- 172.21.15.110:0/3746850449 >> v1:172.21.15.110:6789/0 conn(0x7f91f0168ed0 legacy=0x7f91f00ca480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:51.556 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.549+0000 7f91edffb700 1 --2- 172.21.15.110:0/3746850449 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f91f01692a0 0x7f91d8007ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:51.557 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.549+0000 7f91edffb700 1 --2- 172.21.15.110:0/3746850449 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f91d8008470 0x7f91d8008c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:51.557 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.549+0000 7f91edffb700 1 --2- 172.21.15.110:0/3746850449 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f91d80074b0 0x7f91d8009160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:51.557 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.549+0000 7f91edffb700 1 -- 172.21.15.110:0/3746850449 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91f00cab90 con 0x7f91d8008470 2023-10-06T03:29:51.557 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.549+0000 7f91edffb700 1 -- 172.21.15.110:0/3746850449 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91f006bc50 con 0x7f91d80074b0 2023-10-06T03:29:51.557 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.549+0000 7f91edffb700 1 -- 172.21.15.110:0/3746850449 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91d80098e0 con 0x7f91f01692a0 2023-10-06T03:29:51.557 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.549+0000 7f91ee7fc700 1 --2- 172.21.15.110:0/3746850449 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f91d8008470 0x7f91d8008c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:51.557 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.549+0000 7f91ef7fe700 1 --2- 172.21.15.110:0/3746850449 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f91d80074b0 0x7f91d8009160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:51.557 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.549+0000 7f91eeffd700 1 --2- 172.21.15.110:0/3746850449 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f91f01692a0 0x7f91d8007ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:51.557 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.549+0000 7f91ee7fc700 1 -- 172.21.15.110:0/3746850449 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f91f01692a0 msgr2=0x7f91d8007ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:29:51.557 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.549+0000 7f91ee7fc700 1 --2- 172.21.15.110:0/3746850449 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f91f01692a0 0x7f91d8007ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:51.557 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.549+0000 7f91ee7fc700 1 -- 172.21.15.110:0/3746850449 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f91d80074b0 msgr2=0x7f91d8009160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:29:51.557 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.549+0000 7f91ee7fc700 1 --2- 172.21.15.110:0/3746850449 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f91d80074b0 0x7f91d8009160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:51.558 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.549+0000 7f91ee7fc700 1 -- 172.21.15.110:0/3746850449 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91f006bf30 con 0x7f91d8008470 2023-10-06T03:29:51.558 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.549+0000 7f91eeffd700 1 --2- 172.21.15.110:0/3746850449 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f91f01692a0 0x7f91d8007ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-06T03:29:51.558 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.549+0000 7f91ee7fc700 1 --2- 172.21.15.110:0/3746850449 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f91d8008470 0x7f91d8008c20 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f91e400c880 tx=0x7f91e400e040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=646f5c0204474dd8 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:29:51.558 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.549+0000 7f91edffb700 1 -- 172.21.15.110:0/3746850449 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91e400c020 con 0x7f91d8008470 2023-10-06T03:29:51.558 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.549+0000 7f91edffb700 1 -- 172.21.15.110:0/3746850449 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91e400c180 con 0x7f91d8008470 2023-10-06T03:29:51.558 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.549+0000 7f91edffb700 1 -- 172.21.15.110:0/3746850449 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91e400c520 con 0x7f91d8008470 2023-10-06T03:29:51.558 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.549+0000 7fa804e0a700 1 Processor -- start 2023-10-06T03:29:51.559 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.553+0000 7f91f5a50700 1 -- 172.21.15.110:0/3746850449 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f91d8008470 msgr2=0x7f91d8008c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:51.559 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.553+0000 7f91f5a50700 1 --2- 172.21.15.110:0/3746850449 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f91d8008470 0x7f91d8008c20 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f91e400c880 tx=0x7f91e400e040 comp rx=0 tx=0).stop 2023-10-06T03:29:51.559 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.553+0000 7f91ef7fe700 1 -- 172.21.15.110:0/3746850449 reap_dead start 2023-10-06T03:29:51.559 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.553+0000 7f91f5a50700 1 -- 172.21.15.110:0/3746850449 shutdown_connections 2023-10-06T03:29:51.559 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.553+0000 7f91f5a50700 1 -- 172.21.15.110:0/3746850449 >> 172.21.15.110:0/3746850449 conn(0x7f91f00cc560 msgr2=0x7f91f00c8500 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:29:51.560 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.553+0000 7fa804e0a700 1 -- start start 2023-10-06T03:29:51.560 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.553+0000 7f91f5a50700 1 -- 172.21.15.110:0/3746850449 shutdown_connections 2023-10-06T03:29:51.560 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.553+0000 7f91f5a50700 1 -- 172.21.15.110:0/3746850449 wait complete. 2023-10-06T03:29:51.561 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.553+0000 7fa804e0a700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa800154cf0 0x7fa8000c7330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:51.561 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.553+0000 7fa804e0a700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fa8001550c0 0x7fa8001595e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:51.562 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.553+0000 7fa804e0a700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fa80006c3c0 0x7fa8000c80e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:51.562 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.553+0000 7fa804e0a700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa80006cbd0 con 0x7fa8001550c0 2023-10-06T03:29:51.562 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.553+0000 7fa804e0a700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa8000c6ea0 con 0x7fa80006c3c0 2023-10-06T03:29:51.563 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.553+0000 7fa804e0a700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8000c6570 con 0x7fa800154cf0 2023-10-06T03:29:51.563 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.553+0000 7fa7ff7fe700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa800154cf0 0x7fa8000c7330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:51.563 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.553+0000 7fa7fffff700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fa80006c3c0 0x7fa8000c80e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:51.563 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.553+0000 7fa7fffff700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fa80006c3c0 0x7fa8000c80e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.110:36298/0 (socket says 172.21.15.110:36298) 2023-10-06T03:29:51.563 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.553+0000 7fa7fffff700 1 -- 172.21.15.110:0/1891552441 learned_addr learned my addr 172.21.15.110:0/1891552441 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:29:51.563 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.553+0000 7fa7feffd700 1 --2- 172.21.15.110:0/1891552441 >> v2:172.21.15.110:3300/0 conn(0x7fa8001550c0 0x7fa8001595e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:51.563 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.553+0000 7fa7fffff700 1 -- 172.21.15.110:0/1891552441 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa800154cf0 msgr2=0x7fa8000c7330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:29:51.563 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.553+0000 7fa7fffff700 1 --2- 172.21.15.110:0/1891552441 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa800154cf0 0x7fa8000c7330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:51.563 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.553+0000 7fa7fffff700 1 -- 172.21.15.110:0/1891552441 >> v2:172.21.15.110:3300/0 conn(0x7fa8001550c0 msgr2=0x7fa8001595e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:29:51.563 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.553+0000 7fa7fffff700 1 --2- 172.21.15.110:0/1891552441 >> v2:172.21.15.110:3300/0 conn(0x7fa8001550c0 0x7fa8001595e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:51.564 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.553+0000 7fa7fffff700 1 -- 172.21.15.110:0/1891552441 --> v2:172.21.15.192:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8000c6850 con 0x7fa80006c3c0 2023-10-06T03:29:51.564 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.553+0000 7faa2dcb7700 1 Processor -- start 2023-10-06T03:29:51.564 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.553+0000 7fa7fffff700 1 --2- 172.21.15.110:0/1891552441 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa80006c3c0 0x7fa8000c80e0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fa8000c6000 tx=0x7fa7f8009f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=357444c8f2cf773b server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:29:51.564 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.553+0000 7f91f5a50700 1 Processor -- start 2023-10-06T03:29:51.565 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.553+0000 7f91f5a50700 1 -- start start 2023-10-06T03:29:51.565 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.553+0000 7faa2dcb7700 1 -- start start 2023-10-06T03:29:51.565 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.553+0000 7fa7fe7fc700 1 -- 172.21.15.110:0/1891552441 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7f80092e0 con 0x7fa80006c3c0 2023-10-06T03:29:51.565 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.553+0000 7fa7fe7fc700 1 -- 172.21.15.110:0/1891552441 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7f800b950 con 0x7fa80006c3c0 2023-10-06T03:29:51.565 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.553+0000 7fa7fe7fc700 1 -- 172.21.15.110:0/1891552441 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7f800bcf0 con 0x7fa80006c3c0 2023-10-06T03:29:51.566 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.553+0000 7f91f5a50700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f91d8008470 0x7f91f01573d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:51.566 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.557+0000 7f91f5a50700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f91f00c3a00 0x7f91f00b1280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:51.566 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.557+0000 7f91f5a50700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f91f00af1e0 0x7f91f00af680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:51.566 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.557+0000 7f91f5a50700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91f00c41e0 con 0x7f91d8008470 2023-10-06T03:29:51.566 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.557+0000 7f91eeffd700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f91d8008470 0x7f91f01573d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:51.566 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.557+0000 7f91eeffd700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f91d8008470 0x7f91f01573d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:47514/0 (socket says 172.21.15.110:47514) 2023-10-06T03:29:51.567 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.557+0000 7f91eeffd700 1 -- 172.21.15.110:0/1001659278 learned_addr learned my addr 172.21.15.110:0/1001659278 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:29:51.567 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.557+0000 7fa804e0a700 1 -- 172.21.15.110:0/1891552441 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa80006c3c0 msgr2=0x7fa8000c80e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:51.567 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.557+0000 7fa804e0a700 1 --2- 172.21.15.110:0/1891552441 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa80006c3c0 0x7fa8000c80e0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fa8000c6000 tx=0x7fa7f8009f90 comp rx=0 tx=0).stop 2023-10-06T03:29:51.567 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.557+0000 7fa804e0a700 1 -- 172.21.15.110:0/1891552441 shutdown_connections 2023-10-06T03:29:51.567 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.557+0000 7fa804e0a700 1 --2- 172.21.15.110:0/1891552441 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa80006c3c0 0x7fa8000c80e0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:51.567 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.557+0000 7fa804e0a700 1 --2- 172.21.15.110:0/1891552441 >> v2:172.21.15.110:3300/0 conn(0x7fa8001550c0 0x7fa8001595e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:51.567 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.557+0000 7fa804e0a700 1 --2- 172.21.15.110:0/1891552441 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa800154cf0 0x7fa8000c7330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:51.568 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.557+0000 7fa804e0a700 1 -- 172.21.15.110:0/1891552441 >> 172.21.15.110:0/1891552441 conn(0x7fa80006dce0 msgr2=0x7fa80006e0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:29:51.568 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.557+0000 7f91ee7fc700 1 --2- 172.21.15.110:0/1001659278 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f91f00c3a00 0x7f91f00b1280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:51.568 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.557+0000 7f91ef7fe700 1 --2- 172.21.15.110:0/1001659278 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f91f00af1e0 0x7f91f00af680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:51.568 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.557+0000 7f91f5a50700 1 -- 172.21.15.110:0/1001659278 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91f00c4d90 con 0x7f91f00af1e0 2023-10-06T03:29:51.568 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.557+0000 7f91f5a50700 1 -- 172.21.15.110:0/1001659278 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91f00afbc0 con 0x7f91f00c3a00 2023-10-06T03:29:51.569 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.557+0000 7fa804e0a700 1 -- 172.21.15.110:0/1891552441 shutdown_connections 2023-10-06T03:29:51.569 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.557+0000 7fa804e0a700 1 -- 172.21.15.110:0/1891552441 wait complete. 2023-10-06T03:29:51.569 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.557+0000 7f91eeffd700 1 -- 172.21.15.110:0/1001659278 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f91f00c3a00 msgr2=0x7f91f00b1280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:51.569 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.557+0000 7f91eeffd700 1 --2- 172.21.15.110:0/1001659278 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f91f00c3a00 0x7f91f00b1280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:51.569 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.557+0000 7f91eeffd700 1 -- 172.21.15.110:0/1001659278 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f91f00af1e0 msgr2=0x7f91f00af680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:51.569 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.557+0000 7f91eeffd700 1 --2- 172.21.15.110:0/1001659278 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f91f00af1e0 0x7f91f00af680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:51.570 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.557+0000 7f91eeffd700 1 -- 172.21.15.110:0/1001659278 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91f0164ea0 con 0x7f91d8008470 2023-10-06T03:29:51.570 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.557+0000 7f91eeffd700 1 --2- 172.21.15.110:0/1001659278 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f91d8008470 0x7f91f01573d0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f91e000edd0 tx=0x7f91e0005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:29:51.570 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.557+0000 7faa2dcb7700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7faa280c7a70 0x7faa280c7e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:51.570 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.557+0000 7fa804e0a700 1 Processor -- start 2023-10-06T03:29:51.571 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.557+0000 7f91d6ffd700 1 -- 172.21.15.110:0/1001659278 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91e000cac0 con 0x7f91d8008470 2023-10-06T03:29:51.571 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.557+0000 7f91d6ffd700 1 -- 172.21.15.110:0/1001659278 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91e000cc20 con 0x7f91d8008470 2023-10-06T03:29:51.571 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.557+0000 7f91d6ffd700 1 -- 172.21.15.110:0/1001659278 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91e0013420 con 0x7f91d8008470 2023-10-06T03:29:51.571 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.557+0000 7f91f5a50700 1 -- 172.21.15.110:0/1001659278 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91f00c3f00 con 0x7f91d8008470 2023-10-06T03:29:51.571 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.557+0000 7faa277fe700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7faa280c7a70 0x7faa280c7e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:51.572 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.557+0000 7faa277fe700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7faa280c7a70 0x7faa280c7e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:47530/0 (socket says 172.21.15.110:47530) 2023-10-06T03:29:51.572 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.557+0000 7faa277fe700 1 -- 172.21.15.110:0/3943602192 learned_addr learned my addr 172.21.15.110:0/3943602192 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:29:51.572 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.557+0000 7f91f5a50700 1 -- 172.21.15.110:0/1001659278 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f91f00c5260 con 0x7f91d8008470 2023-10-06T03:29:51.572 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.557+0000 7fa804e0a700 1 -- start start 2023-10-06T03:29:51.573 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.561+0000 7fa804e0a700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa80006c3c0 0x7fa800156d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:51.573 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.561+0000 7fa804e0a700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa800154cf0 0x7fa80016d4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:51.573 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.561+0000 7fa804e0a700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa8001550c0 0x7fa8000acc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:51.573 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.561+0000 7fa804e0a700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8000c2ae0 con 0x7fa800154cf0 2023-10-06T03:29:51.573 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.561+0000 7fa804e0a700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa800170ad0 con 0x7fa8001550c0 2023-10-06T03:29:51.573 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.561+0000 7fa804e0a700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa800170db0 con 0x7fa80006c3c0 2023-10-06T03:29:51.573 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.561+0000 7f91d6ffd700 1 -- 172.21.15.110:0/1001659278 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7f91e00137d0 con 0x7f91d8008470 2023-10-06T03:29:51.574 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.561+0000 7fa7feffd700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa800154cf0 0x7fa80016d4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:51.574 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.561+0000 7fa7feffd700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa800154cf0 0x7fa80016d4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:47532/0 (socket says 172.21.15.110:47532) 2023-10-06T03:29:51.574 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.561+0000 7fa7feffd700 1 -- 172.21.15.110:0/2403884471 learned_addr learned my addr 172.21.15.110:0/2403884471 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:29:51.574 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.561+0000 7fa7ff7fe700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa80006c3c0 0x7fa800156d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:51.574 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.561+0000 7f91d6ffd700 1 --2- 172.21.15.110:0/1001659278 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f91dc060620 0x7f91dc062ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:51.574 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.561+0000 7f91d6ffd700 1 -- 172.21.15.110:0/1001659278 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6809+0+0 (secure 0 0 0) 0x7f91e007cec0 con 0x7f91d8008470 2023-10-06T03:29:51.575 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.561+0000 7fa7feffd700 1 -- 172.21.15.110:0/2403884471 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa80006c3c0 msgr2=0x7fa800156d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:51.575 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.561+0000 7fa7feffd700 1 --2- 172.21.15.110:0/2403884471 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa80006c3c0 0x7fa800156d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:51.575 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.561+0000 7fa7feffd700 1 -- 172.21.15.110:0/2403884471 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa8001550c0 msgr2=0x7fa8000acc50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-06T03:29:51.575 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.561+0000 7fa7feffd700 1 --2- 172.21.15.110:0/2403884471 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa8001550c0 0x7fa8000acc50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:51.575 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.561+0000 7fa7feffd700 1 -- 172.21.15.110:0/2403884471 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7f8009850 con 0x7fa800154cf0 2023-10-06T03:29:51.575 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.561+0000 7f91ee7fc700 1 --2- 172.21.15.110:0/1001659278 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f91dc060620 0x7f91dc062ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:51.576 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.561+0000 7faa2dcb7700 1 --2- 172.21.15.110:0/3943602192 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7faa2815a250 0x7faa280c9dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:51.576 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.561+0000 7faa2dcb7700 1 -- 172.21.15.110:0/3943602192 --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa280c9660 con 0x7faa280c5b60 2023-10-06T03:29:51.576 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.561+0000 7faa2dcb7700 1 -- 172.21.15.110:0/3943602192 --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faa280cb220 con 0x7faa280c7a70 2023-10-06T03:29:51.576 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.561+0000 7faa2dcb7700 1 -- 172.21.15.110:0/3943602192 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa280cb500 con 0x7faa2815a250 2023-10-06T03:29:51.576 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.561+0000 7fa7feffd700 1 --2- 172.21.15.110:0/2403884471 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa800154cf0 0x7fa80016d4d0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fa7f4006b00 tx=0x7fa7f400b430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:29:51.577 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.561+0000 7f91d4ff9700 1 -- 172.21.15.110:0/1001659278 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f91f0070ff0 con 0x7f91d8008470 2023-10-06T03:29:51.577 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.561+0000 7faa27fff700 1 --2- 172.21.15.110:0/3943602192 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7faa2815a250 0x7faa280c9dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:51.577 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.561+0000 7f91ee7fc700 1 --2- 172.21.15.110:0/1001659278 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f91dc060620 0x7f91dc062ad0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f91e400b490 tx=0x7f91e4013000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:29:51.578 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.561+0000 7faa277fe700 1 -- 172.21.15.110:0/3943602192 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7faa2815a250 msgr2=0x7faa280c9dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:29:51.578 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.561+0000 7faa277fe700 1 --2- 172.21.15.110:0/3943602192 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7faa2815a250 0x7faa280c9dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:51.578 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.561+0000 7faa277fe700 1 -- 172.21.15.110:0/3943602192 >> v1:172.21.15.192:6789/0 conn(0x7faa280c5b60 legacy=0x7faa2806b7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:51.578 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.561+0000 7faa277fe700 1 -- 172.21.15.110:0/3943602192 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa280c98d0 con 0x7faa280c7a70 2023-10-06T03:29:51.578 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.561+0000 7faa267fc700 1 -- 172.21.15.110:0/3943602192 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3600036747 0 0) 0x7faa280c9660 con 0x7faa280c5b60 2023-10-06T03:29:51.578 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.561+0000 7faa277fe700 1 --2- 172.21.15.110:0/3943602192 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7faa280c7a70 0x7faa280c7e40 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7faa180099b0 tx=0x7faa18009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a05054f2f2b3573a server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:29:51.579 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.565+0000 7fa7e77fe700 1 -- 172.21.15.110:0/2403884471 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7f400dac0 con 0x7fa800154cf0 2023-10-06T03:29:51.579 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.569+0000 7fa7e77fe700 1 -- 172.21.15.110:0/2403884471 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7f400dc20 con 0x7fa800154cf0 2023-10-06T03:29:51.579 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.569+0000 7faa267fc700 1 -- 172.21.15.110:0/3943602192 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa18009ea0 con 0x7faa280c7a70 2023-10-06T03:29:51.579 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.569+0000 7faa267fc700 1 -- 172.21.15.110:0/3943602192 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa1800b860 con 0x7faa280c7a70 2023-10-06T03:29:51.579 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.569+0000 7faa267fc700 1 -- 172.21.15.110:0/3943602192 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa1800bc00 con 0x7faa280c7a70 2023-10-06T03:29:51.579 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.569+0000 7f91d6ffd700 1 -- 172.21.15.110:0/1001659278 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f91e004d040 con 0x7f91d8008470 2023-10-06T03:29:51.580 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.569+0000 7fa804e0a700 1 -- 172.21.15.110:0/2403884471 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa80016da10 con 0x7fa800154cf0 2023-10-06T03:29:51.580 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.569+0000 7fa804e0a700 1 -- 172.21.15.110:0/2403884471 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa80016e010 con 0x7fa800154cf0 2023-10-06T03:29:51.580 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.569+0000 7fa7e77fe700 1 -- 172.21.15.110:0/2403884471 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7f4010440 con 0x7fa800154cf0 2023-10-06T03:29:51.581 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.569+0000 7faa2dcb7700 1 -- 172.21.15.110:0/3943602192 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7faa280c7a70 msgr2=0x7faa280c7e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:51.581 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.569+0000 7faa2dcb7700 1 --2- 172.21.15.110:0/3943602192 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7faa280c7a70 0x7faa280c7e40 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7faa180099b0 tx=0x7faa18009250 comp rx=0 tx=0).stop 2023-10-06T03:29:51.581 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.569+0000 7fa7e77fe700 1 -- 172.21.15.110:0/2403884471 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7fa7f40107b0 con 0x7fa800154cf0 2023-10-06T03:29:51.581 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.569+0000 7faa2dcb7700 1 -- 172.21.15.110:0/3943602192 shutdown_connections 2023-10-06T03:29:51.582 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.569+0000 7faa2dcb7700 1 --2- 172.21.15.110:0/3943602192 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7faa2815a250 0x7faa280c9dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:51.582 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.569+0000 7faa2dcb7700 1 --2- 172.21.15.110:0/3943602192 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7faa280c7a70 0x7faa280c7e40 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:51.582 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.569+0000 7faa2dcb7700 1 -- 172.21.15.110:0/3943602192 >> 172.21.15.110:0/3943602192 conn(0x7faa280cc560 msgr2=0x7faa280c8500 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:29:51.582 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.569+0000 7fa804e0a700 1 -- 172.21.15.110:0/2403884471 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa7e00052b0 con 0x7fa800154cf0 2023-10-06T03:29:51.582 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.569+0000 7fa7e77fe700 1 --2- 172.21.15.110:0/2403884471 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fa7ec0606a0 0x7fa7ec062b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:51.583 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.573+0000 7fa7ff7fe700 1 --2- 172.21.15.110:0/2403884471 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fa7ec0606a0 0x7fa7ec062b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:51.583 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.573+0000 7fa7ff7fe700 1 --2- 172.21.15.110:0/2403884471 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fa7ec0606a0 0x7fa7ec062b50 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fa7f0009db0 tx=0x7fa7f00091a0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:29:51.583 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.573+0000 7fa7e77fe700 1 -- 172.21.15.110:0/2403884471 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6809+0+0 (secure 0 0 0) 0x7fa7f4014030 con 0x7fa800154cf0 2023-10-06T03:29:51.584 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.573+0000 7faa2dcb7700 1 -- 172.21.15.110:0/3943602192 shutdown_connections 2023-10-06T03:29:51.584 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.573+0000 7faa2dcb7700 1 -- 172.21.15.110:0/3943602192 wait complete. 2023-10-06T03:29:51.584 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.573+0000 7faa2dcb7700 1 Processor -- start 2023-10-06T03:29:51.584 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.573+0000 7faa2dcb7700 1 -- start start 2023-10-06T03:29:51.584 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.577+0000 7faa2dcb7700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7faa280c5b60 0x7faa2814e1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:51.585 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.577+0000 7faa2dcb7700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7faa280c7a70 0x7faa281500a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:51.585 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.577+0000 7faa2dcb7700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7faa2815a250 0x7faa280c3900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:51.585 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.577+0000 7faa26ffd700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7faa280c7a70 0x7faa281500a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:51.585 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.577+0000 7faa26ffd700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7faa280c7a70 0x7faa281500a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:47544/0 (socket says 172.21.15.110:47544) 2023-10-06T03:29:51.585 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.577+0000 7faa277fe700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7faa280c5b60 0x7faa2814e1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:51.585 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.577+0000 7faa277fe700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7faa280c5b60 0x7faa2814e1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.110:36308/0 (socket says 172.21.15.110:36308) 2023-10-06T03:29:51.585 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.577+0000 7faa26ffd700 1 -- 172.21.15.110:0/3400627858 learned_addr learned my addr 172.21.15.110:0/3400627858 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:29:51.585 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.577+0000 7faa2dcb7700 1 -- 172.21.15.110:0/3400627858 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa28150a60 con 0x7faa280c7a70 2023-10-06T03:29:51.585 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.577+0000 7faa2dcb7700 1 -- 172.21.15.110:0/3400627858 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa280c2b40 con 0x7faa280c5b60 2023-10-06T03:29:51.585 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.577+0000 7faa27fff700 1 --2- 172.21.15.110:0/3400627858 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7faa2815a250 0x7faa280c3900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:51.585 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.577+0000 7faa2dcb7700 1 -- 172.21.15.110:0/3400627858 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa280c2e20 con 0x7faa2815a250 2023-10-06T03:29:51.586 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.577+0000 7faa26ffd700 1 -- 172.21.15.110:0/3400627858 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7faa2815a250 msgr2=0x7faa280c3900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:51.586 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.577+0000 7faa26ffd700 1 --2- 172.21.15.110:0/3400627858 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7faa2815a250 0x7faa280c3900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:51.586 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.577+0000 7faa26ffd700 1 -- 172.21.15.110:0/3400627858 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7faa280c5b60 msgr2=0x7faa2814e1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:51.586 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.577+0000 7faa26ffd700 1 --2- 172.21.15.110:0/3400627858 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7faa280c5b60 0x7faa2814e1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:51.586 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.577+0000 7faa26ffd700 1 -- 172.21.15.110:0/3400627858 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa180094f0 con 0x7faa280c7a70 2023-10-06T03:29:51.586 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.577+0000 7faa26ffd700 1 --2- 172.21.15.110:0/3400627858 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7faa280c7a70 0x7faa281500a0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7faa1c001eb0 tx=0x7faa1c007d40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:29:51.586 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.581+0000 7fa7e77fe700 1 -- 172.21.15.110:0/2403884471 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fa7f404d570 con 0x7fa800154cf0 2023-10-06T03:29:51.587 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.581+0000 7faa0f7fe700 1 -- 172.21.15.110:0/3400627858 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa1c011590 con 0x7faa280c7a70 2023-10-06T03:29:51.588 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.581+0000 7faa2dcb7700 1 -- 172.21.15.110:0/3400627858 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa280c3100 con 0x7faa280c7a70 2023-10-06T03:29:51.588 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.581+0000 7faa2dcb7700 1 -- 172.21.15.110:0/3400627858 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa280c52d0 con 0x7faa280c7a70 2023-10-06T03:29:51.588 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.581+0000 7faa0f7fe700 1 -- 172.21.15.110:0/3400627858 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa1c0116f0 con 0x7faa280c7a70 2023-10-06T03:29:51.588 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.581+0000 7faa0f7fe700 1 -- 172.21.15.110:0/3400627858 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa1c011ab0 con 0x7faa280c7a70 2023-10-06T03:29:51.588 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.581+0000 7faa0f7fe700 1 -- 172.21.15.110:0/3400627858 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7faa1c01b3b0 con 0x7faa280c7a70 2023-10-06T03:29:51.588 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.581+0000 7faa2dcb7700 1 -- 172.21.15.110:0/3400627858 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa280c35e0 con 0x7faa280c7a70 2023-10-06T03:29:51.588 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.585+0000 7faa0f7fe700 1 --2- 172.21.15.110:0/3400627858 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7faa100607c0 0x7faa10062c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:51.588 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.585+0000 7faa277fe700 1 --2- 172.21.15.110:0/3400627858 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7faa100607c0 0x7faa10062c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:51.589 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.585+0000 7faa277fe700 1 --2- 172.21.15.110:0/3400627858 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7faa100607c0 0x7faa10062c70 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7faa180099b0 tx=0x7faa18017000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:29:51.589 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.585+0000 7faa0f7fe700 1 -- 172.21.15.110:0/3400627858 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6809+0+0 (secure 0 0 0) 0x7faa1c07c030 con 0x7faa280c7a70 2023-10-06T03:29:51.596 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.593+0000 7faa0f7fe700 1 -- 172.21.15.110:0/3400627858 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7faa1c04c1b0 con 0x7faa280c7a70 2023-10-06T03:29:51.621 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.617+0000 7ff5fca25700 1 Processor -- start 2023-10-06T03:29:51.623 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.617+0000 7ff5fca25700 1 -- start start 2023-10-06T03:29:51.623 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.617+0000 7ff5fca25700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7ff5f8154d70 0x7ff5f80c9750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:51.623 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.617+0000 7ff5fca25700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5f80c1c40 con 0x7ff5f80c75e0 2023-10-06T03:29:51.623 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.617+0000 7ff5fca25700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5f80c7a60 con 0x7ff5f8155140 2023-10-06T03:29:51.623 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.617+0000 7ff5fca25700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff5f80c27e0 con 0x7ff5f8154d70 2023-10-06T03:29:51.623 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.617+0000 7ff5f659c700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7ff5f8154d70 0x7ff5f80c9750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:51.624 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.617+0000 7ff5f659c700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7ff5f8154d70 0x7ff5f80c9750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:47556/0 (socket says 172.21.15.110:47556) 2023-10-06T03:29:51.624 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.617+0000 7ff5f659c700 1 -- 172.21.15.110:0/5466723 learned_addr learned my addr 172.21.15.110:0/5466723 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:29:51.624 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.621+0000 7ff5f659c700 1 -- 172.21.15.110:0/5466723 >> v1:172.21.15.192:6789/0 conn(0x7ff5f8155140 legacy=0x7ff5f80c20d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:51.624 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.621+0000 7ff5f659c700 1 -- 172.21.15.110:0/5466723 >> v1:172.21.15.110:6789/0 conn(0x7ff5f80c75e0 legacy=0x7ff5f80ca3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:51.624 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.621+0000 7ff5f559a700 1 -- 172.21.15.110:0/5466723 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1334356094 0 0) 0x7ff5f80c7a60 con 0x7ff5f8155140 2023-10-06T03:29:51.624 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.621+0000 7ff5f659c700 1 -- 172.21.15.110:0/5466723 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5f80c7d10 con 0x7ff5f8154d70 2023-10-06T03:29:51.624 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.621+0000 7f97f7058700 1 Processor -- start 2023-10-06T03:29:51.625 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.621+0000 7ff5f659c700 1 --2- 172.21.15.110:0/5466723 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff5f8154d70 0x7ff5f80c9750 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7ff5e800aa50 tx=0x7ff5e800a2f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=26badb53d03fcba9 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:29:51.625 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.621+0000 7ff5f559a700 1 -- 172.21.15.110:0/5466723 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5e8010e20 con 0x7ff5f8154d70 2023-10-06T03:29:51.625 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.621+0000 7ff5f559a700 1 -- 172.21.15.110:0/5466723 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5e800ab90 con 0x7ff5f8154d70 2023-10-06T03:29:51.625 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.621+0000 7ff5f559a700 1 -- 172.21.15.110:0/5466723 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5e801b430 con 0x7ff5f8154d70 2023-10-06T03:29:51.625 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.621+0000 7ff5fca25700 1 -- 172.21.15.110:0/5466723 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff5f8154d70 msgr2=0x7ff5f80c9750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:51.626 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.621+0000 7ff5fca25700 1 --2- 172.21.15.110:0/5466723 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff5f8154d70 0x7ff5f80c9750 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7ff5e800aa50 tx=0x7ff5e800a2f0 comp rx=0 tx=0).stop 2023-10-06T03:29:51.626 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.621+0000 7f97f7058700 1 -- start start 2023-10-06T03:29:51.626 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.621+0000 7f97f7058700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f97f0154d70 0x7f97f00c9750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:51.627 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.621+0000 7ff5fca25700 1 -- 172.21.15.110:0/5466723 shutdown_connections 2023-10-06T03:29:51.627 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.621+0000 7ff5fca25700 1 --2- 172.21.15.110:0/5466723 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff5f8154d70 0x7ff5f80c9750 secure :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7ff5e800aa50 tx=0x7ff5e800a2f0 comp rx=0 tx=0).stop 2023-10-06T03:29:51.627 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.621+0000 7ff5fca25700 1 -- 172.21.15.110:0/5466723 >> 172.21.15.110:0/5466723 conn(0x7ff5f80cc560 msgr2=0x7ff5f80c8500 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:29:51.627 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.621+0000 7f97f7058700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f97f0155140 0x7f97f00c75e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:51.627 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.621+0000 7f97f7058700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97f00c1c40 con 0x7f97f00c20d0 2023-10-06T03:29:51.629 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.621+0000 7ff5fca25700 1 -- 172.21.15.110:0/5466723 shutdown_connections 2023-10-06T03:29:51.629 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.621+0000 7f97f7058700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f97f00c7b20 con 0x7f97f0154d70 2023-10-06T03:29:51.630 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.621+0000 7f97f7058700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97f00c7d10 con 0x7f97f0155140 2023-10-06T03:29:51.630 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.621+0000 7ff5fca25700 1 -- 172.21.15.110:0/5466723 wait complete. 2023-10-06T03:29:51.631 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.621+0000 7f97f55f5700 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f97f00c20d0 0x7f97f00ca3d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:40114/0 (socket says 172.21.15.110:40114) 2023-10-06T03:29:51.631 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.621+0000 7f97f55f5700 1 -- 172.21.15.110:0/4025440945 learned_addr learned my addr 172.21.15.110:0/4025440945 (peer_addr_for_me v1:172.21.15.110:0/0) 2023-10-06T03:29:51.632 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.621+0000 7f97f4df4700 1 --2- 172.21.15.110:0/4025440945 >> v2:172.21.15.192:3300/0 conn(0x7f97f0154d70 0x7f97f00c9750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:51.632 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.621+0000 7f97ef7fe700 1 -- 172.21.15.110:0/4025440945 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2500929329 0 0) 0x7f97f00c1c40 con 0x7f97f00c20d0 2023-10-06T03:29:51.632 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.621+0000 7f97ef7fe700 1 -- 172.21.15.110:0/4025440945 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f97d8003570 con 0x7f97f00c20d0 2023-10-06T03:29:51.632 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.621+0000 7f97f4df4700 1 -- 172.21.15.110:0/4025440945 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f97f0155140 msgr2=0x7f97f00c75e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:29:51.632 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.621+0000 7f97f4df4700 1 --2- 172.21.15.110:0/4025440945 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f97f0155140 0x7f97f00c75e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:51.632 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.621+0000 7f97f4df4700 1 -- 172.21.15.110:0/4025440945 >> v1:172.21.15.110:6789/0 conn(0x7f97f00c20d0 legacy=0x7f97f00ca3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:51.632 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.621+0000 7f97f4df4700 1 -- 172.21.15.110:0/4025440945 --> v2:172.21.15.192:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97f00c15f0 con 0x7f97f0154d70 2023-10-06T03:29:51.632 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.621+0000 7f97f4df4700 1 --2- 172.21.15.110:0/4025440945 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f97f0154d70 0x7f97f00c9750 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f97e000aa90 tx=0x7f97e000a330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=825fd0af377e30dc server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:29:51.632 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.621+0000 7f97ef7fe700 1 -- 172.21.15.110:0/4025440945 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97e000c970 con 0x7f97f0154d70 2023-10-06T03:29:51.632 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.625+0000 7f97ef7fe700 1 -- 172.21.15.110:0/4025440945 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97e000cad0 con 0x7f97f0154d70 2023-10-06T03:29:51.632 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.625+0000 7f97ef7fe700 1 -- 172.21.15.110:0/4025440945 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97e000ce50 con 0x7f97f0154d70 2023-10-06T03:29:51.633 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.625+0000 7ff5fca25700 1 Processor -- start 2023-10-06T03:29:51.633 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.625+0000 7ff5fca25700 1 -- start start 2023-10-06T03:29:51.634 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.625+0000 7f97f7058700 1 -- 172.21.15.110:0/4025440945 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f97f0154d70 msgr2=0x7f97f00c9750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:51.634 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.625+0000 7f97f7058700 1 --2- 172.21.15.110:0/4025440945 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f97f0154d70 0x7f97f00c9750 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f97e000aa90 tx=0x7f97e000a330 comp rx=0 tx=0).stop 2023-10-06T03:29:51.634 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.625+0000 7f97f7058700 1 -- 172.21.15.110:0/4025440945 shutdown_connections 2023-10-06T03:29:51.634 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.625+0000 7f97f7058700 1 --2- 172.21.15.110:0/4025440945 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f97f0155140 0x7f97f00c75e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:51.634 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.625+0000 7f97f7058700 1 --2- 172.21.15.110:0/4025440945 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f97f0154d70 0x7f97f00c9750 secure :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f97e000aa90 tx=0x7f97e000a330 comp rx=0 tx=0).stop 2023-10-06T03:29:51.634 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.625+0000 7f97f7058700 1 -- 172.21.15.110:0/4025440945 >> 172.21.15.110:0/4025440945 conn(0x7f97f00cc560 msgr2=0x7f97f00c8500 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:29:51.634 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.625+0000 7f97f7058700 1 -- 172.21.15.110:0/4025440945 shutdown_connections 2023-10-06T03:29:51.635 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.625+0000 7ff5fca25700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff5f80c75e0 0x7ff5f8165280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:51.635 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.625+0000 7ff5fca25700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff5f8155140 0x7ff5f81576d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:51.635 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.625+0000 7ff5fca25700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff5f8161a10 0x7ff5f81556b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:51.635 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.625+0000 7ff5fca25700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5f81657c0 con 0x7ff5f8155140 2023-10-06T03:29:51.635 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.625+0000 7ff5f5d9b700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff5f8155140 0x7ff5f81576d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:51.635 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.625+0000 7ff5f5d9b700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff5f8155140 0x7ff5f81576d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:47568/0 (socket says 172.21.15.110:47568) 2023-10-06T03:29:51.636 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.625+0000 7ff5f659c700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff5f80c75e0 0x7ff5f8165280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:51.636 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.625+0000 7ff5f5d9b700 1 -- 172.21.15.110:0/3039315338 learned_addr learned my addr 172.21.15.110:0/3039315338 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:29:51.636 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.625+0000 7ff5fca25700 1 -- 172.21.15.110:0/3039315338 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5f81596e0 con 0x7ff5f80c75e0 2023-10-06T03:29:51.636 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.625+0000 7ff5fca25700 1 -- 172.21.15.110:0/3039315338 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5f8157c10 con 0x7ff5f8161a10 2023-10-06T03:29:51.636 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.625+0000 7ff5f6d9d700 1 --2- 172.21.15.110:0/3039315338 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff5f8161a10 0x7ff5f81556b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:51.636 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.625+0000 7efed52ba700 1 Processor -- start 2023-10-06T03:29:51.636 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.625+0000 7efed52ba700 1 -- start start 2023-10-06T03:29:51.637 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.625+0000 7ff5f5d9b700 1 -- 172.21.15.110:0/3039315338 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff5f8161a10 msgr2=0x7ff5f81556b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:51.637 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.625+0000 7ff5f5d9b700 1 --2- 172.21.15.110:0/3039315338 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff5f8161a10 0x7ff5f81556b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:51.638 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.625+0000 7ff5f5d9b700 1 -- 172.21.15.110:0/3039315338 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff5f80c75e0 msgr2=0x7ff5f8165280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:51.638 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.625+0000 7ff5f5d9b700 1 --2- 172.21.15.110:0/3039315338 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff5f80c75e0 0x7ff5f8165280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:51.638 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.625+0000 7ff5f5d9b700 1 -- 172.21.15.110:0/3039315338 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5e800a590 con 0x7ff5f8155140 2023-10-06T03:29:51.638 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.625+0000 7ff5f5d9b700 1 --2- 172.21.15.110:0/3039315338 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff5f8155140 0x7ff5f81576d0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7ff5f000cb60 tx=0x7ff5f0001770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:29:51.639 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.625+0000 7efed52ba700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7efed00694a0 0x7efed00698f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:51.639 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.625+0000 7efed52ba700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7efed01677b0 0x7efed0167c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:51.639 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.625+0000 7efed52ba700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efed006be80 con 0x7efed00aac30 2023-10-06T03:29:51.639 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.625+0000 7efed52ba700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efed006a4b0 con 0x7efed00694a0 2023-10-06T03:29:51.639 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.625+0000 7efed52ba700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efed0168190 con 0x7efed01677b0 2023-10-06T03:29:51.639 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.625+0000 7f97f7058700 1 -- 172.21.15.110:0/4025440945 wait complete. 2023-10-06T03:29:51.640 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.625+0000 7ff5e67fc700 1 -- 172.21.15.110:0/3039315338 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5f000a7e0 con 0x7ff5f8155140 2023-10-06T03:29:51.640 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.625+0000 7ff5fca25700 1 -- 172.21.15.110:0/3039315338 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff5f8157ee0 con 0x7ff5f8155140 2023-10-06T03:29:51.640 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.625+0000 7efecffff700 1 --1- >> v1:172.21.15.192:6789/0 conn(0x7efed00aac30 0x7efed006c310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.110:55690/0 (socket says 172.21.15.110:55690) 2023-10-06T03:29:51.641 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.625+0000 7efed4ab9700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7efed01677b0 0x7efed0167c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:51.641 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.625+0000 7efecffff700 1 -- 172.21.15.110:0/3219450765 learned_addr learned my addr 172.21.15.110:0/3219450765 (peer_addr_for_me v1:172.21.15.110:0/0) 2023-10-06T03:29:51.641 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.625+0000 7efecf7fe700 1 --2- 172.21.15.110:0/3219450765 >> v2:172.21.15.110:3300/0 conn(0x7efed00694a0 0x7efed00698f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:51.641 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.625+0000 7ff5fca25700 1 -- 172.21.15.110:0/3039315338 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5f81584a0 con 0x7ff5f8155140 2023-10-06T03:29:51.642 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.629+0000 7efed4ab9700 1 -- 172.21.15.110:0/3219450765 >> v2:172.21.15.110:3300/0 conn(0x7efed00694a0 msgr2=0x7efed00698f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:29:51.642 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.629+0000 7efed4ab9700 1 --2- 172.21.15.110:0/3219450765 >> v2:172.21.15.110:3300/0 conn(0x7efed00694a0 0x7efed00698f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:51.642 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.629+0000 7efed4ab9700 1 -- 172.21.15.110:0/3219450765 >> v1:172.21.15.192:6789/0 conn(0x7efed00aac30 legacy=0x7efed006c310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:51.642 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.629+0000 7efed4ab9700 1 -- 172.21.15.110:0/3219450765 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efed00ab000 con 0x7efed01677b0 2023-10-06T03:29:51.642 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.629+0000 7efecf7fe700 1 --2- 172.21.15.110:0/3219450765 >> v2:172.21.15.110:3300/0 conn(0x7efed00694a0 0x7efed00698f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-06T03:29:51.643 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.629+0000 7ff5e67fc700 1 -- 172.21.15.110:0/3039315338 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5f000a940 con 0x7ff5f8155140 2023-10-06T03:29:51.643 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.629+0000 7ff5e67fc700 1 -- 172.21.15.110:0/3039315338 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5f000ace0 con 0x7ff5f8155140 2023-10-06T03:29:51.644 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.629+0000 7efed4ab9700 1 --2- 172.21.15.110:0/3219450765 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7efed01677b0 0x7efed0167c50 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7efec8009e60 tx=0x7efec8009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=40c443a02757b548 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:29:51.644 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.629+0000 7efeceffd700 1 -- 172.21.15.110:0/3219450765 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efec800e070 con 0x7efed01677b0 2023-10-06T03:29:51.645 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.629+0000 7efeceffd700 1 -- 172.21.15.110:0/3219450765 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efec800ba60 con 0x7efed01677b0 2023-10-06T03:29:51.645 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.629+0000 7efeceffd700 1 -- 172.21.15.110:0/3219450765 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efec800be00 con 0x7efed01677b0 2023-10-06T03:29:51.645 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.629+0000 7ff5fca25700 1 -- 172.21.15.110:0/3039315338 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff5dc0052b0 con 0x7ff5f8155140 2023-10-06T03:29:51.645 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.629+0000 7ff5e67fc700 1 -- 172.21.15.110:0/3039315338 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7ff5f0012630 con 0x7ff5f8155140 2023-10-06T03:29:51.645 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.629+0000 7ff5e67fc700 1 --2- 172.21.15.110:0/3039315338 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7ff5d8060620 0x7ff5d8062ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:51.646 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.629+0000 7efed52ba700 1 -- 172.21.15.110:0/3219450765 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7efed01677b0 msgr2=0x7efed0167c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:51.646 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.629+0000 7efed52ba700 1 --2- 172.21.15.110:0/3219450765 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7efed01677b0 0x7efed0167c50 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7efec8009e60 tx=0x7efec8009780 comp rx=0 tx=0).stop 2023-10-06T03:29:51.646 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.629+0000 7ff5f659c700 1 --2- 172.21.15.110:0/3039315338 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7ff5d8060620 0x7ff5d8062ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:51.646 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.629+0000 7ff5e67fc700 1 -- 172.21.15.110:0/3039315338 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6809+0+0 (secure 0 0 0) 0x7ff5f007cf00 con 0x7ff5f8155140 2023-10-06T03:29:51.647 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.629+0000 7ff5f659c700 1 --2- 172.21.15.110:0/3039315338 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7ff5d8060620 0x7ff5d8062ad0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7ff5e800aa50 tx=0x7ff5e801c000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:29:51.648 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.633+0000 7efed52ba700 1 -- 172.21.15.110:0/3219450765 shutdown_connections 2023-10-06T03:29:51.648 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.633+0000 7efed52ba700 1 --2- 172.21.15.110:0/3219450765 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7efed01677b0 0x7efed0167c50 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:51.648 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.633+0000 7efed52ba700 1 --2- 172.21.15.110:0/3219450765 >> v2:172.21.15.110:3300/0 conn(0x7efed00694a0 0x7efed00698f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:51.648 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.633+0000 7efed52ba700 1 -- 172.21.15.110:0/3219450765 >> 172.21.15.110:0/3219450765 conn(0x7efed006dce0 msgr2=0x7efed006e0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:29:51.648 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.633+0000 7efed52ba700 1 -- 172.21.15.110:0/3219450765 shutdown_connections 2023-10-06T03:29:51.649 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.633+0000 7efed52ba700 1 -- 172.21.15.110:0/3219450765 wait complete. 2023-10-06T03:29:51.649 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.633+0000 7efed52ba700 1 Processor -- start 2023-10-06T03:29:51.649 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.633+0000 7efed52ba700 1 -- start start 2023-10-06T03:29:51.649 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.633+0000 7efed52ba700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7efed00694a0 0x7efed00afcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:51.649 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.633+0000 7efed52ba700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7efed00aac30 0x7efed00c4260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:51.649 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.633+0000 7efed52ba700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efed01677b0 0x7efed00c47a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:51.650 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.633+0000 7efed52ba700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efed00b0e20 con 0x7efed00694a0 2023-10-06T03:29:51.650 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.633+0000 7efed52ba700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efed0182d50 con 0x7efed01677b0 2023-10-06T03:29:51.650 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.633+0000 7efed52ba700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efed0182a70 con 0x7efed00aac30 2023-10-06T03:29:51.650 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.633+0000 7efecffff700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7efed00694a0 0x7efed00afcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:51.650 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.633+0000 7efecffff700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7efed00694a0 0x7efed00afcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:47596/0 (socket says 172.21.15.110:47596) 2023-10-06T03:29:51.651 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.633+0000 7efecffff700 1 -- 172.21.15.110:0/3342450740 learned_addr learned my addr 172.21.15.110:0/3342450740 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:29:51.651 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.633+0000 7efecf7fe700 1 --2- 172.21.15.110:0/3342450740 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7efed00aac30 0x7efed00c4260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:51.651 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.633+0000 7efed4ab9700 1 --2- 172.21.15.110:0/3342450740 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efed01677b0 0x7efed00c47a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:51.651 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.637+0000 7efecffff700 1 -- 172.21.15.110:0/3342450740 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7efed00aac30 msgr2=0x7efed00c4260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:51.651 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.637+0000 7efecffff700 1 --2- 172.21.15.110:0/3342450740 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7efed00aac30 0x7efed00c4260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:51.652 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.637+0000 7efecffff700 1 -- 172.21.15.110:0/3342450740 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efed01677b0 msgr2=0x7efed00c47a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:51.652 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.637+0000 7efecffff700 1 --2- 172.21.15.110:0/3342450740 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efed01677b0 0x7efed00c47a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:51.652 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.637+0000 7efecffff700 1 -- 172.21.15.110:0/3342450740 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efec4009420 con 0x7efed00694a0 2023-10-06T03:29:51.652 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.637+0000 7efecffff700 1 --2- 172.21.15.110:0/3342450740 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7efed00694a0 0x7efed00afcb0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7efec000c790 tx=0x7efec0007ef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:29:51.655 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.629+0000 7f97f7058700 1 Processor -- start 2023-10-06T03:29:51.655 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.629+0000 7f97f7058700 1 -- start start 2023-10-06T03:29:51.655 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.629+0000 7f97f7058700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f97f00c20d0 0x7f97f00b0b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:51.655 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.629+0000 7f97f7058700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f97f0155140 0x7f97f00c3aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:51.655 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.629+0000 7f97f7058700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f97f00b10a0 0x7f97f00af650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:51.655 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.629+0000 7f97f7058700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97f00abe20 con 0x7f97f00b10a0 2023-10-06T03:29:51.655 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.629+0000 7f97f7058700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97f00aaf30 con 0x7f97f00c20d0 2023-10-06T03:29:51.656 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.629+0000 7f97f7058700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97f00ab210 con 0x7f97f0155140 2023-10-06T03:29:51.656 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.629+0000 7f97effff700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f97f0155140 0x7f97f00c3aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:51.656 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.629+0000 7f97effff700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f97f0155140 0x7f97f00c3aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3301/0 says I am v2:172.21.15.110:34664/0 (socket says 172.21.15.110:34664) 2023-10-06T03:29:51.656 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.629+0000 7f97effff700 1 -- 172.21.15.110:0/2483651091 learned_addr learned my addr 172.21.15.110:0/2483651091 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:29:51.656 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.629+0000 7f97f55f5700 1 --2- 172.21.15.110:0/2483651091 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f97f00b10a0 0x7f97f00af650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:51.656 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.629+0000 7f97effff700 1 -- 172.21.15.110:0/2483651091 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f97f00c20d0 msgr2=0x7f97f00b0b60 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-06T03:29:51.656 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.629+0000 7f97effff700 1 --2- 172.21.15.110:0/2483651091 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f97f00c20d0 0x7f97f00b0b60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:51.656 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.629+0000 7f97effff700 1 -- 172.21.15.110:0/2483651091 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f97f00b10a0 msgr2=0x7f97f00af650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:51.656 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.629+0000 7f97effff700 1 --2- 172.21.15.110:0/2483651091 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f97f00b10a0 0x7f97f00af650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:51.656 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.629+0000 7f97effff700 1 -- 172.21.15.110:0/2483651091 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97e000a5d0 con 0x7f97f0155140 2023-10-06T03:29:51.657 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.629+0000 7f97f55f5700 1 --2- 172.21.15.110:0/2483651091 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f97f00b10a0 0x7f97f00af650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-06T03:29:51.657 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.629+0000 7f97effff700 1 --2- 172.21.15.110:0/2483651091 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f97f0155140 0x7f97f00c3aa0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f97e400b810 tx=0x7f97e4009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:29:51.657 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.629+0000 7f97ecff9700 1 -- 172.21.15.110:0/2483651091 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97e400f5f0 con 0x7f97f0155140 2023-10-06T03:29:51.657 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.629+0000 7f97ecff9700 1 -- 172.21.15.110:0/2483651091 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97e400f770 con 0x7f97f0155140 2023-10-06T03:29:51.657 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.629+0000 7f97f7058700 1 -- 172.21.15.110:0/2483651091 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97f00ab4f0 con 0x7f97f0155140 2023-10-06T03:29:51.657 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.629+0000 7f97ecff9700 1 -- 172.21.15.110:0/2483651091 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97e400fb30 con 0x7f97f0155140 2023-10-06T03:29:51.657 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.629+0000 7f97f7058700 1 -- 172.21.15.110:0/2483651091 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97f00abab0 con 0x7f97f0155140 2023-10-06T03:29:51.657 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.637+0000 7f97ecff9700 1 -- 172.21.15.110:0/2483651091 <== mon.2 v2:172.21.15.192:3301/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7f97e4010480 con 0x7f97f0155140 2023-10-06T03:29:51.657 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.637+0000 7f97ecff9700 1 --2- 172.21.15.110:0/2483651091 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f97dc0a45e0 0x7f97dc0a6a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:51.657 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.637+0000 7f97ecff9700 1 -- 172.21.15.110:0/2483651091 <== mon.2 v2:172.21.15.192:3301/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6809+0+0 (secure 0 0 0) 0x7f97e407cc20 con 0x7f97f0155140 2023-10-06T03:29:51.657 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.637+0000 7f97f4df4700 1 --2- 172.21.15.110:0/2483651091 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f97dc0a45e0 0x7f97dc0a6a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:51.658 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.637+0000 7f97f7058700 1 -- 172.21.15.110:0/2483651091 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f97d00052b0 con 0x7f97f0155140 2023-10-06T03:29:51.658 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.637+0000 7ff5e67fc700 1 -- 172.21.15.110:0/3039315338 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7ff5f004d160 con 0x7ff5f8155140 2023-10-06T03:29:51.658 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.637+0000 7efebffff700 1 -- 172.21.15.110:0/3342450740 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efec0005b30 con 0x7efed00694a0 2023-10-06T03:29:51.659 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.637+0000 7efebffff700 1 -- 172.21.15.110:0/3342450740 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efec0005cb0 con 0x7efed00694a0 2023-10-06T03:29:51.659 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.637+0000 7efebffff700 1 -- 172.21.15.110:0/3342450740 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efec0007320 con 0x7efed00694a0 2023-10-06T03:29:51.659 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.637+0000 7efed52ba700 1 -- 172.21.15.110:0/3342450740 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efec8009a20 con 0x7efed00694a0 2023-10-06T03:29:51.659 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.637+0000 7efed52ba700 1 -- 172.21.15.110:0/3342450740 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efed017eca0 con 0x7efed00694a0 2023-10-06T03:29:51.659 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.641+0000 7f97f4df4700 1 --2- 172.21.15.110:0/2483651091 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f97dc0a45e0 0x7f97dc0a6a90 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f97f006ee50 tx=0x7f97e0019000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:29:51.659 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.641+0000 7f91e089d700 1 Processor -- start 2023-10-06T03:29:51.660 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.641+0000 7f91e089d700 1 -- start start 2023-10-06T03:29:51.660 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.641+0000 7efebffff700 1 -- 172.21.15.110:0/3342450740 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7efec001b3b0 con 0x7efed00694a0 2023-10-06T03:29:51.660 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.641+0000 7efed52ba700 1 -- 172.21.15.110:0/3342450740 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efed017e820 con 0x7efed00694a0 2023-10-06T03:29:51.661 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.641+0000 7f91e089d700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f91dc0c8a30 0x7f91dc15cdc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:51.661 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.641+0000 7f91e089d700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f91dc15d300 0x7f91dc06b9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:51.661 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.641+0000 7f91e089d700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91dc0694a0 con 0x7f91dc156980 2023-10-06T03:29:51.661 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.641+0000 7f91e089d700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f91dc06a020 con 0x7f91dc15d300 2023-10-06T03:29:51.661 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.641+0000 7f91e089d700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f91dc06a4b0 con 0x7f91dc0c8a30 2023-10-06T03:29:51.661 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.641+0000 7f91db59e700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f91dc15d300 0x7f91dc06b9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:51.661 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.641+0000 7f91db59e700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f91dc15d300 0x7f91dc06b9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:47600/0 (socket says 172.21.15.110:47600) 2023-10-06T03:29:51.662 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.641+0000 7f91db59e700 1 -- 172.21.15.110:0/2325177139 learned_addr learned my addr 172.21.15.110:0/2325177139 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:29:51.662 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.641+0000 7f91da59c700 1 --2- 172.21.15.110:0/2325177139 >> v2:172.21.15.192:3300/0 conn(0x7f91dc0c8a30 0x7f91dc15cdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:51.662 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.641+0000 7efebffff700 1 --2- 172.21.15.110:0/3342450740 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7efeb4060520 0x7efeb40629d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:51.662 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.641+0000 7efebffff700 1 -- 172.21.15.110:0/3342450740 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6809+0+0 (secure 0 0 0) 0x7efec007c180 con 0x7efed00694a0 2023-10-06T03:29:51.663 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.641+0000 7f91db59e700 1 -- 172.21.15.110:0/2325177139 >> v2:172.21.15.192:3300/0 conn(0x7f91dc0c8a30 msgr2=0x7f91dc15cdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:29:51.663 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.641+0000 7f91db59e700 1 --2- 172.21.15.110:0/2325177139 >> v2:172.21.15.192:3300/0 conn(0x7f91dc0c8a30 0x7f91dc15cdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:51.663 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.641+0000 7f91db59e700 1 -- 172.21.15.110:0/2325177139 >> v1:172.21.15.110:6789/0 conn(0x7f91dc156980 legacy=0x7f91dc0c8320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:51.663 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.641+0000 7f91db59e700 1 -- 172.21.15.110:0/2325177139 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91dc0696f0 con 0x7f91dc15d300 2023-10-06T03:29:51.663 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.641+0000 7f91db59e700 1 --2- 172.21.15.110:0/2325177139 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f91dc15d300 0x7f91dc06b9e0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f91dc0ae6a0 tx=0x7f91d4009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=858619a837e6b711 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:29:51.664 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.645+0000 7efecf7fe700 1 --2- 172.21.15.110:0/3342450740 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7efeb4060520 0x7efeb40629d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:51.664 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.645+0000 7efecf7fe700 1 --2- 172.21.15.110:0/3342450740 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7efeb4060520 0x7efeb40629d0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7efed00cb130 tx=0x7efec40091a0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:29:51.665 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.645+0000 7f91d9d9b700 1 -- 172.21.15.110:0/2325177139 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91d400e070 con 0x7f91dc15d300 2023-10-06T03:29:51.665 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.645+0000 7f326b59e700 1 Processor -- start 2023-10-06T03:29:51.665 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.645+0000 7f91d9d9b700 1 -- 172.21.15.110:0/2325177139 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91d400ba20 con 0x7f91dc15d300 2023-10-06T03:29:51.665 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.645+0000 7efebffff700 1 -- 172.21.15.110:0/3342450740 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7efed017e820 con 0x7efed00694a0 2023-10-06T03:29:51.666 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.649+0000 7f91e089d700 1 -- 172.21.15.110:0/2325177139 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f91dc15d300 msgr2=0x7f91dc06b9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:51.666 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.649+0000 7f91e089d700 1 --2- 172.21.15.110:0/2325177139 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f91dc15d300 0x7f91dc06b9e0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f91dc0ae6a0 tx=0x7f91d4009840 comp rx=0 tx=0).stop 2023-10-06T03:29:51.666 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.649+0000 7f91e089d700 1 -- 172.21.15.110:0/2325177139 shutdown_connections 2023-10-06T03:29:51.666 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.649+0000 7f91e089d700 1 --2- 172.21.15.110:0/2325177139 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f91dc15d300 0x7f91dc06b9e0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:51.666 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.649+0000 7f91e089d700 1 --2- 172.21.15.110:0/2325177139 >> v2:172.21.15.192:3300/0 conn(0x7f91dc0c8a30 0x7f91dc15cdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:51.666 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.649+0000 7f91e089d700 1 -- 172.21.15.110:0/2325177139 >> 172.21.15.110:0/2325177139 conn(0x7f91dc06dce0 msgr2=0x7f91dc06e0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:29:51.667 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.645+0000 7f326b59e700 1 -- start start 2023-10-06T03:29:51.667 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.649+0000 7f326b59e700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f326c154cf0 0x7f326c0c7330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:51.667 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.649+0000 7f326b59e700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f326c1550c0 0x7f326c1595e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:51.667 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.649+0000 7f326b59e700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f326c06c3c0 0x7f326c0c80e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:51.667 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.649+0000 7f326b59e700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f326c06cbd0 con 0x7f326c1550c0 2023-10-06T03:29:51.667 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.649+0000 7f326b59e700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f326c0c6ea0 con 0x7f326c06c3c0 2023-10-06T03:29:51.667 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.649+0000 7f326b59e700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f326c0c6570 con 0x7f326c154cf0 2023-10-06T03:29:51.668 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.649+0000 7f3269d9b700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f326c1550c0 0x7f326c1595e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:51.668 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.649+0000 7f3269d9b700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f326c1550c0 0x7f326c1595e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:47614/0 (socket says 172.21.15.110:47614) 2023-10-06T03:29:51.668 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.649+0000 7f3269d9b700 1 -- 172.21.15.110:0/538460933 learned_addr learned my addr 172.21.15.110:0/538460933 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:29:51.668 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.649+0000 7f91e089d700 1 -- 172.21.15.110:0/2325177139 shutdown_connections 2023-10-06T03:29:51.668 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.649+0000 7f91e089d700 1 -- 172.21.15.110:0/2325177139 wait complete. 2023-10-06T03:29:51.669 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.649+0000 7f326a59c700 1 --2- 172.21.15.110:0/538460933 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f326c154cf0 0x7f326c0c7330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:51.669 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.649+0000 7f326ad9d700 1 --2- 172.21.15.110:0/538460933 >> v2:172.21.15.192:3300/0 conn(0x7f326c06c3c0 0x7f326c0c80e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:51.669 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.649+0000 7f3269d9b700 1 -- 172.21.15.110:0/538460933 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f326c154cf0 msgr2=0x7f326c0c7330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:29:51.669 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.649+0000 7f3269d9b700 1 --2- 172.21.15.110:0/538460933 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f326c154cf0 0x7f326c0c7330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:51.669 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.649+0000 7f3269d9b700 1 -- 172.21.15.110:0/538460933 >> v2:172.21.15.192:3300/0 conn(0x7f326c06c3c0 msgr2=0x7f326c0c80e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:29:51.669 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.649+0000 7f3269d9b700 1 --2- 172.21.15.110:0/538460933 >> v2:172.21.15.192:3300/0 conn(0x7f326c06c3c0 0x7f326c0c80e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:51.669 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.649+0000 7f3269d9b700 1 -- 172.21.15.110:0/538460933 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f326c0c6850 con 0x7f326c1550c0 2023-10-06T03:29:51.669 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.649+0000 7f3269d9b700 1 --2- 172.21.15.110:0/538460933 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f326c1550c0 0x7f326c1595e0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f326c0c6000 tx=0x7f3264007690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=676cd1d914d8660f server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:29:51.670 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.649+0000 7f326959a700 1 -- 172.21.15.110:0/538460933 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3264010070 con 0x7f326c1550c0 2023-10-06T03:29:51.670 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.649+0000 7f326959a700 1 -- 172.21.15.110:0/538460933 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f326400dc80 con 0x7f326c1550c0 2023-10-06T03:29:51.670 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.649+0000 7f326959a700 1 -- 172.21.15.110:0/538460933 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3264007b40 con 0x7f326c1550c0 2023-10-06T03:29:51.670 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.649+0000 7f91e089d700 1 Processor -- start 2023-10-06T03:29:51.670 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.649+0000 7f91e089d700 1 -- start start 2023-10-06T03:29:51.671 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.649+0000 7f326b59e700 1 -- 172.21.15.110:0/538460933 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f326c1550c0 msgr2=0x7f326c1595e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:51.671 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.649+0000 7f326b59e700 1 --2- 172.21.15.110:0/538460933 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f326c1550c0 0x7f326c1595e0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f326c0c6000 tx=0x7f3264007690 comp rx=0 tx=0).stop 2023-10-06T03:29:51.671 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.649+0000 7f326b59e700 1 -- 172.21.15.110:0/538460933 shutdown_connections 2023-10-06T03:29:51.672 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.649+0000 7f326b59e700 1 --2- 172.21.15.110:0/538460933 >> v2:172.21.15.192:3300/0 conn(0x7f326c06c3c0 0x7f326c0c80e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:51.672 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.649+0000 7f326b59e700 1 --2- 172.21.15.110:0/538460933 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f326c1550c0 0x7f326c1595e0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:51.672 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.649+0000 7f326b59e700 1 --2- 172.21.15.110:0/538460933 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f326c154cf0 0x7f326c0c7330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:51.672 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.649+0000 7f326b59e700 1 -- 172.21.15.110:0/538460933 >> 172.21.15.110:0/538460933 conn(0x7f326c06dce0 msgr2=0x7f326c06e0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:29:51.672 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.649+0000 7f326b59e700 1 -- 172.21.15.110:0/538460933 shutdown_connections 2023-10-06T03:29:51.672 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.649+0000 7f326b59e700 1 -- 172.21.15.110:0/538460933 wait complete. 2023-10-06T03:29:51.673 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.649+0000 7f91e089d700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f91dc0c8a30 0x7f91dc0b0e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:51.673 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.649+0000 7f91e089d700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f91dc156980 0x7f91dc151f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:51.673 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.649+0000 7f91e089d700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f91dc15d300 0x7f91dc0ae8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:51.673 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.649+0000 7f91e089d700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91dc0b03f0 con 0x7f91dc15d300 2023-10-06T03:29:51.673 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.649+0000 7f91e089d700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91dc0b09b0 con 0x7f91dc156980 2023-10-06T03:29:51.673 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.649+0000 7f91e089d700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91dc0b1380 con 0x7f91dc0c8a30 2023-10-06T03:29:51.673 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.649+0000 7f91db59e700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f91dc15d300 0x7f91dc0ae8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:51.673 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.649+0000 7f91db59e700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f91dc15d300 0x7f91dc0ae8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:47624/0 (socket says 172.21.15.110:47624) 2023-10-06T03:29:51.674 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.649+0000 7f91db59e700 1 -- 172.21.15.110:0/1543111619 learned_addr learned my addr 172.21.15.110:0/1543111619 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:29:51.674 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.649+0000 7f91dad9d700 1 --2- 172.21.15.110:0/1543111619 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f91dc0c8a30 0x7f91dc0b0e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:51.674 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.653+0000 7f91db59e700 1 -- 172.21.15.110:0/1543111619 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f91dc0c8a30 msgr2=0x7f91dc0b0e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:51.674 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.653+0000 7f91db59e700 1 --2- 172.21.15.110:0/1543111619 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f91dc0c8a30 0x7f91dc0b0e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:51.674 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.653+0000 7f91db59e700 1 -- 172.21.15.110:0/1543111619 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f91dc156980 msgr2=0x7f91dc151f10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-06T03:29:51.675 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.653+0000 7f91db59e700 1 --2- 172.21.15.110:0/1543111619 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f91dc156980 0x7f91dc151f10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:51.675 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.653+0000 7f91db59e700 1 -- 172.21.15.110:0/1543111619 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91d4009ae0 con 0x7f91dc15d300 2023-10-06T03:29:51.675 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.653+0000 7f91db59e700 1 --2- 172.21.15.110:0/1543111619 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f91dc15d300 0x7f91dc0ae8e0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f91d400b6c0 tx=0x7f91d4018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:29:51.676 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.653+0000 7f326b59e700 1 Processor -- start 2023-10-06T03:29:51.678 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.653+0000 7f326b59e700 1 -- start start 2023-10-06T03:29:51.678 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.653+0000 7f326b59e700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f326c06c3c0 0x7f326c165130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:51.678 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.653+0000 7f326b59e700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f326c154cf0 0x7f326c1578b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:51.678 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.653+0000 7f326b59e700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f326c1550c0 0x7f326c157df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:51.678 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.653+0000 7f326b59e700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f326c156610 con 0x7f326c154cf0 2023-10-06T03:29:51.678 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.653+0000 7f326b59e700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f326c14ea80 con 0x7f326c06c3c0 2023-10-06T03:29:51.678 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.653+0000 7f326b59e700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f326c157420 con 0x7f326c1550c0 2023-10-06T03:29:51.679 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.653+0000 7f3269d9b700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f326c154cf0 0x7f326c1578b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:51.679 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.653+0000 7f3269d9b700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f326c154cf0 0x7f326c1578b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:47626/0 (socket says 172.21.15.110:47626) 2023-10-06T03:29:51.679 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.653+0000 7f3269d9b700 1 -- 172.21.15.110:0/2127891273 learned_addr learned my addr 172.21.15.110:0/2127891273 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:29:51.679 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.653+0000 7f326a59c700 1 --2- 172.21.15.110:0/2127891273 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f326c06c3c0 0x7f326c165130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:51.679 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.653+0000 7f326ad9d700 1 --2- 172.21.15.110:0/2127891273 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f326c1550c0 0x7f326c157df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:51.679 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.653+0000 7f3269d9b700 1 -- 172.21.15.110:0/2127891273 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f326c1550c0 msgr2=0x7f326c157df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:51.679 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.653+0000 7f3269d9b700 1 --2- 172.21.15.110:0/2127891273 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f326c1550c0 0x7f326c157df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:51.679 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.653+0000 7f3269d9b700 1 -- 172.21.15.110:0/2127891273 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f326c06c3c0 msgr2=0x7f326c165130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:51.679 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.653+0000 7f3269d9b700 1 --2- 172.21.15.110:0/2127891273 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f326c06c3c0 0x7f326c165130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:51.679 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.653+0000 7f3269d9b700 1 -- 172.21.15.110:0/2127891273 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f326400be70 con 0x7f326c154cf0 2023-10-06T03:29:51.679 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.653+0000 7f3269d9b700 1 --2- 172.21.15.110:0/2127891273 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f326c154cf0 0x7f326c1578b0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f326400d8c0 tx=0x7f326401b000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:29:51.680 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.657+0000 7f91c2ffd700 1 -- 172.21.15.110:0/1543111619 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91d4025460 con 0x7f91dc15d300 2023-10-06T03:29:51.680 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.657+0000 7f91c2ffd700 1 -- 172.21.15.110:0/1543111619 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91d40255e0 con 0x7f91dc15d300 2023-10-06T03:29:51.681 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.657+0000 7f91c2ffd700 1 -- 172.21.15.110:0/1543111619 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91d40259a0 con 0x7f91dc15d300 2023-10-06T03:29:51.681 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.657+0000 7f91e089d700 1 -- 172.21.15.110:0/1543111619 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91dc0af3a0 con 0x7f91dc15d300 2023-10-06T03:29:51.681 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.657+0000 7f91e089d700 1 -- 172.21.15.110:0/1543111619 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f91dc0b0640 con 0x7f91dc15d300 2023-10-06T03:29:51.681 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.657+0000 7f91c2ffd700 1 -- 172.21.15.110:0/1543111619 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7f91d4025d50 con 0x7f91dc15d300 2023-10-06T03:29:51.682 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.657+0000 7f91c2ffd700 1 --2- 172.21.15.110:0/1543111619 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f91c8060620 0x7f91c8062ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:51.682 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.657+0000 7f91c2ffd700 1 -- 172.21.15.110:0/1543111619 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6809+0+0 (secure 0 0 0) 0x7f91d407e9a0 con 0x7f91dc15d300 2023-10-06T03:29:51.682 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.657+0000 7f91dad9d700 1 --2- 172.21.15.110:0/1543111619 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f91c8060620 0x7f91c8062ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:51.682 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.657+0000 7f91dad9d700 1 --2- 172.21.15.110:0/1543111619 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f91c8060620 0x7f91c8062ad0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f91cc003aa0 tx=0x7f91cc000bc0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:29:51.682 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.665+0000 7f325a7fc700 1 -- 172.21.15.110:0/2127891273 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3264007000 con 0x7f326c154cf0 2023-10-06T03:29:51.682 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.665+0000 7f325a7fc700 1 -- 172.21.15.110:0/2127891273 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3264000f40 con 0x7f326c154cf0 2023-10-06T03:29:51.683 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.665+0000 7f97ecff9700 1 -- 172.21.15.110:0/2483651091 <== mon.2 v2:172.21.15.192:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f97e4050830 con 0x7f97f0155140 2023-10-06T03:29:51.683 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.665+0000 7f91e089d700 1 -- 172.21.15.110:0/1543111619 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f91dc070ff0 con 0x7f91dc15d300 2023-10-06T03:29:51.683 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.665+0000 7f325a7fc700 1 -- 172.21.15.110:0/2127891273 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f326401f660 con 0x7f326c154cf0 2023-10-06T03:29:51.683 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.665+0000 7f326b59e700 1 -- 172.21.15.110:0/2127891273 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f326c158330 con 0x7f326c154cf0 2023-10-06T03:29:51.683 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.665+0000 7f326b59e700 1 -- 172.21.15.110:0/2127891273 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f326c156af0 con 0x7f326c154cf0 2023-10-06T03:29:51.684 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.669+0000 7f91c2ffd700 1 -- 172.21.15.110:0/1543111619 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f91d404ec70 con 0x7f91dc15d300 2023-10-06T03:29:51.685 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.673+0000 7f325a7fc700 1 -- 172.21.15.110:0/2127891273 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7f3264003040 con 0x7f326c154cf0 2023-10-06T03:29:51.685 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.673+0000 7f325a7fc700 1 --2- 172.21.15.110:0/2127891273 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f324c0606a0 0x7f324c062b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:51.685 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.673+0000 7f326a59c700 1 --2- 172.21.15.110:0/2127891273 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f324c0606a0 0x7f324c062b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:51.686 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.673+0000 7f326a59c700 1 --2- 172.21.15.110:0/2127891273 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f324c0606a0 0x7f324c062b50 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f3260006fd0 tx=0x7f3260008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:29:51.686 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.673+0000 7f325a7fc700 1 -- 172.21.15.110:0/2127891273 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6809+0+0 (secure 0 0 0) 0x7f326407df50 con 0x7f326c154cf0 2023-10-06T03:29:51.686 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.673+0000 7f326b59e700 1 -- 172.21.15.110:0/2127891273 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f326c070ff0 con 0x7f326c154cf0 2023-10-06T03:29:51.686 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.681+0000 7f325a7fc700 1 -- 172.21.15.110:0/2127891273 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f326404e0d0 con 0x7f326c154cf0 2023-10-06T03:29:51.877 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.873+0000 7f91d4ff9700 1 -- 172.21.15.110:0/1001659278 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f91f006c210 con 0x7f91d8008470 2023-10-06T03:29:51.878 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.873+0000 7f91d6ffd700 1 -- 172.21.15.110:0/1001659278 <== mon.0 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f91e0050660 con 0x7f91d8008470 2023-10-06T03:29:51.880 INFO:teuthology.orchestra.run.smithi110.stdout:60129542152 2023-10-06T03:29:51.883 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.877+0000 7f91f5a50700 1 -- 172.21.15.110:0/1001659278 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f91dc060620 msgr2=0x7f91dc062ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:51.883 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.877+0000 7f91f5a50700 1 --2- 172.21.15.110:0/1001659278 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f91dc060620 0x7f91dc062ad0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f91e400b490 tx=0x7f91e4013000 comp rx=0 tx=0).stop 2023-10-06T03:29:51.883 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.877+0000 7f91f5a50700 1 -- 172.21.15.110:0/1001659278 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f91d8008470 msgr2=0x7f91f01573d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:51.883 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.877+0000 7f91f5a50700 1 --2- 172.21.15.110:0/1001659278 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f91d8008470 0x7f91f01573d0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f91e000edd0 tx=0x7f91e0005840 comp rx=0 tx=0).stop 2023-10-06T03:29:51.883 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.877+0000 7f91f5a50700 1 -- 172.21.15.110:0/1001659278 shutdown_connections 2023-10-06T03:29:51.883 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.877+0000 7f91f5a50700 1 --2- 172.21.15.110:0/1001659278 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f91dc060620 0x7f91dc062ad0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:51.883 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.877+0000 7f91f5a50700 1 --2- 172.21.15.110:0/1001659278 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f91f00af1e0 0x7f91f00af680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:51.883 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.877+0000 7f91f5a50700 1 --2- 172.21.15.110:0/1001659278 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f91f00c3a00 0x7f91f00b1280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:51.884 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.877+0000 7f91f5a50700 1 --2- 172.21.15.110:0/1001659278 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f91d8008470 0x7f91f01573d0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:51.884 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.877+0000 7f91f5a50700 1 -- 172.21.15.110:0/1001659278 >> 172.21.15.110:0/1001659278 conn(0x7f91f00cc560 msgr2=0x7f91f00c5d70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:29:51.884 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.877+0000 7fa804e0a700 1 -- 172.21.15.110:0/2403884471 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7fa7e0005560 con 0x7fa800154cf0 2023-10-06T03:29:51.885 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.877+0000 7f91f5a50700 1 -- 172.21.15.110:0/1001659278 shutdown_connections 2023-10-06T03:29:51.885 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.877+0000 7faa2dcb7700 1 -- 172.21.15.110:0/3400627858 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7faa28070ff0 con 0x7faa280c7a70 2023-10-06T03:29:51.885 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.877+0000 7f91f5a50700 1 -- 172.21.15.110:0/1001659278 wait complete. 2023-10-06T03:29:51.886 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.881+0000 7faa0f7fe700 1 -- 172.21.15.110:0/3400627858 <== mon.0 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7faa1c017020 con 0x7faa280c7a70 2023-10-06T03:29:51.887 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.881+0000 7fa7e77fe700 1 -- 172.21.15.110:0/2403884471 <== mon.0 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fa7f4050b90 con 0x7fa800154cf0 2023-10-06T03:29:51.887 INFO:teuthology.orchestra.run.smithi110.stdout:64424509450 2023-10-06T03:29:51.887 INFO:teuthology.orchestra.run.smithi110.stdout:68719476746 2023-10-06T03:29:51.889 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.885+0000 7faa2dcb7700 1 -- 172.21.15.110:0/3400627858 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7faa100607c0 msgr2=0x7faa10062c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:51.889 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.885+0000 7faa2dcb7700 1 --2- 172.21.15.110:0/3400627858 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7faa100607c0 0x7faa10062c70 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7faa180099b0 tx=0x7faa18017000 comp rx=0 tx=0).stop 2023-10-06T03:29:51.890 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.885+0000 7faa2dcb7700 1 -- 172.21.15.110:0/3400627858 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7faa280c7a70 msgr2=0x7faa281500a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:51.890 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.885+0000 7faa2dcb7700 1 --2- 172.21.15.110:0/3400627858 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7faa280c7a70 0x7faa281500a0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7faa1c001eb0 tx=0x7faa1c007d40 comp rx=0 tx=0).stop 2023-10-06T03:29:51.890 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.885+0000 7faa2dcb7700 1 -- 172.21.15.110:0/3400627858 shutdown_connections 2023-10-06T03:29:51.890 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.885+0000 7faa2dcb7700 1 --2- 172.21.15.110:0/3400627858 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7faa100607c0 0x7faa10062c70 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:51.891 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.885+0000 7faa2dcb7700 1 --2- 172.21.15.110:0/3400627858 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7faa2815a250 0x7faa280c3900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:51.891 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.885+0000 7faa2dcb7700 1 --2- 172.21.15.110:0/3400627858 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7faa280c7a70 0x7faa281500a0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:51.891 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.885+0000 7faa2dcb7700 1 --2- 172.21.15.110:0/3400627858 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7faa280c5b60 0x7faa2814e1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:51.891 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.885+0000 7faa2dcb7700 1 -- 172.21.15.110:0/3400627858 >> 172.21.15.110:0/3400627858 conn(0x7faa280cc560 msgr2=0x7faa2806c860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:29:51.891 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.885+0000 7faa2dcb7700 1 -- 172.21.15.110:0/3400627858 shutdown_connections 2023-10-06T03:29:51.891 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.885+0000 7faa2dcb7700 1 -- 172.21.15.110:0/3400627858 wait complete. 2023-10-06T03:29:51.893 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.889+0000 7fa804e0a700 1 -- 172.21.15.110:0/2403884471 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fa7ec0606a0 msgr2=0x7fa7ec062b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:51.894 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.889+0000 7fa804e0a700 1 --2- 172.21.15.110:0/2403884471 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fa7ec0606a0 0x7fa7ec062b50 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fa7f0009db0 tx=0x7fa7f00091a0 comp rx=0 tx=0).stop 2023-10-06T03:29:51.894 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.889+0000 7fa804e0a700 1 -- 172.21.15.110:0/2403884471 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa800154cf0 msgr2=0x7fa80016d4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:51.894 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.889+0000 7fa804e0a700 1 --2- 172.21.15.110:0/2403884471 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa800154cf0 0x7fa80016d4d0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fa7f4006b00 tx=0x7fa7f400b430 comp rx=0 tx=0).stop 2023-10-06T03:29:51.894 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.889+0000 7fa804e0a700 1 -- 172.21.15.110:0/2403884471 shutdown_connections 2023-10-06T03:29:51.894 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.889+0000 7fa804e0a700 1 --2- 172.21.15.110:0/2403884471 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fa7ec0606a0 0x7fa7ec062b50 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:51.894 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.889+0000 7fa804e0a700 1 --2- 172.21.15.110:0/2403884471 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa8001550c0 0x7fa8000acc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:51.895 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.889+0000 7fa804e0a700 1 --2- 172.21.15.110:0/2403884471 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa800154cf0 0x7fa80016d4d0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:51.895 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.889+0000 7fa804e0a700 1 --2- 172.21.15.110:0/2403884471 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa80006c3c0 0x7fa800156d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:51.895 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.889+0000 7fa804e0a700 1 -- 172.21.15.110:0/2403884471 >> 172.21.15.110:0/2403884471 conn(0x7fa80006dce0 msgr2=0x7fa8000aaaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:29:51.895 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.889+0000 7fa804e0a700 1 -- 172.21.15.110:0/2403884471 shutdown_connections 2023-10-06T03:29:51.895 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.889+0000 7fa804e0a700 1 -- 172.21.15.110:0/2403884471 wait complete. 2023-10-06T03:29:51.898 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542152 got 60129542152 for osd.1 2023-10-06T03:29:51.898 DEBUG:teuthology.parallel:result is None 2023-10-06T03:29:51.906 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509450 got 64424509450 for osd.0 2023-10-06T03:29:51.907 DEBUG:teuthology.parallel:result is None 2023-10-06T03:29:51.909 INFO:tasks.ceph.ceph_manager.ceph:need seq 68719476746 got 68719476746 for osd.6 2023-10-06T03:29:51.909 DEBUG:teuthology.parallel:result is None 2023-10-06T03:29:51.924 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.921+0000 7ff5fca25700 1 -- 172.21.15.110:0/3039315338 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7ff5dc005830 con 0x7ff5f8155140 2023-10-06T03:29:51.924 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.921+0000 7ff5e67fc700 1 -- 172.21.15.110:0/3039315338 <== mon.0 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7ff5f0016020 con 0x7ff5f8155140 2023-10-06T03:29:51.925 INFO:teuthology.orchestra.run.smithi110.stdout:68719476746 2023-10-06T03:29:51.927 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.921+0000 7ff5fca25700 1 -- 172.21.15.110:0/3039315338 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7ff5d8060620 msgr2=0x7ff5d8062ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:51.928 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.921+0000 7ff5fca25700 1 --2- 172.21.15.110:0/3039315338 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7ff5d8060620 0x7ff5d8062ad0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7ff5e800aa50 tx=0x7ff5e801c000 comp rx=0 tx=0).stop 2023-10-06T03:29:51.928 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.921+0000 7ff5fca25700 1 -- 172.21.15.110:0/3039315338 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff5f8155140 msgr2=0x7ff5f81576d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:51.928 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.921+0000 7ff5fca25700 1 --2- 172.21.15.110:0/3039315338 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff5f8155140 0x7ff5f81576d0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7ff5f000cb60 tx=0x7ff5f0001770 comp rx=0 tx=0).stop 2023-10-06T03:29:51.928 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.921+0000 7ff5fca25700 1 -- 172.21.15.110:0/3039315338 shutdown_connections 2023-10-06T03:29:51.928 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.921+0000 7ff5fca25700 1 --2- 172.21.15.110:0/3039315338 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7ff5d8060620 0x7ff5d8062ad0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:51.929 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.921+0000 7ff5fca25700 1 --2- 172.21.15.110:0/3039315338 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff5f8161a10 0x7ff5f81556b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:51.929 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.921+0000 7ff5fca25700 1 --2- 172.21.15.110:0/3039315338 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff5f8155140 0x7ff5f81576d0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:51.929 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.921+0000 7ff5fca25700 1 --2- 172.21.15.110:0/3039315338 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff5f80c75e0 0x7ff5f8165280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:51.929 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.921+0000 7ff5fca25700 1 -- 172.21.15.110:0/3039315338 >> 172.21.15.110:0/3039315338 conn(0x7ff5f80cc560 msgr2=0x7ff5f806a5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:29:51.930 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.925+0000 7ff5fca25700 1 -- 172.21.15.110:0/3039315338 shutdown_connections 2023-10-06T03:29:51.930 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.925+0000 7ff5fca25700 1 -- 172.21.15.110:0/3039315338 wait complete. 2023-10-06T03:29:51.937 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.933+0000 7f91e089d700 1 -- 172.21.15.110:0/1543111619 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7f91dc0b0af0 con 0x7f91dc15d300 2023-10-06T03:29:51.937 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.933+0000 7f91c2ffd700 1 -- 172.21.15.110:0/1543111619 <== mon.0 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f91d4052290 con 0x7f91dc15d300 2023-10-06T03:29:51.937 INFO:teuthology.orchestra.run.smithi110.stdout:60129542153 2023-10-06T03:29:51.940 INFO:tasks.ceph.ceph_manager.ceph:need seq 68719476746 got 68719476746 for osd.5 2023-10-06T03:29:51.941 DEBUG:teuthology.parallel:result is None 2023-10-06T03:29:51.941 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.937+0000 7f91e089d700 1 -- 172.21.15.110:0/1543111619 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f91c8060620 msgr2=0x7f91c8062ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:51.941 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.937+0000 7f91e089d700 1 --2- 172.21.15.110:0/1543111619 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f91c8060620 0x7f91c8062ad0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f91cc003aa0 tx=0x7f91cc000bc0 comp rx=0 tx=0).stop 2023-10-06T03:29:51.941 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.937+0000 7f91e089d700 1 -- 172.21.15.110:0/1543111619 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f91dc15d300 msgr2=0x7f91dc0ae8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:51.941 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.937+0000 7f91e089d700 1 --2- 172.21.15.110:0/1543111619 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f91dc15d300 0x7f91dc0ae8e0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f91d400b6c0 tx=0x7f91d4018000 comp rx=0 tx=0).stop 2023-10-06T03:29:51.942 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.937+0000 7f91e089d700 1 -- 172.21.15.110:0/1543111619 shutdown_connections 2023-10-06T03:29:51.942 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.937+0000 7f91e089d700 1 --2- 172.21.15.110:0/1543111619 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f91c8060620 0x7f91c8062ad0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:51.942 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.937+0000 7f91e089d700 1 --2- 172.21.15.110:0/1543111619 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f91dc15d300 0x7f91dc0ae8e0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:51.942 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.937+0000 7f91e089d700 1 --2- 172.21.15.110:0/1543111619 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f91dc156980 0x7f91dc151f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:51.942 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.937+0000 7f91e089d700 1 --2- 172.21.15.110:0/1543111619 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f91dc0c8a30 0x7f91dc0b0e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:51.942 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.937+0000 7f91e089d700 1 -- 172.21.15.110:0/1543111619 >> 172.21.15.110:0/1543111619 conn(0x7f91dc06dce0 msgr2=0x7f91dc0cab30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:29:51.942 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.937+0000 7f91e089d700 1 -- 172.21.15.110:0/1543111619 shutdown_connections 2023-10-06T03:29:51.942 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.937+0000 7f91e089d700 1 -- 172.21.15.110:0/1543111619 wait complete. 2023-10-06T03:29:51.948 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542153 got 60129542153 for osd.3 2023-10-06T03:29:51.949 DEBUG:teuthology.parallel:result is None 2023-10-06T03:29:51.951 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.945+0000 7f326b59e700 1 -- 172.21.15.110:0/2127891273 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f326c157560 con 0x7f326c154cf0 2023-10-06T03:29:51.951 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.949+0000 7f325a7fc700 1 -- 172.21.15.110:0/2127891273 <== mon.0 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f3264024090 con 0x7f326c154cf0 2023-10-06T03:29:51.951 INFO:teuthology.orchestra.run.smithi110.stdout:60129542152 2023-10-06T03:29:51.953 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.949+0000 7f326b59e700 1 -- 172.21.15.110:0/2127891273 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f324c0606a0 msgr2=0x7f324c062b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:51.953 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.949+0000 7f326b59e700 1 --2- 172.21.15.110:0/2127891273 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f324c0606a0 0x7f324c062b50 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f3260006fd0 tx=0x7f3260008040 comp rx=0 tx=0).stop 2023-10-06T03:29:51.954 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.949+0000 7f326b59e700 1 -- 172.21.15.110:0/2127891273 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f326c154cf0 msgr2=0x7f326c1578b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:51.954 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.949+0000 7f326b59e700 1 --2- 172.21.15.110:0/2127891273 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f326c154cf0 0x7f326c1578b0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f326400d8c0 tx=0x7f326401b000 comp rx=0 tx=0).stop 2023-10-06T03:29:51.954 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.949+0000 7f326b59e700 1 -- 172.21.15.110:0/2127891273 shutdown_connections 2023-10-06T03:29:51.954 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.949+0000 7f326b59e700 1 --2- 172.21.15.110:0/2127891273 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f324c0606a0 0x7f324c062b50 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:51.954 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.949+0000 7f326b59e700 1 --2- 172.21.15.110:0/2127891273 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f326c1550c0 0x7f326c157df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:51.954 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.949+0000 7f326b59e700 1 --2- 172.21.15.110:0/2127891273 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f326c154cf0 0x7f326c1578b0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:51.954 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.949+0000 7f326b59e700 1 --2- 172.21.15.110:0/2127891273 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f326c06c3c0 0x7f326c165130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:51.954 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.949+0000 7f326b59e700 1 -- 172.21.15.110:0/2127891273 >> 172.21.15.110:0/2127891273 conn(0x7f326c06dce0 msgr2=0x7f326c0693a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:29:51.954 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.949+0000 7f326b59e700 1 -- 172.21.15.110:0/2127891273 shutdown_connections 2023-10-06T03:29:51.955 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.949+0000 7f326b59e700 1 -- 172.21.15.110:0/2127891273 wait complete. 2023-10-06T03:29:51.955 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.949+0000 7efed52ba700 1 -- 172.21.15.110:0/3342450740 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7efed017e820 con 0x7efed00694a0 2023-10-06T03:29:51.955 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.949+0000 7efebffff700 1 -- 172.21.15.110:0/3342450740 <== mon.0 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7efec0017020 con 0x7efed00694a0 2023-10-06T03:29:51.955 INFO:teuthology.orchestra.run.smithi110.stdout:64424509449 2023-10-06T03:29:51.957 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.953+0000 7efed52ba700 1 -- 172.21.15.110:0/3342450740 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7efeb4060520 msgr2=0x7efeb40629d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:51.957 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.953+0000 7efed52ba700 1 --2- 172.21.15.110:0/3342450740 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7efeb4060520 0x7efeb40629d0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7efed00cb130 tx=0x7efec40091a0 comp rx=0 tx=0).stop 2023-10-06T03:29:51.957 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.953+0000 7efed52ba700 1 -- 172.21.15.110:0/3342450740 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7efed00694a0 msgr2=0x7efed00afcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:51.958 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.953+0000 7efed52ba700 1 --2- 172.21.15.110:0/3342450740 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7efed00694a0 0x7efed00afcb0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7efec000c790 tx=0x7efec0007ef0 comp rx=0 tx=0).stop 2023-10-06T03:29:51.958 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.953+0000 7efed52ba700 1 -- 172.21.15.110:0/3342450740 shutdown_connections 2023-10-06T03:29:51.958 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.953+0000 7efed52ba700 1 --2- 172.21.15.110:0/3342450740 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7efeb4060520 0x7efeb40629d0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:51.958 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.953+0000 7efed52ba700 1 --2- 172.21.15.110:0/3342450740 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efed01677b0 0x7efed00c47a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:51.958 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.953+0000 7efed52ba700 1 --2- 172.21.15.110:0/3342450740 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7efed00aac30 0x7efed00c4260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:51.958 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.953+0000 7efed52ba700 1 --2- 172.21.15.110:0/3342450740 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7efed00694a0 0x7efed00afcb0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:51.958 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.953+0000 7efed52ba700 1 -- 172.21.15.110:0/3342450740 >> 172.21.15.110:0/3342450740 conn(0x7efed006dce0 msgr2=0x7efed00ca180 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:29:51.958 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.953+0000 7efed52ba700 1 -- 172.21.15.110:0/3342450740 shutdown_connections 2023-10-06T03:29:51.958 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.953+0000 7efed52ba700 1 -- 172.21.15.110:0/3342450740 wait complete. 2023-10-06T03:29:51.961 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.957+0000 7f97f7058700 1 -- 172.21.15.110:0/2483651091 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7f97d0005560 con 0x7f97f0155140 2023-10-06T03:29:51.962 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.957+0000 7f97ecff9700 1 -- 172.21.15.110:0/2483651091 <== mon.2 v2:172.21.15.192:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f97e40503c0 con 0x7f97f0155140 2023-10-06T03:29:51.962 INFO:teuthology.orchestra.run.smithi110.stdout:68719476745 2023-10-06T03:29:51.965 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542152 got 60129542152 for osd.2 2023-10-06T03:29:51.965 DEBUG:teuthology.parallel:result is None 2023-10-06T03:29:51.966 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.961+0000 7f97f7058700 1 -- 172.21.15.110:0/2483651091 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f97dc0a45e0 msgr2=0x7f97dc0a6a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:51.966 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.961+0000 7f97f7058700 1 --2- 172.21.15.110:0/2483651091 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f97dc0a45e0 0x7f97dc0a6a90 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f97f006ee50 tx=0x7f97e0019000 comp rx=0 tx=0).stop 2023-10-06T03:29:51.966 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.961+0000 7f97f7058700 1 -- 172.21.15.110:0/2483651091 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f97f0155140 msgr2=0x7f97f00c3aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:51.966 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.961+0000 7f97f7058700 1 --2- 172.21.15.110:0/2483651091 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f97f0155140 0x7f97f00c3aa0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f97e400b810 tx=0x7f97e4009250 comp rx=0 tx=0).stop 2023-10-06T03:29:51.966 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.961+0000 7f97f7058700 1 -- 172.21.15.110:0/2483651091 shutdown_connections 2023-10-06T03:29:51.966 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.961+0000 7f97f7058700 1 --2- 172.21.15.110:0/2483651091 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f97dc0a45e0 0x7f97dc0a6a90 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:51.966 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.961+0000 7f97f7058700 1 --2- 172.21.15.110:0/2483651091 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f97f00b10a0 0x7f97f00af650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:51.967 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.961+0000 7f97f7058700 1 --2- 172.21.15.110:0/2483651091 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f97f0155140 0x7f97f00c3aa0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:51.967 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.961+0000 7f97f7058700 1 --2- 172.21.15.110:0/2483651091 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f97f00c20d0 0x7f97f00b0b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:51.967 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.961+0000 7f97f7058700 1 -- 172.21.15.110:0/2483651091 >> 172.21.15.110:0/2483651091 conn(0x7f97f00cc560 msgr2=0x7f97f015f280 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:29:51.967 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.961+0000 7f97f7058700 1 -- 172.21.15.110:0/2483651091 shutdown_connections 2023-10-06T03:29:51.967 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:51.961+0000 7f97f7058700 1 -- 172.21.15.110:0/2483651091 wait complete. 2023-10-06T03:29:51.970 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509449 got 64424509449 for osd.7 2023-10-06T03:29:51.970 DEBUG:teuthology.parallel:result is None 2023-10-06T03:29:51.975 INFO:tasks.ceph.ceph_manager.ceph:need seq 68719476745 got 68719476745 for osd.4 2023-10-06T03:29:51.975 DEBUG:teuthology.parallel:result is None 2023-10-06T03:29:51.975 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2023-10-06T03:29:51.975 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-06T03:29:52.071 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:52.069+0000 7f0dd9832700 1 Processor -- start 2023-10-06T03:29:52.072 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:52.069+0000 7f0dd9832700 1 -- start start 2023-10-06T03:29:52.075 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:52.069+0000 7f0dd9832700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f0dd4156b80 0x7f0dd40c68d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:52.075 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:52.069+0000 7f0dd9832700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0dd4156f50 0x7f0dd40b0e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:52.075 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:52.069+0000 7f0dd9832700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0dd40c7f30 con 0x7f0dd40c6e10 2023-10-06T03:29:52.075 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:52.069+0000 7f0dd9832700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0dd40c7320 con 0x7f0dd4156b80 2023-10-06T03:29:52.075 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:52.069+0000 7f0dd9832700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0dd40c7600 con 0x7f0dd4156f50 2023-10-06T03:29:52.076 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:52.069+0000 7f0dd9031700 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f0dd40c6e10 0x7f0dd40ad560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:40126/0 (socket says 172.21.15.110:40126) 2023-10-06T03:29:52.076 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:52.069+0000 7f0dd9031700 1 -- 172.21.15.110:0/165700795 learned_addr learned my addr 172.21.15.110:0/165700795 (peer_addr_for_me v1:172.21.15.110:0/0) 2023-10-06T03:29:52.076 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:52.069+0000 7f0dd8830700 1 --2- 172.21.15.110:0/165700795 >> v2:172.21.15.110:3300/0 conn(0x7f0dd4156b80 0x7f0dd40c68d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:52.076 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:52.069+0000 7f0dd3fff700 1 --2- 172.21.15.110:0/165700795 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0dd4156f50 0x7f0dd40b0e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:52.076 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:52.069+0000 7f0dd37fe700 1 -- 172.21.15.110:0/165700795 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2589860624 0 0) 0x7f0dd40c7f30 con 0x7f0dd40c6e10 2023-10-06T03:29:52.076 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:52.069+0000 7f0dd37fe700 1 -- 172.21.15.110:0/165700795 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0dc4003570 con 0x7f0dd40c6e10 2023-10-06T03:29:52.076 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:52.069+0000 7f0dd3fff700 1 -- 172.21.15.110:0/165700795 >> v2:172.21.15.110:3300/0 conn(0x7f0dd4156b80 msgr2=0x7f0dd40c68d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:29:52.076 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:52.069+0000 7f0dd3fff700 1 --2- 172.21.15.110:0/165700795 >> v2:172.21.15.110:3300/0 conn(0x7f0dd4156b80 0x7f0dd40c68d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:52.076 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:52.069+0000 7f0dd3fff700 1 -- 172.21.15.110:0/165700795 >> v1:172.21.15.110:6789/0 conn(0x7f0dd40c6e10 legacy=0x7f0dd40ad560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:52.077 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:52.069+0000 7f0dd3fff700 1 -- 172.21.15.110:0/165700795 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0dd40c78e0 con 0x7f0dd4156f50 2023-10-06T03:29:52.077 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:52.069+0000 7f0dd3fff700 1 --2- 172.21.15.110:0/165700795 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0dd4156f50 0x7f0dd40b0e40 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f0dc00098e0 tx=0x7f0dc00091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=84245d9ace6b713f server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:29:52.077 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:52.069+0000 7f0dd37fe700 1 -- 172.21.15.110:0/165700795 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0dc0011630 con 0x7f0dd4156f50 2023-10-06T03:29:52.077 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:52.069+0000 7f0dd37fe700 1 -- 172.21.15.110:0/165700795 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0dc0011790 con 0x7f0dd4156f50 2023-10-06T03:29:52.077 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:52.069+0000 7f0dd37fe700 1 -- 172.21.15.110:0/165700795 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0dc0011b30 con 0x7f0dd4156f50 2023-10-06T03:29:52.077 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:52.069+0000 7f0dd9832700 1 -- 172.21.15.110:0/165700795 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0dd4156f50 msgr2=0x7f0dd40b0e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:52.077 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:52.069+0000 7f0dd9832700 1 --2- 172.21.15.110:0/165700795 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0dd4156f50 0x7f0dd40b0e40 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f0dc00098e0 tx=0x7f0dc00091a0 comp rx=0 tx=0).stop 2023-10-06T03:29:52.077 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:52.069+0000 7f0dd9832700 1 -- 172.21.15.110:0/165700795 shutdown_connections 2023-10-06T03:29:52.077 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:52.069+0000 7f0dd9832700 1 --2- 172.21.15.110:0/165700795 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0dd4156f50 0x7f0dd40b0e40 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:52.077 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:52.069+0000 7f0dd9832700 1 --2- 172.21.15.110:0/165700795 >> v2:172.21.15.110:3300/0 conn(0x7f0dd4156b80 0x7f0dd40c68d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:52.078 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:52.069+0000 7f0dd9832700 1 -- 172.21.15.110:0/165700795 >> 172.21.15.110:0/165700795 conn(0x7f0dd40cc470 msgr2=0x7f0dd40c8410 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:29:52.078 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:52.069+0000 7f0dd9832700 1 -- 172.21.15.110:0/165700795 shutdown_connections 2023-10-06T03:29:52.078 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:52.069+0000 7f0dd9832700 1 -- 172.21.15.110:0/165700795 wait complete. 2023-10-06T03:29:52.078 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:52.073+0000 7f0dd9832700 1 Processor -- start 2023-10-06T03:29:52.078 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:52.073+0000 7f0dd9832700 1 -- start start 2023-10-06T03:29:52.080 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:52.073+0000 7f0dd9832700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0dd40c6e10 0x7f0dd40c9e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:52.080 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:52.073+0000 7f0dd9832700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0dd4156b80 0x7f0dd4149640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:52.081 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:52.073+0000 7f0dd9832700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0dd4156f50 0x7f0dd40c18e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:52.081 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:52.073+0000 7f0dd9832700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0dd40a5d10 con 0x7f0dd4156f50 2023-10-06T03:29:52.081 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:52.073+0000 7f0dd9832700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0dd40a6800 con 0x7f0dd40c6e10 2023-10-06T03:29:52.081 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:52.073+0000 7f0dd9832700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0dd40a4690 con 0x7f0dd4156b80 2023-10-06T03:29:52.081 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:52.073+0000 7f0dd9031700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0dd4156f50 0x7f0dd40c18e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:52.081 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:52.073+0000 7f0dd3fff700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0dd4156b80 0x7f0dd4149640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:52.081 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:52.073+0000 7f0dd9031700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0dd4156f50 0x7f0dd40c18e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:47642/0 (socket says 172.21.15.110:47642) 2023-10-06T03:29:52.081 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:52.073+0000 7f0dd8830700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0dd40c6e10 0x7f0dd40c9e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:52.081 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:52.073+0000 7f0dd9031700 1 -- 172.21.15.110:0/1621762971 learned_addr learned my addr 172.21.15.110:0/1621762971 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:29:52.081 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:52.073+0000 7f0dd9031700 1 -- 172.21.15.110:0/1621762971 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0dd4156b80 msgr2=0x7f0dd4149640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:52.082 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:52.073+0000 7f0dd9031700 1 --2- 172.21.15.110:0/1621762971 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0dd4156b80 0x7f0dd4149640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:52.082 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:52.073+0000 7f0dd9031700 1 -- 172.21.15.110:0/1621762971 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0dd40c6e10 msgr2=0x7f0dd40c9e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:52.082 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:52.073+0000 7f0dd9031700 1 --2- 172.21.15.110:0/1621762971 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0dd40c6e10 0x7f0dd40c9e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:52.082 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:52.073+0000 7f0dd9031700 1 -- 172.21.15.110:0/1621762971 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0dc0009420 con 0x7f0dd4156f50 2023-10-06T03:29:52.082 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:52.073+0000 7f0dd3fff700 1 --2- 172.21.15.110:0/1621762971 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0dd4156b80 0x7f0dd4149640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-06T03:29:52.082 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:52.073+0000 7f0dd9031700 1 --2- 172.21.15.110:0/1621762971 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0dd4156f50 0x7f0dd40c18e0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f0dcc00cca0 tx=0x7f0dcc001aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:29:52.082 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:52.073+0000 7f0dd0ff9700 1 -- 172.21.15.110:0/1621762971 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0dcc00a890 con 0x7f0dd4156f50 2023-10-06T03:29:52.082 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:52.073+0000 7f0dd0ff9700 1 -- 172.21.15.110:0/1621762971 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0dcc00a9f0 con 0x7f0dd4156f50 2023-10-06T03:29:52.082 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:52.073+0000 7f0dd9832700 1 -- 172.21.15.110:0/1621762971 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0dd40a5f60 con 0x7f0dd4156f50 2023-10-06T03:29:52.082 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:52.073+0000 7f0dd0ff9700 1 -- 172.21.15.110:0/1621762971 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0dcc00ad70 con 0x7f0dd4156f50 2023-10-06T03:29:52.083 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:52.073+0000 7f0dd9832700 1 -- 172.21.15.110:0/1621762971 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0dd40a6520 con 0x7f0dd4156f50 2023-10-06T03:29:52.083 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:52.077+0000 7f0dd9832700 1 -- 172.21.15.110:0/1621762971 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0dd4070ff0 con 0x7f0dd4156f50 2023-10-06T03:29:52.083 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:52.077+0000 7f0dd0ff9700 1 -- 172.21.15.110:0/1621762971 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7f0dcc007300 con 0x7f0dd4156f50 2023-10-06T03:29:52.084 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:52.077+0000 7f0dd0ff9700 1 --2- 172.21.15.110:0/1621762971 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f0db8060550 0x7f0db8062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:52.084 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:52.077+0000 7f0dd0ff9700 1 -- 172.21.15.110:0/1621762971 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6809+0+0 (secure 0 0 0) 0x7f0dcc07cf40 con 0x7f0dd4156f50 2023-10-06T03:29:52.084 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:52.077+0000 7f0dd8830700 1 --2- 172.21.15.110:0/1621762971 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f0db8060550 0x7f0db8062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:52.084 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:52.077+0000 7f0dd8830700 1 --2- 172.21.15.110:0/1621762971 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f0db8060550 0x7f0db8062a00 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f0dc8009db0 tx=0x7f0dc80091a0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:29:52.088 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:52.085+0000 7f0dd0ff9700 1 -- 172.21.15.110:0/1621762971 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f0dcc04d160 con 0x7f0dd4156f50 2023-10-06T03:29:52.234 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:52.229+0000 7f0dd9832700 1 -- 172.21.15.110:0/1621762971 --> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f0dd40a47d0 con 0x7f0db8060550 2023-10-06T03:29:52.249 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:52.245+0000 7f0dd0ff9700 1 -- 172.21.15.110:0/1621762971 <== mgr.4110 v2:172.21.15.110:6832/33268 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+367265 (secure 0 0 0) 0x7f0dd40a47d0 con 0x7f0db8060550 2023-10-06T03:29:52.253 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-10-06T03:29:52.256 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:52.249+0000 7f0dd9832700 1 -- 172.21.15.110:0/1621762971 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f0db8060550 msgr2=0x7f0db8062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:52.256 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:52.249+0000 7f0dd9832700 1 --2- 172.21.15.110:0/1621762971 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f0db8060550 0x7f0db8062a00 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f0dc8009db0 tx=0x7f0dc80091a0 comp rx=0 tx=0).stop 2023-10-06T03:29:52.256 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:52.249+0000 7f0dd9832700 1 -- 172.21.15.110:0/1621762971 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0dd4156f50 msgr2=0x7f0dd40c18e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:52.256 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:52.249+0000 7f0dd9832700 1 --2- 172.21.15.110:0/1621762971 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0dd4156f50 0x7f0dd40c18e0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f0dcc00cca0 tx=0x7f0dcc001aa0 comp rx=0 tx=0).stop 2023-10-06T03:29:52.256 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:52.249+0000 7f0dd9832700 1 -- 172.21.15.110:0/1621762971 shutdown_connections 2023-10-06T03:29:52.257 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:52.249+0000 7f0dd9832700 1 --2- 172.21.15.110:0/1621762971 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f0db8060550 0x7f0db8062a00 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:52.257 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:52.249+0000 7f0dd9832700 1 --2- 172.21.15.110:0/1621762971 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0dd4156f50 0x7f0dd40c18e0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:52.257 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:52.249+0000 7f0dd9832700 1 --2- 172.21.15.110:0/1621762971 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0dd4156b80 0x7f0dd4149640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:52.257 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:52.249+0000 7f0dd9832700 1 --2- 172.21.15.110:0/1621762971 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0dd40c6e10 0x7f0dd40c9e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:52.257 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:52.249+0000 7f0dd9832700 1 -- 172.21.15.110:0/1621762971 >> 172.21.15.110:0/1621762971 conn(0x7f0dd40cc470 msgr2=0x7f0dd4155ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:29:52.257 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:52.249+0000 7f0dd9832700 1 -- 172.21.15.110:0/1621762971 shutdown_connections 2023-10-06T03:29:52.257 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:52.249+0000 7f0dd9832700 1 -- 172.21.15.110:0/1621762971 wait complete. 2023-10-06T03:29:52.257 INFO:teuthology.orchestra.run.smithi110.stderr:dumped all 2023-10-06T03:29:52.266 INFO:teuthology.orchestra.run.smithi110.stdout:{"pg_ready":true,"pg_map":{"version":36,"stamp":"2023-10-06T03:29:50.653327+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":594852,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":207,"num_write_kb":2433,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":159,"ondisk_log_size":159,"up":338,"acting":338,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":338,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2398184,"kb_used_data":3496,"kb_used_omap":0,"kb_used_meta":2394624,"kb_avail":752576536,"statfs":{"total":773094113280,"available":770638372864,"internally_reserved":0,"allocated":3579904,"data_stored":2149912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2452094976},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2178,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"11.037086"},"pg_stats":[{"pgid":"5.19","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.624659+0000","last_change":"2023-10-06T03:29:38.620919+0000","last_active":"2023-10-06T03:29:38.624659+0000","last_peered":"2023-10-06T03:29:38.624659+0000","last_clean":"2023-10-06T03:29:38.624659+0000","last_became_active":"2023-10-06T03:29:38.620293+0000","last_became_peered":"2023-10-06T03:29:38.620293+0000","last_unstale":"2023-10-06T03:29:38.624659+0000","last_undegraded":"2023-10-06T03:29:38.624659+0000","last_fullsized":"2023-10-06T03:29:38.624659+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:37.570693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:37.570693+0000","last_clean_scrub_stamp":"2023-10-06T03:29:37.570693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T13:27:03.828118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7,2],"acting":[1,5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"23'4","reported_seq":18,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.592373+0000","last_change":"2023-10-06T03:29:33.535641+0000","last_active":"2023-10-06T03:29:38.592373+0000","last_peered":"2023-10-06T03:29:38.592373+0000","last_clean":"2023-10-06T03:29:38.592373+0000","last_became_active":"2023-10-06T03:29:33.534410+0000","last_became_peered":"2023-10-06T03:29:33.534410+0000","last_unstale":"2023-10-06T03:29:38.592373+0000","last_undegraded":"2023-10-06T03:29:38.592373+0000","last_fullsized":"2023-10-06T03:29:38.592373+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:32.495616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:32.495616+0000","last_clean_scrub_stamp":"2023-10-06T03:29:32.495616+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T14:51:49.889964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":554,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.594781+0000","last_change":"2023-10-06T03:29:34.546243+0000","last_active":"2023-10-06T03:29:38.594781+0000","last_peered":"2023-10-06T03:29:38.594781+0000","last_clean":"2023-10-06T03:29:38.594781+0000","last_became_active":"2023-10-06T03:29:34.545923+0000","last_became_peered":"2023-10-06T03:29:34.545923+0000","last_unstale":"2023-10-06T03:29:38.594781+0000","last_undegraded":"2023-10-06T03:29:38.594781+0000","last_fullsized":"2023-10-06T03:29:38.594781+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_clean_scrub_stamp":"2023-10-06T03:29:33.500929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T15:20:45.655411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.617434+0000","last_change":"2023-10-06T03:29:38.608716+0000","last_active":"2023-10-06T03:29:38.617434+0000","last_peered":"2023-10-06T03:29:38.617434+0000","last_clean":"2023-10-06T03:29:38.617434+0000","last_became_active":"2023-10-06T03:29:38.607714+0000","last_became_peered":"2023-10-06T03:29:38.607714+0000","last_unstale":"2023-10-06T03:29:38.617434+0000","last_undegraded":"2023-10-06T03:29:38.617434+0000","last_fullsized":"2023-10-06T03:29:38.617434+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:37.570693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:37.570693+0000","last_clean_scrub_stamp":"2023-10-06T03:29:37.570693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T07:11:39.919440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1,2],"acting":[4,6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"23'1","reported_seq":15,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.591195+0000","last_change":"2023-10-06T03:29:33.538145+0000","last_active":"2023-10-06T03:29:38.591195+0000","last_peered":"2023-10-06T03:29:38.591195+0000","last_clean":"2023-10-06T03:29:38.591195+0000","last_became_active":"2023-10-06T03:29:33.536028+0000","last_became_peered":"2023-10-06T03:29:33.536028+0000","last_unstale":"2023-10-06T03:29:38.591195+0000","last_undegraded":"2023-10-06T03:29:38.591195+0000","last_fullsized":"2023-10-06T03:29:38.591195+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:32.495616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:32.495616+0000","last_clean_scrub_stamp":"2023-10-06T03:29:32.495616+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T06:15:00.105181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.589205+0000","last_change":"2023-10-06T03:29:34.546777+0000","last_active":"2023-10-06T03:29:38.589205+0000","last_peered":"2023-10-06T03:29:38.589205+0000","last_clean":"2023-10-06T03:29:38.589205+0000","last_became_active":"2023-10-06T03:29:34.545461+0000","last_became_peered":"2023-10-06T03:29:34.545461+0000","last_unstale":"2023-10-06T03:29:38.589205+0000","last_undegraded":"2023-10-06T03:29:38.589205+0000","last_fullsized":"2023-10-06T03:29:38.589205+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_clean_scrub_stamp":"2023-10-06T03:29:33.500929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T06:40:43.933850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.621795+0000","last_change":"2023-10-06T03:29:38.619453+0000","last_active":"2023-10-06T03:29:38.621795+0000","last_peered":"2023-10-06T03:29:38.621795+0000","last_clean":"2023-10-06T03:29:38.621795+0000","last_became_active":"2023-10-06T03:29:38.618690+0000","last_became_peered":"2023-10-06T03:29:38.618690+0000","last_unstale":"2023-10-06T03:29:38.621795+0000","last_undegraded":"2023-10-06T03:29:38.621795+0000","last_fullsized":"2023-10-06T03:29:38.621795+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:37.570693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:37.570693+0000","last_clean_scrub_stamp":"2023-10-06T03:29:37.570693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T06:00:28.537215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7,4],"acting":[5,0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"23'2","reported_seq":16,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.602095+0000","last_change":"2023-10-06T03:29:33.542309+0000","last_active":"2023-10-06T03:29:38.602095+0000","last_peered":"2023-10-06T03:29:38.602095+0000","last_clean":"2023-10-06T03:29:38.602095+0000","last_became_active":"2023-10-06T03:29:33.542038+0000","last_became_peered":"2023-10-06T03:29:33.542038+0000","last_unstale":"2023-10-06T03:29:38.602095+0000","last_undegraded":"2023-10-06T03:29:38.602095+0000","last_fullsized":"2023-10-06T03:29:38.602095+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:32.495616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:32.495616+0000","last_clean_scrub_stamp":"2023-10-06T03:29:32.495616+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T08:04:27.115289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.592351+0000","last_change":"2023-10-06T03:29:34.540591+0000","last_active":"2023-10-06T03:29:38.592351+0000","last_peered":"2023-10-06T03:29:38.592351+0000","last_clean":"2023-10-06T03:29:38.592351+0000","last_became_active":"2023-10-06T03:29:34.540219+0000","last_became_peered":"2023-10-06T03:29:34.540219+0000","last_unstale":"2023-10-06T03:29:38.592351+0000","last_undegraded":"2023-10-06T03:29:38.592351+0000","last_fullsized":"2023-10-06T03:29:38.592351+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_clean_scrub_stamp":"2023-10-06T03:29:33.500929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T11:28:48.945575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.626642+0000","last_change":"2023-10-06T03:29:38.624230+0000","last_active":"2023-10-06T03:29:38.626642+0000","last_peered":"2023-10-06T03:29:38.626642+0000","last_clean":"2023-10-06T03:29:38.626642+0000","last_became_active":"2023-10-06T03:29:38.623638+0000","last_became_peered":"2023-10-06T03:29:38.623638+0000","last_unstale":"2023-10-06T03:29:38.626642+0000","last_undegraded":"2023-10-06T03:29:38.626642+0000","last_fullsized":"2023-10-06T03:29:38.626642+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:37.570693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:37.570693+0000","last_clean_scrub_stamp":"2023-10-06T03:29:37.570693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T12:11:50.851770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1,6],"acting":[7,4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"23'2","reported_seq":16,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.600846+0000","last_change":"2023-10-06T03:29:33.540179+0000","last_active":"2023-10-06T03:29:38.600846+0000","last_peered":"2023-10-06T03:29:38.600846+0000","last_clean":"2023-10-06T03:29:38.600846+0000","last_became_active":"2023-10-06T03:29:33.539825+0000","last_became_peered":"2023-10-06T03:29:33.539825+0000","last_unstale":"2023-10-06T03:29:38.600846+0000","last_undegraded":"2023-10-06T03:29:38.600846+0000","last_fullsized":"2023-10-06T03:29:38.600846+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:32.495616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:32.495616+0000","last_clean_scrub_stamp":"2023-10-06T03:29:32.495616+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T11:58:53.137774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.593620+0000","last_change":"2023-10-06T03:29:34.540676+0000","last_active":"2023-10-06T03:29:38.593620+0000","last_peered":"2023-10-06T03:29:38.593620+0000","last_clean":"2023-10-06T03:29:38.593620+0000","last_became_active":"2023-10-06T03:29:34.540358+0000","last_became_peered":"2023-10-06T03:29:34.540358+0000","last_unstale":"2023-10-06T03:29:38.593620+0000","last_undegraded":"2023-10-06T03:29:38.593620+0000","last_fullsized":"2023-10-06T03:29:38.593620+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_clean_scrub_stamp":"2023-10-06T03:29:33.500929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T07:51:21.462969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.624627+0000","last_change":"2023-10-06T03:29:38.620564+0000","last_active":"2023-10-06T03:29:38.624627+0000","last_peered":"2023-10-06T03:29:38.624627+0000","last_clean":"2023-10-06T03:29:38.624627+0000","last_became_active":"2023-10-06T03:29:38.620134+0000","last_became_peered":"2023-10-06T03:29:38.620134+0000","last_unstale":"2023-10-06T03:29:38.624627+0000","last_undegraded":"2023-10-06T03:29:38.624627+0000","last_fullsized":"2023-10-06T03:29:38.624627+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:37.570693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:37.570693+0000","last_clean_scrub_stamp":"2023-10-06T03:29:37.570693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T04:19:16.510300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4,0],"acting":[1,3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"23'1","reported_seq":15,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.593175+0000","last_change":"2023-10-06T03:29:33.542267+0000","last_active":"2023-10-06T03:29:38.593175+0000","last_peered":"2023-10-06T03:29:38.593175+0000","last_clean":"2023-10-06T03:29:38.593175+0000","last_became_active":"2023-10-06T03:29:33.541987+0000","last_became_peered":"2023-10-06T03:29:33.541987+0000","last_unstale":"2023-10-06T03:29:38.593175+0000","last_undegraded":"2023-10-06T03:29:38.593175+0000","last_fullsized":"2023-10-06T03:29:38.593175+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:32.495616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:32.495616+0000","last_clean_scrub_stamp":"2023-10-06T03:29:32.495616+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T12:25:10.766051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.600288+0000","last_change":"2023-10-06T03:29:34.547283+0000","last_active":"2023-10-06T03:29:38.600288+0000","last_peered":"2023-10-06T03:29:38.600288+0000","last_clean":"2023-10-06T03:29:38.600288+0000","last_became_active":"2023-10-06T03:29:34.546966+0000","last_became_peered":"2023-10-06T03:29:34.546966+0000","last_unstale":"2023-10-06T03:29:38.600288+0000","last_undegraded":"2023-10-06T03:29:38.600288+0000","last_fullsized":"2023-10-06T03:29:38.600288+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_clean_scrub_stamp":"2023-10-06T03:29:33.500929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T07:55:08.792414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.621924+0000","last_change":"2023-10-06T03:29:38.614364+0000","last_active":"2023-10-06T03:29:38.621924+0000","last_peered":"2023-10-06T03:29:38.621924+0000","last_clean":"2023-10-06T03:29:38.621924+0000","last_became_active":"2023-10-06T03:29:38.613892+0000","last_became_peered":"2023-10-06T03:29:38.613892+0000","last_unstale":"2023-10-06T03:29:38.621924+0000","last_undegraded":"2023-10-06T03:29:38.621924+0000","last_fullsized":"2023-10-06T03:29:38.621924+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:37.570693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:37.570693+0000","last_clean_scrub_stamp":"2023-10-06T03:29:37.570693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T13:00:21.693396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2,0],"acting":[4,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"23'1","reported_seq":15,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.592761+0000","last_change":"2023-10-06T03:29:33.546282+0000","last_active":"2023-10-06T03:29:38.592761+0000","last_peered":"2023-10-06T03:29:38.592761+0000","last_clean":"2023-10-06T03:29:38.592761+0000","last_became_active":"2023-10-06T03:29:33.546039+0000","last_became_peered":"2023-10-06T03:29:33.546039+0000","last_unstale":"2023-10-06T03:29:38.592761+0000","last_undegraded":"2023-10-06T03:29:38.592761+0000","last_fullsized":"2023-10-06T03:29:38.592761+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:32.495616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:32.495616+0000","last_clean_scrub_stamp":"2023-10-06T03:29:32.495616+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T12:01:27.570873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.589771+0000","last_change":"2023-10-06T03:29:34.550953+0000","last_active":"2023-10-06T03:29:38.589771+0000","last_peered":"2023-10-06T03:29:38.589771+0000","last_clean":"2023-10-06T03:29:38.589771+0000","last_became_active":"2023-10-06T03:29:34.550614+0000","last_became_peered":"2023-10-06T03:29:34.550614+0000","last_unstale":"2023-10-06T03:29:38.589771+0000","last_undegraded":"2023-10-06T03:29:38.589771+0000","last_fullsized":"2023-10-06T03:29:38.589771+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_clean_scrub_stamp":"2023-10-06T03:29:33.500929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T11:59:02.611216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.620922+0000","last_change":"2023-10-06T03:29:38.613494+0000","last_active":"2023-10-06T03:29:38.620922+0000","last_peered":"2023-10-06T03:29:38.620922+0000","last_clean":"2023-10-06T03:29:38.620922+0000","last_became_active":"2023-10-06T03:29:38.613121+0000","last_became_peered":"2023-10-06T03:29:38.613121+0000","last_unstale":"2023-10-06T03:29:38.620922+0000","last_undegraded":"2023-10-06T03:29:38.620922+0000","last_fullsized":"2023-10-06T03:29:38.620922+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:37.570693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:37.570693+0000","last_clean_scrub_stamp":"2023-10-06T03:29:37.570693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T03:45:24.009354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3,7],"acting":[6,4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.608215+0000","last_change":"2023-10-06T03:29:33.542907+0000","last_active":"2023-10-06T03:29:38.608215+0000","last_peered":"2023-10-06T03:29:38.608215+0000","last_clean":"2023-10-06T03:29:38.608215+0000","last_became_active":"2023-10-06T03:29:33.542556+0000","last_became_peered":"2023-10-06T03:29:33.542556+0000","last_unstale":"2023-10-06T03:29:38.608215+0000","last_undegraded":"2023-10-06T03:29:38.608215+0000","last_fullsized":"2023-10-06T03:29:38.608215+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:32.495616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:32.495616+0000","last_clean_scrub_stamp":"2023-10-06T03:29:32.495616+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T03:54:50.215488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.599642+0000","last_change":"2023-10-06T03:29:34.541680+0000","last_active":"2023-10-06T03:29:38.599642+0000","last_peered":"2023-10-06T03:29:38.599642+0000","last_clean":"2023-10-06T03:29:38.599642+0000","last_became_active":"2023-10-06T03:29:34.541220+0000","last_became_peered":"2023-10-06T03:29:34.541220+0000","last_unstale":"2023-10-06T03:29:38.599642+0000","last_undegraded":"2023-10-06T03:29:38.599642+0000","last_fullsized":"2023-10-06T03:29:38.599642+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_clean_scrub_stamp":"2023-10-06T03:29:33.500929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T06:09:15.729486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.627293+0000","last_change":"2023-10-06T03:29:38.623302+0000","last_active":"2023-10-06T03:29:38.627293+0000","last_peered":"2023-10-06T03:29:38.627293+0000","last_clean":"2023-10-06T03:29:38.627293+0000","last_became_active":"2023-10-06T03:29:38.622782+0000","last_became_peered":"2023-10-06T03:29:38.622782+0000","last_unstale":"2023-10-06T03:29:38.627293+0000","last_undegraded":"2023-10-06T03:29:38.627293+0000","last_fullsized":"2023-10-06T03:29:38.627293+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:37.570693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:37.570693+0000","last_clean_scrub_stamp":"2023-10-06T03:29:37.570693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T06:00:28.537215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1,2],"acting":[0,7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":14,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.589343+0000","last_change":"2023-10-06T03:29:33.543906+0000","last_active":"2023-10-06T03:29:38.589343+0000","last_peered":"2023-10-06T03:29:38.589343+0000","last_clean":"2023-10-06T03:29:38.589343+0000","last_became_active":"2023-10-06T03:29:33.543551+0000","last_became_peered":"2023-10-06T03:29:33.543551+0000","last_unstale":"2023-10-06T03:29:38.589343+0000","last_undegraded":"2023-10-06T03:29:38.589343+0000","last_fullsized":"2023-10-06T03:29:38.589343+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:32.495616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:32.495616+0000","last_clean_scrub_stamp":"2023-10-06T03:29:32.495616+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T03:34:32.799922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.609821+0000","last_change":"2023-10-06T03:29:34.549763+0000","last_active":"2023-10-06T03:29:38.609821+0000","last_peered":"2023-10-06T03:29:38.609821+0000","last_clean":"2023-10-06T03:29:38.609821+0000","last_became_active":"2023-10-06T03:29:34.549402+0000","last_became_peered":"2023-10-06T03:29:34.549402+0000","last_unstale":"2023-10-06T03:29:38.609821+0000","last_undegraded":"2023-10-06T03:29:38.609821+0000","last_fullsized":"2023-10-06T03:29:38.609821+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_clean_scrub_stamp":"2023-10-06T03:29:33.500929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T07:14:03.653521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.588889+0000","last_change":"2023-10-06T03:29:34.548269+0000","last_active":"2023-10-06T03:29:38.588889+0000","last_peered":"2023-10-06T03:29:38.588889+0000","last_clean":"2023-10-06T03:29:38.588889+0000","last_became_active":"2023-10-06T03:29:34.547982+0000","last_became_peered":"2023-10-06T03:29:34.547982+0000","last_unstale":"2023-10-06T03:29:38.588889+0000","last_undegraded":"2023-10-06T03:29:38.588889+0000","last_fullsized":"2023-10-06T03:29:38.588889+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_clean_scrub_stamp":"2023-10-06T03:29:33.500929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T03:54:50.291053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":14,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.596539+0000","last_change":"2023-10-06T03:29:33.542538+0000","last_active":"2023-10-06T03:29:38.596539+0000","last_peered":"2023-10-06T03:29:38.596539+0000","last_clean":"2023-10-06T03:29:38.596539+0000","last_became_active":"2023-10-06T03:29:33.536222+0000","last_became_peered":"2023-10-06T03:29:33.536222+0000","last_unstale":"2023-10-06T03:29:38.596539+0000","last_undegraded":"2023-10-06T03:29:38.596539+0000","last_fullsized":"2023-10-06T03:29:38.596539+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:32.495616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:32.495616+0000","last_clean_scrub_stamp":"2023-10-06T03:29:32.495616+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T07:14:02.648208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.620952+0000","last_change":"2023-10-06T03:29:38.613462+0000","last_active":"2023-10-06T03:29:38.620952+0000","last_peered":"2023-10-06T03:29:38.620952+0000","last_clean":"2023-10-06T03:29:38.620952+0000","last_became_active":"2023-10-06T03:29:38.613096+0000","last_became_peered":"2023-10-06T03:29:38.613096+0000","last_unstale":"2023-10-06T03:29:38.620952+0000","last_undegraded":"2023-10-06T03:29:38.620952+0000","last_fullsized":"2023-10-06T03:29:38.620952+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:37.570693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:37.570693+0000","last_clean_scrub_stamp":"2023-10-06T03:29:37.570693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T11:28:53.015339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7,1],"acting":[6,4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.589101+0000","last_change":"2023-10-06T03:29:34.546990+0000","last_active":"2023-10-06T03:29:38.589101+0000","last_peered":"2023-10-06T03:29:38.589101+0000","last_clean":"2023-10-06T03:29:38.589101+0000","last_became_active":"2023-10-06T03:29:34.544561+0000","last_became_peered":"2023-10-06T03:29:34.544561+0000","last_unstale":"2023-10-06T03:29:38.589101+0000","last_undegraded":"2023-10-06T03:29:38.589101+0000","last_fullsized":"2023-10-06T03:29:38.589101+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_clean_scrub_stamp":"2023-10-06T03:29:33.500929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T09:13:47.732421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":14,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.602311+0000","last_change":"2023-10-06T03:29:33.536276+0000","last_active":"2023-10-06T03:29:38.602311+0000","last_peered":"2023-10-06T03:29:38.602311+0000","last_clean":"2023-10-06T03:29:38.602311+0000","last_became_active":"2023-10-06T03:29:33.535859+0000","last_became_peered":"2023-10-06T03:29:33.535859+0000","last_unstale":"2023-10-06T03:29:38.602311+0000","last_undegraded":"2023-10-06T03:29:38.602311+0000","last_fullsized":"2023-10-06T03:29:38.602311+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:32.495616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:32.495616+0000","last_clean_scrub_stamp":"2023-10-06T03:29:32.495616+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T07:55:07.787102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.623243+0000","last_change":"2023-10-06T03:29:38.618743+0000","last_active":"2023-10-06T03:29:38.623243+0000","last_peered":"2023-10-06T03:29:38.623243+0000","last_clean":"2023-10-06T03:29:38.623243+0000","last_became_active":"2023-10-06T03:29:38.618260+0000","last_became_peered":"2023-10-06T03:29:38.618260+0000","last_unstale":"2023-10-06T03:29:38.623243+0000","last_undegraded":"2023-10-06T03:29:38.623243+0000","last_fullsized":"2023-10-06T03:29:38.623243+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:37.570693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:37.570693+0000","last_clean_scrub_stamp":"2023-10-06T03:29:37.570693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T04:14:12.838199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6,0],"acting":[7,4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.15","version":"23'3","reported_seq":16,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.606179+0000","last_change":"2023-10-06T03:29:33.541547+0000","last_active":"2023-10-06T03:29:38.606179+0000","last_peered":"2023-10-06T03:29:38.606179+0000","last_clean":"2023-10-06T03:29:38.606179+0000","last_became_active":"2023-10-06T03:29:33.541267+0000","last_became_peered":"2023-10-06T03:29:33.541267+0000","last_unstale":"2023-10-06T03:29:38.606179+0000","last_undegraded":"2023-10-06T03:29:38.606179+0000","last_fullsized":"2023-10-06T03:29:38.606179+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:32.495616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:32.495616+0000","last_clean_scrub_stamp":"2023-10-06T03:29:32.495616+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T12:25:10.766051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.608031+0000","last_change":"2023-10-06T03:29:34.550174+0000","last_active":"2023-10-06T03:29:38.608031+0000","last_peered":"2023-10-06T03:29:38.608031+0000","last_clean":"2023-10-06T03:29:38.608031+0000","last_became_active":"2023-10-06T03:29:34.549630+0000","last_became_peered":"2023-10-06T03:29:34.549630+0000","last_unstale":"2023-10-06T03:29:38.608031+0000","last_undegraded":"2023-10-06T03:29:38.608031+0000","last_fullsized":"2023-10-06T03:29:38.608031+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_clean_scrub_stamp":"2023-10-06T03:29:33.500929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T13:02:33.356490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.620450+0000","last_change":"2023-10-06T03:29:38.614643+0000","last_active":"2023-10-06T03:29:38.620450+0000","last_peered":"2023-10-06T03:29:38.620450+0000","last_clean":"2023-10-06T03:29:38.620450+0000","last_became_active":"2023-10-06T03:29:38.614242+0000","last_became_peered":"2023-10-06T03:29:38.614242+0000","last_unstale":"2023-10-06T03:29:38.620450+0000","last_undegraded":"2023-10-06T03:29:38.620450+0000","last_fullsized":"2023-10-06T03:29:38.620450+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:37.570693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:37.570693+0000","last_clean_scrub_stamp":"2023-10-06T03:29:37.570693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T06:29:58.268896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,5],"acting":[3,0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.14","version":"28'9","reported_seq":23,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T03:29:44.383979+0000","last_change":"2023-10-06T03:29:33.538661+0000","last_active":"2023-10-06T03:29:44.383979+0000","last_peered":"2023-10-06T03:29:44.383979+0000","last_clean":"2023-10-06T03:29:44.383979+0000","last_became_active":"2023-10-06T03:29:33.538296+0000","last_became_peered":"2023-10-06T03:29:33.538296+0000","last_unstale":"2023-10-06T03:29:44.383979+0000","last_undegraded":"2023-10-06T03:29:44.383979+0000","last_fullsized":"2023-10-06T03:29:44.383979+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:32.495616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:32.495616+0000","last_clean_scrub_stamp":"2023-10-06T03:29:32.495616+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T10:47:52.547584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3714,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.593839+0000","last_change":"2023-10-06T03:29:34.549928+0000","last_active":"2023-10-06T03:29:38.593839+0000","last_peered":"2023-10-06T03:29:38.593839+0000","last_clean":"2023-10-06T03:29:38.593839+0000","last_became_active":"2023-10-06T03:29:34.549421+0000","last_became_peered":"2023-10-06T03:29:34.549421+0000","last_unstale":"2023-10-06T03:29:38.593839+0000","last_undegraded":"2023-10-06T03:29:38.593839+0000","last_fullsized":"2023-10-06T03:29:38.593839+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_clean_scrub_stamp":"2023-10-06T03:29:33.500929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T03:45:19.939590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.624617+0000","last_change":"2023-10-06T03:29:38.620526+0000","last_active":"2023-10-06T03:29:38.624617+0000","last_peered":"2023-10-06T03:29:38.624617+0000","last_clean":"2023-10-06T03:29:38.624617+0000","last_became_active":"2023-10-06T03:29:38.620095+0000","last_became_peered":"2023-10-06T03:29:38.620095+0000","last_unstale":"2023-10-06T03:29:38.624617+0000","last_undegraded":"2023-10-06T03:29:38.624617+0000","last_fullsized":"2023-10-06T03:29:38.624617+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:37.570693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:37.570693+0000","last_clean_scrub_stamp":"2023-10-06T03:29:37.570693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T04:00:40.252544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3,7],"acting":[1,5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.594637+0000","last_change":"2023-10-06T03:29:34.538055+0000","last_active":"2023-10-06T03:29:38.594637+0000","last_peered":"2023-10-06T03:29:38.594637+0000","last_clean":"2023-10-06T03:29:38.594637+0000","last_became_active":"2023-10-06T03:29:34.537755+0000","last_became_peered":"2023-10-06T03:29:34.537755+0000","last_unstale":"2023-10-06T03:29:38.594637+0000","last_undegraded":"2023-10-06T03:29:38.594637+0000","last_fullsized":"2023-10-06T03:29:38.594637+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_clean_scrub_stamp":"2023-10-06T03:29:33.500929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T08:21:53.296989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":16,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.610021+0000","last_change":"2023-10-06T03:29:29.387402+0000","last_active":"2023-10-06T03:29:38.610021+0000","last_peered":"2023-10-06T03:29:38.610021+0000","last_clean":"2023-10-06T03:29:38.610021+0000","last_became_active":"2023-10-06T03:29:29.386812+0000","last_became_peered":"2023-10-06T03:29:29.386812+0000","last_unstale":"2023-10-06T03:29:38.610021+0000","last_undegraded":"2023-10-06T03:29:38.610021+0000","last_fullsized":"2023-10-06T03:29:38.610021+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:28.357506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:28.357506+0000","last_clean_scrub_stamp":"2023-10-06T03:29:28.357506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T12:09:43.895810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"27'1","reported_seq":15,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:39.494514+0000","last_change":"2023-10-06T03:29:33.542307+0000","last_active":"2023-10-06T03:29:39.494514+0000","last_peered":"2023-10-06T03:29:39.494514+0000","last_clean":"2023-10-06T03:29:39.494514+0000","last_became_active":"2023-10-06T03:29:33.541670+0000","last_became_peered":"2023-10-06T03:29:33.541670+0000","last_unstale":"2023-10-06T03:29:39.494514+0000","last_undegraded":"2023-10-06T03:29:39.494514+0000","last_fullsized":"2023-10-06T03:29:39.494514+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:32.495616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:32.495616+0000","last_clean_scrub_stamp":"2023-10-06T03:29:32.495616+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T05:03:47.111820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.620435+0000","last_change":"2023-10-06T03:29:38.613923+0000","last_active":"2023-10-06T03:29:38.620435+0000","last_peered":"2023-10-06T03:29:38.620435+0000","last_clean":"2023-10-06T03:29:38.620435+0000","last_became_active":"2023-10-06T03:29:38.612452+0000","last_became_peered":"2023-10-06T03:29:38.612452+0000","last_unstale":"2023-10-06T03:29:38.620435+0000","last_undegraded":"2023-10-06T03:29:38.620435+0000","last_fullsized":"2023-10-06T03:29:38.620435+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:37.570693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:37.570693+0000","last_clean_scrub_stamp":"2023-10-06T03:29:37.570693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T05:52:46.994576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4,0],"acting":[3,1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.589121+0000","last_change":"2023-10-06T03:29:34.539246+0000","last_active":"2023-10-06T03:29:38.589121+0000","last_peered":"2023-10-06T03:29:38.589121+0000","last_clean":"2023-10-06T03:29:38.589121+0000","last_became_active":"2023-10-06T03:29:34.538925+0000","last_became_peered":"2023-10-06T03:29:34.538925+0000","last_unstale":"2023-10-06T03:29:38.589121+0000","last_undegraded":"2023-10-06T03:29:38.589121+0000","last_fullsized":"2023-10-06T03:29:38.589121+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_clean_scrub_stamp":"2023-10-06T03:29:33.500929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T03:59:12.193877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":16,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.609252+0000","last_change":"2023-10-06T03:29:29.386238+0000","last_active":"2023-10-06T03:29:38.609252+0000","last_peered":"2023-10-06T03:29:38.609252+0000","last_clean":"2023-10-06T03:29:38.609252+0000","last_became_active":"2023-10-06T03:29:29.385865+0000","last_became_peered":"2023-10-06T03:29:29.385865+0000","last_unstale":"2023-10-06T03:29:38.609252+0000","last_undegraded":"2023-10-06T03:29:38.609252+0000","last_fullsized":"2023-10-06T03:29:38.609252+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:28.357506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:28.357506+0000","last_clean_scrub_stamp":"2023-10-06T03:29:28.357506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T12:43:34.714730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"23'2","reported_seq":16,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.593260+0000","last_change":"2023-10-06T03:29:33.539135+0000","last_active":"2023-10-06T03:29:38.593260+0000","last_peered":"2023-10-06T03:29:38.593260+0000","last_clean":"2023-10-06T03:29:38.593260+0000","last_became_active":"2023-10-06T03:29:33.538818+0000","last_became_peered":"2023-10-06T03:29:33.538818+0000","last_unstale":"2023-10-06T03:29:38.593260+0000","last_undegraded":"2023-10-06T03:29:38.593260+0000","last_fullsized":"2023-10-06T03:29:38.593260+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:32.495616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:32.495616+0000","last_clean_scrub_stamp":"2023-10-06T03:29:32.495616+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T03:54:50.215488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.622166+0000","last_change":"2023-10-06T03:29:38.614527+0000","last_active":"2023-10-06T03:29:38.622166+0000","last_peered":"2023-10-06T03:29:38.622166+0000","last_clean":"2023-10-06T03:29:38.622166+0000","last_became_active":"2023-10-06T03:29:38.614090+0000","last_became_peered":"2023-10-06T03:29:38.614090+0000","last_unstale":"2023-10-06T03:29:38.622166+0000","last_undegraded":"2023-10-06T03:29:38.622166+0000","last_fullsized":"2023-10-06T03:29:38.622166+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:37.570693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:37.570693+0000","last_clean_scrub_stamp":"2023-10-06T03:29:37.570693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T05:47:25.729169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7,0],"acting":[4,3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.592223+0000","last_change":"2023-10-06T03:29:34.537232+0000","last_active":"2023-10-06T03:29:38.592223+0000","last_peered":"2023-10-06T03:29:38.592223+0000","last_clean":"2023-10-06T03:29:38.592223+0000","last_became_active":"2023-10-06T03:29:34.536984+0000","last_became_peered":"2023-10-06T03:29:34.536984+0000","last_unstale":"2023-10-06T03:29:38.592223+0000","last_undegraded":"2023-10-06T03:29:38.592223+0000","last_fullsized":"2023-10-06T03:29:38.592223+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_clean_scrub_stamp":"2023-10-06T03:29:33.500929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T03:42:52.876376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":16,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.605996+0000","last_change":"2023-10-06T03:29:29.384279+0000","last_active":"2023-10-06T03:29:38.605996+0000","last_peered":"2023-10-06T03:29:38.605996+0000","last_clean":"2023-10-06T03:29:38.605996+0000","last_became_active":"2023-10-06T03:29:29.383533+0000","last_became_peered":"2023-10-06T03:29:29.383533+0000","last_unstale":"2023-10-06T03:29:38.605996+0000","last_undegraded":"2023-10-06T03:29:38.605996+0000","last_fullsized":"2023-10-06T03:29:38.605996+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:28.357506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:28.357506+0000","last_clean_scrub_stamp":"2023-10-06T03:29:28.357506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T08:34:19.641525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.609261+0000","last_change":"2023-10-06T03:29:33.542909+0000","last_active":"2023-10-06T03:29:38.609261+0000","last_peered":"2023-10-06T03:29:38.609261+0000","last_clean":"2023-10-06T03:29:38.609261+0000","last_became_active":"2023-10-06T03:29:33.542573+0000","last_became_peered":"2023-10-06T03:29:33.542573+0000","last_unstale":"2023-10-06T03:29:38.609261+0000","last_undegraded":"2023-10-06T03:29:38.609261+0000","last_fullsized":"2023-10-06T03:29:38.609261+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:32.495616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:32.495616+0000","last_clean_scrub_stamp":"2023-10-06T03:29:32.495616+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T14:26:49.262597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.620982+0000","last_change":"2023-10-06T03:29:38.615475+0000","last_active":"2023-10-06T03:29:38.620982+0000","last_peered":"2023-10-06T03:29:38.620982+0000","last_clean":"2023-10-06T03:29:38.620982+0000","last_became_active":"2023-10-06T03:29:38.615049+0000","last_became_peered":"2023-10-06T03:29:38.615049+0000","last_unstale":"2023-10-06T03:29:38.620982+0000","last_undegraded":"2023-10-06T03:29:38.620982+0000","last_fullsized":"2023-10-06T03:29:38.620982+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:37.570693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:37.570693+0000","last_clean_scrub_stamp":"2023-10-06T03:29:37.570693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T14:34:30.474789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5,2],"acting":[6,0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.608125+0000","last_change":"2023-10-06T03:29:34.543257+0000","last_active":"2023-10-06T03:29:38.608125+0000","last_peered":"2023-10-06T03:29:38.608125+0000","last_clean":"2023-10-06T03:29:38.608125+0000","last_became_active":"2023-10-06T03:29:34.537574+0000","last_became_peered":"2023-10-06T03:29:34.537574+0000","last_unstale":"2023-10-06T03:29:38.608125+0000","last_undegraded":"2023-10-06T03:29:38.608125+0000","last_fullsized":"2023-10-06T03:29:38.608125+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_clean_scrub_stamp":"2023-10-06T03:29:33.500929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T13:34:24.699976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":16,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.609348+0000","last_change":"2023-10-06T03:29:29.382183+0000","last_active":"2023-10-06T03:29:38.609348+0000","last_peered":"2023-10-06T03:29:38.609348+0000","last_clean":"2023-10-06T03:29:38.609348+0000","last_became_active":"2023-10-06T03:29:29.381918+0000","last_became_peered":"2023-10-06T03:29:29.381918+0000","last_unstale":"2023-10-06T03:29:38.609348+0000","last_undegraded":"2023-10-06T03:29:38.609348+0000","last_fullsized":"2023-10-06T03:29:38.609348+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:28.357506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:28.357506+0000","last_clean_scrub_stamp":"2023-10-06T03:29:28.357506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T11:56:26.013716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":14,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.602189+0000","last_change":"2023-10-06T03:29:33.540375+0000","last_active":"2023-10-06T03:29:38.602189+0000","last_peered":"2023-10-06T03:29:38.602189+0000","last_clean":"2023-10-06T03:29:38.602189+0000","last_became_active":"2023-10-06T03:29:33.540016+0000","last_became_peered":"2023-10-06T03:29:33.540016+0000","last_unstale":"2023-10-06T03:29:38.602189+0000","last_undegraded":"2023-10-06T03:29:38.602189+0000","last_fullsized":"2023-10-06T03:29:38.602189+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:32.495616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:32.495616+0000","last_clean_scrub_stamp":"2023-10-06T03:29:32.495616+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T10:47:52.547584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.623995+0000","last_change":"2023-10-06T03:29:38.613181+0000","last_active":"2023-10-06T03:29:38.623995+0000","last_peered":"2023-10-06T03:29:38.623995+0000","last_clean":"2023-10-06T03:29:38.623995+0000","last_became_active":"2023-10-06T03:29:38.612787+0000","last_became_peered":"2023-10-06T03:29:38.612787+0000","last_unstale":"2023-10-06T03:29:38.623995+0000","last_undegraded":"2023-10-06T03:29:38.623995+0000","last_fullsized":"2023-10-06T03:29:38.623995+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:37.570693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:37.570693+0000","last_clean_scrub_stamp":"2023-10-06T03:29:37.570693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T07:38:31.118869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6,5],"acting":[0,1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.609205+0000","last_change":"2023-10-06T03:29:34.543207+0000","last_active":"2023-10-06T03:29:38.609205+0000","last_peered":"2023-10-06T03:29:38.609205+0000","last_clean":"2023-10-06T03:29:38.609205+0000","last_became_active":"2023-10-06T03:29:34.537291+0000","last_became_peered":"2023-10-06T03:29:34.537291+0000","last_unstale":"2023-10-06T03:29:38.609205+0000","last_undegraded":"2023-10-06T03:29:38.609205+0000","last_fullsized":"2023-10-06T03:29:38.609205+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_clean_scrub_stamp":"2023-10-06T03:29:33.500929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T06:40:43.933850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":14,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.594240+0000","last_change":"2023-10-06T03:29:33.543412+0000","last_active":"2023-10-06T03:29:38.594240+0000","last_peered":"2023-10-06T03:29:38.594240+0000","last_clean":"2023-10-06T03:29:38.594240+0000","last_became_active":"2023-10-06T03:29:33.543114+0000","last_became_peered":"2023-10-06T03:29:33.543114+0000","last_unstale":"2023-10-06T03:29:38.594240+0000","last_undegraded":"2023-10-06T03:29:38.594240+0000","last_fullsized":"2023-10-06T03:29:38.594240+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:32.495616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:32.495616+0000","last_clean_scrub_stamp":"2023-10-06T03:29:32.495616+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T15:12:57.750941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":17,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.600966+0000","last_change":"2023-10-06T03:29:29.384415+0000","last_active":"2023-10-06T03:29:38.600966+0000","last_peered":"2023-10-06T03:29:38.600966+0000","last_clean":"2023-10-06T03:29:38.600966+0000","last_became_active":"2023-10-06T03:29:29.383894+0000","last_became_peered":"2023-10-06T03:29:29.383894+0000","last_unstale":"2023-10-06T03:29:38.600966+0000","last_undegraded":"2023-10-06T03:29:38.600966+0000","last_fullsized":"2023-10-06T03:29:38.600966+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:28.357506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:28.357506+0000","last_clean_scrub_stamp":"2023-10-06T03:29:28.357506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T13:09:00.657107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.622959+0000","last_change":"2023-10-06T03:29:38.613076+0000","last_active":"2023-10-06T03:29:38.622959+0000","last_peered":"2023-10-06T03:29:38.622959+0000","last_clean":"2023-10-06T03:29:38.622959+0000","last_became_active":"2023-10-06T03:29:38.612646+0000","last_became_peered":"2023-10-06T03:29:38.612646+0000","last_unstale":"2023-10-06T03:29:38.622959+0000","last_undegraded":"2023-10-06T03:29:38.622959+0000","last_fullsized":"2023-10-06T03:29:38.622959+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:37.570693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:37.570693+0000","last_clean_scrub_stamp":"2023-10-06T03:29:37.570693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T06:07:31.871068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4,6],"acting":[0,1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.609278+0000","last_change":"2023-10-06T03:29:34.543346+0000","last_active":"2023-10-06T03:29:38.609278+0000","last_peered":"2023-10-06T03:29:38.609278+0000","last_clean":"2023-10-06T03:29:38.609278+0000","last_became_active":"2023-10-06T03:29:34.537438+0000","last_became_peered":"2023-10-06T03:29:34.537438+0000","last_unstale":"2023-10-06T03:29:38.609278+0000","last_undegraded":"2023-10-06T03:29:38.609278+0000","last_fullsized":"2023-10-06T03:29:38.609278+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_clean_scrub_stamp":"2023-10-06T03:29:33.500929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T09:13:47.732421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.609403+0000","last_change":"2023-10-06T03:29:33.544242+0000","last_active":"2023-10-06T03:29:38.609403+0000","last_peered":"2023-10-06T03:29:38.609403+0000","last_clean":"2023-10-06T03:29:38.609403+0000","last_became_active":"2023-10-06T03:29:33.543938+0000","last_became_peered":"2023-10-06T03:29:33.543938+0000","last_unstale":"2023-10-06T03:29:38.609403+0000","last_undegraded":"2023-10-06T03:29:38.609403+0000","last_fullsized":"2023-10-06T03:29:38.609403+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:32.495616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:32.495616+0000","last_clean_scrub_stamp":"2023-10-06T03:29:32.495616+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T10:09:45.061520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":16,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.599964+0000","last_change":"2023-10-06T03:29:29.606195+0000","last_active":"2023-10-06T03:29:38.599964+0000","last_peered":"2023-10-06T03:29:38.599964+0000","last_clean":"2023-10-06T03:29:38.599964+0000","last_became_active":"2023-10-06T03:29:29.605489+0000","last_became_peered":"2023-10-06T03:29:29.605489+0000","last_unstale":"2023-10-06T03:29:38.599964+0000","last_undegraded":"2023-10-06T03:29:38.599964+0000","last_fullsized":"2023-10-06T03:29:38.599964+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:28.357506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:28.357506+0000","last_clean_scrub_stamp":"2023-10-06T03:29:28.357506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T10:21:35.864895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.626333+0000","last_change":"2023-10-06T03:29:38.624434+0000","last_active":"2023-10-06T03:29:38.626333+0000","last_peered":"2023-10-06T03:29:38.626333+0000","last_clean":"2023-10-06T03:29:38.626333+0000","last_became_active":"2023-10-06T03:29:38.623676+0000","last_became_peered":"2023-10-06T03:29:38.623676+0000","last_unstale":"2023-10-06T03:29:38.626333+0000","last_undegraded":"2023-10-06T03:29:38.626333+0000","last_fullsized":"2023-10-06T03:29:38.626333+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:37.570693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:37.570693+0000","last_clean_scrub_stamp":"2023-10-06T03:29:37.570693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T12:26:20.152406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,1,7],"acting":[2,5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.599731+0000","last_change":"2023-10-06T03:29:34.550793+0000","last_active":"2023-10-06T03:29:38.599731+0000","last_peered":"2023-10-06T03:29:38.599731+0000","last_clean":"2023-10-06T03:29:38.599731+0000","last_became_active":"2023-10-06T03:29:34.550345+0000","last_became_peered":"2023-10-06T03:29:34.550345+0000","last_unstale":"2023-10-06T03:29:38.599731+0000","last_undegraded":"2023-10-06T03:29:38.599731+0000","last_fullsized":"2023-10-06T03:29:38.599731+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_clean_scrub_stamp":"2023-10-06T03:29:33.500929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T11:28:48.945575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":14,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.591494+0000","last_change":"2023-10-06T03:29:33.543304+0000","last_active":"2023-10-06T03:29:38.591494+0000","last_peered":"2023-10-06T03:29:38.591494+0000","last_clean":"2023-10-06T03:29:38.591494+0000","last_became_active":"2023-10-06T03:29:33.542483+0000","last_became_peered":"2023-10-06T03:29:33.542483+0000","last_unstale":"2023-10-06T03:29:38.591494+0000","last_undegraded":"2023-10-06T03:29:38.591494+0000","last_fullsized":"2023-10-06T03:29:38.591494+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:32.495616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:32.495616+0000","last_clean_scrub_stamp":"2023-10-06T03:29:32.495616+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T11:58:53.137774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":16,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.603498+0000","last_change":"2023-10-06T03:29:29.384353+0000","last_active":"2023-10-06T03:29:38.603498+0000","last_peered":"2023-10-06T03:29:38.603498+0000","last_clean":"2023-10-06T03:29:38.603498+0000","last_became_active":"2023-10-06T03:29:29.383884+0000","last_became_peered":"2023-10-06T03:29:29.383884+0000","last_unstale":"2023-10-06T03:29:38.603498+0000","last_undegraded":"2023-10-06T03:29:38.603498+0000","last_fullsized":"2023-10-06T03:29:38.603498+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:28.357506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:28.357506+0000","last_clean_scrub_stamp":"2023-10-06T03:29:28.357506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T14:43:09.107140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.629976+0000","last_change":"2023-10-06T03:29:38.626568+0000","last_active":"2023-10-06T03:29:38.629976+0000","last_peered":"2023-10-06T03:29:38.629976+0000","last_clean":"2023-10-06T03:29:38.629976+0000","last_became_active":"2023-10-06T03:29:38.625323+0000","last_became_peered":"2023-10-06T03:29:38.625323+0000","last_unstale":"2023-10-06T03:29:38.629976+0000","last_undegraded":"2023-10-06T03:29:38.629976+0000","last_fullsized":"2023-10-06T03:29:38.629976+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:37.570693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:37.570693+0000","last_clean_scrub_stamp":"2023-10-06T03:29:37.570693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T04:51:55.443872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0,4],"acting":[5,1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.609554+0000","last_change":"2023-10-06T03:29:34.549281+0000","last_active":"2023-10-06T03:29:38.609554+0000","last_peered":"2023-10-06T03:29:38.609554+0000","last_clean":"2023-10-06T03:29:38.609554+0000","last_became_active":"2023-10-06T03:29:34.548900+0000","last_became_peered":"2023-10-06T03:29:34.548900+0000","last_unstale":"2023-10-06T03:29:38.609554+0000","last_undegraded":"2023-10-06T03:29:38.609554+0000","last_fullsized":"2023-10-06T03:29:38.609554+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_clean_scrub_stamp":"2023-10-06T03:29:33.500929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T09:06:08.844003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":14,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.589353+0000","last_change":"2023-10-06T03:29:33.534969+0000","last_active":"2023-10-06T03:29:38.589353+0000","last_peered":"2023-10-06T03:29:38.589353+0000","last_clean":"2023-10-06T03:29:38.589353+0000","last_became_active":"2023-10-06T03:29:33.534402+0000","last_became_peered":"2023-10-06T03:29:33.534402+0000","last_unstale":"2023-10-06T03:29:38.589353+0000","last_undegraded":"2023-10-06T03:29:38.589353+0000","last_fullsized":"2023-10-06T03:29:38.589353+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:32.495616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:32.495616+0000","last_clean_scrub_stamp":"2023-10-06T03:29:32.495616+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T11:00:52.451461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":17,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.601254+0000","last_change":"2023-10-06T03:29:29.386120+0000","last_active":"2023-10-06T03:29:38.601254+0000","last_peered":"2023-10-06T03:29:38.601254+0000","last_clean":"2023-10-06T03:29:38.601254+0000","last_became_active":"2023-10-06T03:29:29.385692+0000","last_became_peered":"2023-10-06T03:29:29.385692+0000","last_unstale":"2023-10-06T03:29:38.601254+0000","last_undegraded":"2023-10-06T03:29:38.601254+0000","last_fullsized":"2023-10-06T03:29:38.601254+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:28.357506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:28.357506+0000","last_clean_scrub_stamp":"2023-10-06T03:29:28.357506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T10:21:35.864895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":169,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.603558+0000","last_change":"2023-10-06T03:29:21.978311+0000","last_active":"2023-10-06T03:29:38.603558+0000","last_peered":"2023-10-06T03:29:38.603558+0000","last_clean":"2023-10-06T03:29:38.603558+0000","last_became_active":"2023-10-06T03:29:21.978017+0000","last_became_peered":"2023-10-06T03:29:21.978017+0000","last_unstale":"2023-10-06T03:29:38.603558+0000","last_undegraded":"2023-10-06T03:29:38.603558+0000","last_fullsized":"2023-10-06T03:29:38.603558+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:20.957955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:20.957955+0000","last_clean_scrub_stamp":"2023-10-06T03:29:20.957955+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T07:33:10.429084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.614806+0000","last_change":"2023-10-06T03:29:38.612549+0000","last_active":"2023-10-06T03:29:38.614806+0000","last_peered":"2023-10-06T03:29:38.614806+0000","last_clean":"2023-10-06T03:29:38.614806+0000","last_became_active":"2023-10-06T03:29:38.611790+0000","last_became_peered":"2023-10-06T03:29:38.611790+0000","last_unstale":"2023-10-06T03:29:38.614806+0000","last_undegraded":"2023-10-06T03:29:38.614806+0000","last_fullsized":"2023-10-06T03:29:38.614806+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:37.570693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:37.570693+0000","last_clean_scrub_stamp":"2023-10-06T03:29:37.570693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T12:17:41.605048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5,3],"acting":[7,2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.608679+0000","last_change":"2023-10-06T03:29:34.545006+0000","last_active":"2023-10-06T03:29:38.608679+0000","last_peered":"2023-10-06T03:29:38.608679+0000","last_clean":"2023-10-06T03:29:38.608679+0000","last_became_active":"2023-10-06T03:29:34.544682+0000","last_became_peered":"2023-10-06T03:29:34.544682+0000","last_unstale":"2023-10-06T03:29:38.608679+0000","last_undegraded":"2023-10-06T03:29:38.608679+0000","last_fullsized":"2023-10-06T03:29:38.608679+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_clean_scrub_stamp":"2023-10-06T03:29:33.500929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T03:59:12.193877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":14,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.589559+0000","last_change":"2023-10-06T03:29:33.545053+0000","last_active":"2023-10-06T03:29:38.589559+0000","last_peered":"2023-10-06T03:29:38.589559+0000","last_clean":"2023-10-06T03:29:38.589559+0000","last_became_active":"2023-10-06T03:29:33.544749+0000","last_became_peered":"2023-10-06T03:29:33.544749+0000","last_unstale":"2023-10-06T03:29:38.589559+0000","last_undegraded":"2023-10-06T03:29:38.589559+0000","last_fullsized":"2023-10-06T03:29:38.589559+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:32.495616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:32.495616+0000","last_clean_scrub_stamp":"2023-10-06T03:29:32.495616+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T14:26:49.262597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.621981+0000","last_change":"2023-10-06T03:29:38.614480+0000","last_active":"2023-10-06T03:29:38.621981+0000","last_peered":"2023-10-06T03:29:38.621981+0000","last_clean":"2023-10-06T03:29:38.621981+0000","last_became_active":"2023-10-06T03:29:38.613976+0000","last_became_peered":"2023-10-06T03:29:38.613976+0000","last_unstale":"2023-10-06T03:29:38.621981+0000","last_undegraded":"2023-10-06T03:29:38.621981+0000","last_fullsized":"2023-10-06T03:29:38.621981+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:37.570693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:37.570693+0000","last_clean_scrub_stamp":"2023-10-06T03:29:37.570693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T09:04:13.741349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0,2],"acting":[4,5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.592722+0000","last_change":"2023-10-06T03:29:34.551311+0000","last_active":"2023-10-06T03:29:38.592722+0000","last_peered":"2023-10-06T03:29:38.592722+0000","last_clean":"2023-10-06T03:29:38.592722+0000","last_became_active":"2023-10-06T03:29:34.550961+0000","last_became_peered":"2023-10-06T03:29:34.550961+0000","last_unstale":"2023-10-06T03:29:38.592722+0000","last_undegraded":"2023-10-06T03:29:38.592722+0000","last_fullsized":"2023-10-06T03:29:38.592722+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_clean_scrub_stamp":"2023-10-06T03:29:33.500929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T14:20:50.168977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":14,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.592869+0000","last_change":"2023-10-06T03:29:33.538732+0000","last_active":"2023-10-06T03:29:38.592869+0000","last_peered":"2023-10-06T03:29:38.592869+0000","last_clean":"2023-10-06T03:29:38.592869+0000","last_became_active":"2023-10-06T03:29:33.538430+0000","last_became_peered":"2023-10-06T03:29:33.538430+0000","last_unstale":"2023-10-06T03:29:38.592869+0000","last_undegraded":"2023-10-06T03:29:38.592869+0000","last_fullsized":"2023-10-06T03:29:38.592869+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:32.495616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:32.495616+0000","last_clean_scrub_stamp":"2023-10-06T03:29:32.495616+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T12:57:50.614907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.621727+0000","last_change":"2023-10-06T03:29:38.617352+0000","last_active":"2023-10-06T03:29:38.621727+0000","last_peered":"2023-10-06T03:29:38.621727+0000","last_clean":"2023-10-06T03:29:38.621727+0000","last_became_active":"2023-10-06T03:29:38.616718+0000","last_became_peered":"2023-10-06T03:29:38.616718+0000","last_unstale":"2023-10-06T03:29:38.621727+0000","last_undegraded":"2023-10-06T03:29:38.621727+0000","last_fullsized":"2023-10-06T03:29:38.621727+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:37.570693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:37.570693+0000","last_clean_scrub_stamp":"2023-10-06T03:29:37.570693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T11:16:01.323266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6,2],"acting":[5,4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.592923+0000","last_change":"2023-10-06T03:29:34.541404+0000","last_active":"2023-10-06T03:29:38.592923+0000","last_peered":"2023-10-06T03:29:38.592923+0000","last_clean":"2023-10-06T03:29:38.592923+0000","last_became_active":"2023-10-06T03:29:34.541037+0000","last_became_peered":"2023-10-06T03:29:34.541037+0000","last_unstale":"2023-10-06T03:29:38.592923+0000","last_undegraded":"2023-10-06T03:29:38.592923+0000","last_fullsized":"2023-10-06T03:29:38.592923+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_clean_scrub_stamp":"2023-10-06T03:29:33.500929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T08:57:07.064845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.610030+0000","last_change":"2023-10-06T03:29:33.542889+0000","last_active":"2023-10-06T03:29:38.610030+0000","last_peered":"2023-10-06T03:29:38.610030+0000","last_clean":"2023-10-06T03:29:38.610030+0000","last_became_active":"2023-10-06T03:29:33.542527+0000","last_became_peered":"2023-10-06T03:29:33.542527+0000","last_unstale":"2023-10-06T03:29:38.610030+0000","last_undegraded":"2023-10-06T03:29:38.610030+0000","last_fullsized":"2023-10-06T03:29:38.610030+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:32.495616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:32.495616+0000","last_clean_scrub_stamp":"2023-10-06T03:29:32.495616+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T09:16:59.387601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.624663+0000","last_change":"2023-10-06T03:29:38.616847+0000","last_active":"2023-10-06T03:29:38.624663+0000","last_peered":"2023-10-06T03:29:38.624663+0000","last_clean":"2023-10-06T03:29:38.624663+0000","last_became_active":"2023-10-06T03:29:38.616359+0000","last_became_peered":"2023-10-06T03:29:38.616359+0000","last_unstale":"2023-10-06T03:29:38.624663+0000","last_undegraded":"2023-10-06T03:29:38.624663+0000","last_fullsized":"2023-10-06T03:29:38.624663+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:37.570693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:37.570693+0000","last_clean_scrub_stamp":"2023-10-06T03:29:37.570693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T10:33:46.594949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0,6],"acting":[1,4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.593466+0000","last_change":"2023-10-06T03:29:34.542358+0000","last_active":"2023-10-06T03:29:38.593466+0000","last_peered":"2023-10-06T03:29:38.593466+0000","last_clean":"2023-10-06T03:29:38.593466+0000","last_became_active":"2023-10-06T03:29:34.542038+0000","last_became_peered":"2023-10-06T03:29:34.542038+0000","last_unstale":"2023-10-06T03:29:38.593466+0000","last_undegraded":"2023-10-06T03:29:38.593466+0000","last_fullsized":"2023-10-06T03:29:38.593466+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_clean_scrub_stamp":"2023-10-06T03:29:33.500929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T11:19:26.387732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"23'2","reported_seq":16,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.590740+0000","last_change":"2023-10-06T03:29:33.544626+0000","last_active":"2023-10-06T03:29:38.590740+0000","last_peered":"2023-10-06T03:29:38.590740+0000","last_clean":"2023-10-06T03:29:38.590740+0000","last_became_active":"2023-10-06T03:29:33.544301+0000","last_became_peered":"2023-10-06T03:29:33.544301+0000","last_unstale":"2023-10-06T03:29:38.590740+0000","last_undegraded":"2023-10-06T03:29:38.590740+0000","last_fullsized":"2023-10-06T03:29:38.590740+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:32.495616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:32.495616+0000","last_clean_scrub_stamp":"2023-10-06T03:29:32.495616+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T09:13:04.451900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.613447+0000","last_change":"2023-10-06T03:29:38.610963+0000","last_active":"2023-10-06T03:29:38.613447+0000","last_peered":"2023-10-06T03:29:38.613447+0000","last_clean":"2023-10-06T03:29:38.613447+0000","last_became_active":"2023-10-06T03:29:38.610492+0000","last_became_peered":"2023-10-06T03:29:38.610492+0000","last_unstale":"2023-10-06T03:29:38.613447+0000","last_undegraded":"2023-10-06T03:29:38.613447+0000","last_fullsized":"2023-10-06T03:29:38.613447+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:37.570693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:37.570693+0000","last_clean_scrub_stamp":"2023-10-06T03:29:37.570693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T04:15:31.186577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5,6],"acting":[2,7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.592107+0000","last_change":"2023-10-06T03:29:34.547048+0000","last_active":"2023-10-06T03:29:38.592107+0000","last_peered":"2023-10-06T03:29:38.592107+0000","last_clean":"2023-10-06T03:29:38.592107+0000","last_became_active":"2023-10-06T03:29:34.546757+0000","last_became_peered":"2023-10-06T03:29:34.546757+0000","last_unstale":"2023-10-06T03:29:38.592107+0000","last_undegraded":"2023-10-06T03:29:38.592107+0000","last_fullsized":"2023-10-06T03:29:38.592107+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_clean_scrub_stamp":"2023-10-06T03:29:33.500929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T14:20:50.168977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":14,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.600978+0000","last_change":"2023-10-06T03:29:33.537618+0000","last_active":"2023-10-06T03:29:38.600978+0000","last_peered":"2023-10-06T03:29:38.600978+0000","last_clean":"2023-10-06T03:29:38.600978+0000","last_became_active":"2023-10-06T03:29:33.537140+0000","last_became_peered":"2023-10-06T03:29:33.537140+0000","last_unstale":"2023-10-06T03:29:38.600978+0000","last_undegraded":"2023-10-06T03:29:38.600978+0000","last_fullsized":"2023-10-06T03:29:38.600978+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:32.495616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:32.495616+0000","last_clean_scrub_stamp":"2023-10-06T03:29:32.495616+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T12:25:10.766051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.623146+0000","last_change":"2023-10-06T03:29:38.616303+0000","last_active":"2023-10-06T03:29:38.623146+0000","last_peered":"2023-10-06T03:29:38.623146+0000","last_clean":"2023-10-06T03:29:38.623146+0000","last_became_active":"2023-10-06T03:29:38.615882+0000","last_became_peered":"2023-10-06T03:29:38.615882+0000","last_unstale":"2023-10-06T03:29:38.623146+0000","last_undegraded":"2023-10-06T03:29:38.623146+0000","last_fullsized":"2023-10-06T03:29:38.623146+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:37.570693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:37.570693+0000","last_clean_scrub_stamp":"2023-10-06T03:29:37.570693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T11:19:30.457497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3,6],"acting":[2,4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.610059+0000","last_change":"2023-10-06T03:29:34.546395+0000","last_active":"2023-10-06T03:29:38.610059+0000","last_peered":"2023-10-06T03:29:38.610059+0000","last_clean":"2023-10-06T03:29:38.610059+0000","last_became_active":"2023-10-06T03:29:34.546016+0000","last_became_peered":"2023-10-06T03:29:34.546016+0000","last_unstale":"2023-10-06T03:29:38.610059+0000","last_undegraded":"2023-10-06T03:29:38.610059+0000","last_fullsized":"2023-10-06T03:29:38.610059+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_clean_scrub_stamp":"2023-10-06T03:29:33.500929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T14:51:50.895276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"23'1","reported_seq":14,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.605833+0000","last_change":"2023-10-06T03:29:33.536994+0000","last_active":"2023-10-06T03:29:38.605833+0000","last_peered":"2023-10-06T03:29:38.605833+0000","last_clean":"2023-10-06T03:29:38.605833+0000","last_became_active":"2023-10-06T03:29:33.535768+0000","last_became_peered":"2023-10-06T03:29:33.535768+0000","last_unstale":"2023-10-06T03:29:38.605833+0000","last_undegraded":"2023-10-06T03:29:38.605833+0000","last_fullsized":"2023-10-06T03:29:38.605833+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:32.495616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:32.495616+0000","last_clean_scrub_stamp":"2023-10-06T03:29:32.495616+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T06:53:50.776266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.615680+0000","last_change":"2023-10-06T03:29:38.614617+0000","last_active":"2023-10-06T03:29:38.615680+0000","last_peered":"2023-10-06T03:29:38.615680+0000","last_clean":"2023-10-06T03:29:38.615680+0000","last_became_active":"2023-10-06T03:29:38.614134+0000","last_became_peered":"2023-10-06T03:29:38.614134+0000","last_unstale":"2023-10-06T03:29:38.615680+0000","last_undegraded":"2023-10-06T03:29:38.615680+0000","last_fullsized":"2023-10-06T03:29:38.615680+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:37.570693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:37.570693+0000","last_clean_scrub_stamp":"2023-10-06T03:29:37.570693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T07:51:25.532733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5,6],"acting":[2,0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.592039+0000","last_change":"2023-10-06T03:29:34.535355+0000","last_active":"2023-10-06T03:29:38.592039+0000","last_peered":"2023-10-06T03:29:38.592039+0000","last_clean":"2023-10-06T03:29:38.592039+0000","last_became_active":"2023-10-06T03:29:34.534944+0000","last_became_peered":"2023-10-06T03:29:34.534944+0000","last_unstale":"2023-10-06T03:29:38.592039+0000","last_undegraded":"2023-10-06T03:29:38.592039+0000","last_fullsized":"2023-10-06T03:29:38.592039+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_clean_scrub_stamp":"2023-10-06T03:29:33.500929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T09:39:33.168497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"23'2","reported_seq":15,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.605956+0000","last_change":"2023-10-06T03:29:33.545575+0000","last_active":"2023-10-06T03:29:38.605956+0000","last_peered":"2023-10-06T03:29:38.605956+0000","last_clean":"2023-10-06T03:29:38.605956+0000","last_became_active":"2023-10-06T03:29:33.545110+0000","last_became_peered":"2023-10-06T03:29:33.545110+0000","last_unstale":"2023-10-06T03:29:38.605956+0000","last_undegraded":"2023-10-06T03:29:38.605956+0000","last_fullsized":"2023-10-06T03:29:38.605956+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:32.495616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:32.495616+0000","last_clean_scrub_stamp":"2023-10-06T03:29:32.495616+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T14:51:49.889964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.623767+0000","last_change":"2023-10-06T03:29:38.620856+0000","last_active":"2023-10-06T03:29:38.623767+0000","last_peered":"2023-10-06T03:29:38.623767+0000","last_clean":"2023-10-06T03:29:38.623767+0000","last_became_active":"2023-10-06T03:29:38.620374+0000","last_became_peered":"2023-10-06T03:29:38.620374+0000","last_unstale":"2023-10-06T03:29:38.623767+0000","last_undegraded":"2023-10-06T03:29:38.623767+0000","last_fullsized":"2023-10-06T03:29:38.623767+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:37.570693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:37.570693+0000","last_clean_scrub_stamp":"2023-10-06T03:29:37.570693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T08:05:22.622058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1,5],"acting":[2,0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.600871+0000","last_change":"2023-10-06T03:29:34.537767+0000","last_active":"2023-10-06T03:29:38.600871+0000","last_peered":"2023-10-06T03:29:38.600871+0000","last_clean":"2023-10-06T03:29:38.600871+0000","last_became_active":"2023-10-06T03:29:34.537414+0000","last_became_peered":"2023-10-06T03:29:34.537414+0000","last_unstale":"2023-10-06T03:29:38.600871+0000","last_undegraded":"2023-10-06T03:29:38.600871+0000","last_fullsized":"2023-10-06T03:29:38.600871+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_clean_scrub_stamp":"2023-10-06T03:29:33.500929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T08:29:43.429905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"23'1","reported_seq":14,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.606271+0000","last_change":"2023-10-06T03:29:33.543577+0000","last_active":"2023-10-06T03:29:38.606271+0000","last_peered":"2023-10-06T03:29:38.606271+0000","last_clean":"2023-10-06T03:29:38.606271+0000","last_became_active":"2023-10-06T03:29:33.543265+0000","last_became_peered":"2023-10-06T03:29:33.543265+0000","last_unstale":"2023-10-06T03:29:38.606271+0000","last_undegraded":"2023-10-06T03:29:38.606271+0000","last_fullsized":"2023-10-06T03:29:38.606271+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:32.495616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:32.495616+0000","last_clean_scrub_stamp":"2023-10-06T03:29:32.495616+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T09:06:07.838690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.623296+0000","last_change":"2023-10-06T03:29:38.619910+0000","last_active":"2023-10-06T03:29:38.623296+0000","last_peered":"2023-10-06T03:29:38.623296+0000","last_clean":"2023-10-06T03:29:38.623296+0000","last_became_active":"2023-10-06T03:29:38.619517+0000","last_became_peered":"2023-10-06T03:29:38.619517+0000","last_unstale":"2023-10-06T03:29:38.623296+0000","last_undegraded":"2023-10-06T03:29:38.623296+0000","last_fullsized":"2023-10-06T03:29:38.623296+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:37.570693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:37.570693+0000","last_clean_scrub_stamp":"2023-10-06T03:29:37.570693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T04:29:21.981307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3,4],"acting":[7,6,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.627322+0000","last_change":"2023-10-06T03:29:38.623954+0000","last_active":"2023-10-06T03:29:38.627322+0000","last_peered":"2023-10-06T03:29:38.627322+0000","last_clean":"2023-10-06T03:29:38.627322+0000","last_became_active":"2023-10-06T03:29:38.623537+0000","last_became_peered":"2023-10-06T03:29:38.623537+0000","last_unstale":"2023-10-06T03:29:38.627322+0000","last_undegraded":"2023-10-06T03:29:38.627322+0000","last_fullsized":"2023-10-06T03:29:38.627322+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:37.570693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:37.570693+0000","last_clean_scrub_stamp":"2023-10-06T03:29:37.570693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T07:38:31.118869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1,6],"acting":[5,3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"23'8","reported_seq":21,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.609831+0000","last_change":"2023-10-06T03:29:33.538136+0000","last_active":"2023-10-06T03:29:38.609831+0000","last_peered":"2023-10-06T03:29:38.609831+0000","last_clean":"2023-10-06T03:29:38.609831+0000","last_became_active":"2023-10-06T03:29:33.537656+0000","last_became_peered":"2023-10-06T03:29:33.537656+0000","last_unstale":"2023-10-06T03:29:38.609831+0000","last_undegraded":"2023-10-06T03:29:38.609831+0000","last_fullsized":"2023-10-06T03:29:38.609831+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:32.495616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:32.495616+0000","last_clean_scrub_stamp":"2023-10-06T03:29:32.495616+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T07:23:17.430374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.592883+0000","last_change":"2023-10-06T03:29:34.546204+0000","last_active":"2023-10-06T03:29:38.592883+0000","last_peered":"2023-10-06T03:29:38.592883+0000","last_clean":"2023-10-06T03:29:38.592883+0000","last_became_active":"2023-10-06T03:29:34.545585+0000","last_became_peered":"2023-10-06T03:29:34.545585+0000","last_unstale":"2023-10-06T03:29:38.592883+0000","last_undegraded":"2023-10-06T03:29:38.592883+0000","last_fullsized":"2023-10-06T03:29:38.592883+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_clean_scrub_stamp":"2023-10-06T03:29:33.500929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T05:40:45.388611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.620710+0000","last_change":"2023-10-06T03:29:38.612955+0000","last_active":"2023-10-06T03:29:38.620710+0000","last_peered":"2023-10-06T03:29:38.620710+0000","last_clean":"2023-10-06T03:29:38.620710+0000","last_became_active":"2023-10-06T03:29:38.612404+0000","last_became_peered":"2023-10-06T03:29:38.612404+0000","last_unstale":"2023-10-06T03:29:38.620710+0000","last_undegraded":"2023-10-06T03:29:38.620710+0000","last_fullsized":"2023-10-06T03:29:38.620710+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:37.570693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:37.570693+0000","last_clean_scrub_stamp":"2023-10-06T03:29:37.570693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T08:25:07.069848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7,0],"acting":[3,1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.603401+0000","last_change":"2023-10-06T03:29:33.545324+0000","last_active":"2023-10-06T03:29:38.603401+0000","last_peered":"2023-10-06T03:29:38.603401+0000","last_clean":"2023-10-06T03:29:38.603401+0000","last_became_active":"2023-10-06T03:29:33.545029+0000","last_became_peered":"2023-10-06T03:29:33.545029+0000","last_unstale":"2023-10-06T03:29:38.603401+0000","last_undegraded":"2023-10-06T03:29:38.603401+0000","last_fullsized":"2023-10-06T03:29:38.603401+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:32.495616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:32.495616+0000","last_clean_scrub_stamp":"2023-10-06T03:29:32.495616+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T07:14:02.648208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.599807+0000","last_change":"2023-10-06T03:29:34.535716+0000","last_active":"2023-10-06T03:29:38.599807+0000","last_peered":"2023-10-06T03:29:38.599807+0000","last_clean":"2023-10-06T03:29:38.599807+0000","last_became_active":"2023-10-06T03:29:34.535107+0000","last_became_peered":"2023-10-06T03:29:34.535107+0000","last_unstale":"2023-10-06T03:29:38.599807+0000","last_undegraded":"2023-10-06T03:29:38.599807+0000","last_fullsized":"2023-10-06T03:29:38.599807+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_clean_scrub_stamp":"2023-10-06T03:29:33.500929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T11:19:26.387732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":14,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.591766+0000","last_change":"2023-10-06T03:29:33.541792+0000","last_active":"2023-10-06T03:29:38.591766+0000","last_peered":"2023-10-06T03:29:38.591766+0000","last_clean":"2023-10-06T03:29:38.591766+0000","last_became_active":"2023-10-06T03:29:33.535547+0000","last_became_peered":"2023-10-06T03:29:33.535547+0000","last_unstale":"2023-10-06T03:29:38.591766+0000","last_undegraded":"2023-10-06T03:29:38.591766+0000","last_fullsized":"2023-10-06T03:29:38.591766+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:32.495616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:32.495616+0000","last_clean_scrub_stamp":"2023-10-06T03:29:32.495616+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T07:55:07.787102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.602006+0000","last_change":"2023-10-06T03:29:34.540666+0000","last_active":"2023-10-06T03:29:38.602006+0000","last_peered":"2023-10-06T03:29:38.602006+0000","last_clean":"2023-10-06T03:29:38.602006+0000","last_became_active":"2023-10-06T03:29:34.540176+0000","last_became_peered":"2023-10-06T03:29:34.540176+0000","last_unstale":"2023-10-06T03:29:38.602006+0000","last_undegraded":"2023-10-06T03:29:38.602006+0000","last_fullsized":"2023-10-06T03:29:38.602006+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_clean_scrub_stamp":"2023-10-06T03:29:33.500929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T14:11:35.669872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.617535+0000","last_change":"2023-10-06T03:29:38.610899+0000","last_active":"2023-10-06T03:29:38.617535+0000","last_peered":"2023-10-06T03:29:38.617535+0000","last_clean":"2023-10-06T03:29:38.617535+0000","last_became_active":"2023-10-06T03:29:38.610494+0000","last_became_peered":"2023-10-06T03:29:38.610494+0000","last_unstale":"2023-10-06T03:29:38.617535+0000","last_undegraded":"2023-10-06T03:29:38.617535+0000","last_fullsized":"2023-10-06T03:29:38.617535+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:37.570693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:37.570693+0000","last_clean_scrub_stamp":"2023-10-06T03:29:37.570693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T08:19:47.139035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2,1],"acting":[3,7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"23'2","reported_seq":15,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.604109+0000","last_change":"2023-10-06T03:29:33.544328+0000","last_active":"2023-10-06T03:29:38.604109+0000","last_peered":"2023-10-06T03:29:38.604109+0000","last_clean":"2023-10-06T03:29:38.604109+0000","last_became_active":"2023-10-06T03:29:33.544010+0000","last_became_peered":"2023-10-06T03:29:33.544010+0000","last_unstale":"2023-10-06T03:29:38.604109+0000","last_undegraded":"2023-10-06T03:29:38.604109+0000","last_fullsized":"2023-10-06T03:29:38.604109+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:32.495616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:32.495616+0000","last_clean_scrub_stamp":"2023-10-06T03:29:32.495616+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T11:17:52.119337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.592119+0000","last_change":"2023-10-06T03:29:34.551765+0000","last_active":"2023-10-06T03:29:38.592119+0000","last_peered":"2023-10-06T03:29:38.592119+0000","last_clean":"2023-10-06T03:29:38.592119+0000","last_became_active":"2023-10-06T03:29:34.551437+0000","last_became_peered":"2023-10-06T03:29:34.551437+0000","last_unstale":"2023-10-06T03:29:38.592119+0000","last_undegraded":"2023-10-06T03:29:38.592119+0000","last_fullsized":"2023-10-06T03:29:38.592119+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_clean_scrub_stamp":"2023-10-06T03:29:33.500929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T11:47:20.603758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.629984+0000","last_change":"2023-10-06T03:29:38.626781+0000","last_active":"2023-10-06T03:29:38.629984+0000","last_peered":"2023-10-06T03:29:38.629984+0000","last_clean":"2023-10-06T03:29:38.629984+0000","last_became_active":"2023-10-06T03:29:38.625489+0000","last_became_peered":"2023-10-06T03:29:38.625489+0000","last_unstale":"2023-10-06T03:29:38.629984+0000","last_undegraded":"2023-10-06T03:29:38.629984+0000","last_fullsized":"2023-10-06T03:29:38.629984+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:37.570693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:37.570693+0000","last_clean_scrub_stamp":"2023-10-06T03:29:37.570693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T06:13:47.912662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3,0],"acting":[5,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.20","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.609684+0000","last_change":"2023-10-06T03:29:34.546226+0000","last_active":"2023-10-06T03:29:38.609684+0000","last_peered":"2023-10-06T03:29:38.609684+0000","last_clean":"2023-10-06T03:29:38.609684+0000","last_became_active":"2023-10-06T03:29:34.545400+0000","last_became_peered":"2023-10-06T03:29:34.545400+0000","last_unstale":"2023-10-06T03:29:38.609684+0000","last_undegraded":"2023-10-06T03:29:38.609684+0000","last_fullsized":"2023-10-06T03:29:38.609684+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_clean_scrub_stamp":"2023-10-06T03:29:33.500929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T07:55:08.792414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.21","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.588987+0000","last_change":"2023-10-06T03:29:34.551672+0000","last_active":"2023-10-06T03:29:38.588987+0000","last_peered":"2023-10-06T03:29:38.588987+0000","last_clean":"2023-10-06T03:29:38.588987+0000","last_became_active":"2023-10-06T03:29:34.551337+0000","last_became_peered":"2023-10-06T03:29:34.551337+0000","last_unstale":"2023-10-06T03:29:38.588987+0000","last_undegraded":"2023-10-06T03:29:38.588987+0000","last_fullsized":"2023-10-06T03:29:38.588987+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_clean_scrub_stamp":"2023-10-06T03:29:33.500929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T05:15:29.367282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.22","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.592542+0000","last_change":"2023-10-06T03:29:34.549749+0000","last_active":"2023-10-06T03:29:38.592542+0000","last_peered":"2023-10-06T03:29:38.592542+0000","last_clean":"2023-10-06T03:29:38.592542+0000","last_became_active":"2023-10-06T03:29:34.549468+0000","last_became_peered":"2023-10-06T03:29:34.549468+0000","last_unstale":"2023-10-06T03:29:38.592542+0000","last_undegraded":"2023-10-06T03:29:38.592542+0000","last_fullsized":"2023-10-06T03:29:38.592542+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_clean_scrub_stamp":"2023-10-06T03:29:33.500929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T14:11:35.669872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.23","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.590075+0000","last_change":"2023-10-06T03:29:34.545511+0000","last_active":"2023-10-06T03:29:38.590075+0000","last_peered":"2023-10-06T03:29:38.590075+0000","last_clean":"2023-10-06T03:29:38.590075+0000","last_became_active":"2023-10-06T03:29:34.545183+0000","last_became_peered":"2023-10-06T03:29:34.545183+0000","last_unstale":"2023-10-06T03:29:38.590075+0000","last_undegraded":"2023-10-06T03:29:38.590075+0000","last_fullsized":"2023-10-06T03:29:38.590075+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_clean_scrub_stamp":"2023-10-06T03:29:33.500929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T07:54:17.345077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.24","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.609286+0000","last_change":"2023-10-06T03:29:34.543928+0000","last_active":"2023-10-06T03:29:38.609286+0000","last_peered":"2023-10-06T03:29:38.609286+0000","last_clean":"2023-10-06T03:29:38.609286+0000","last_became_active":"2023-10-06T03:29:34.543604+0000","last_became_peered":"2023-10-06T03:29:34.543604+0000","last_unstale":"2023-10-06T03:29:38.609286+0000","last_undegraded":"2023-10-06T03:29:38.609286+0000","last_fullsized":"2023-10-06T03:29:38.609286+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_clean_scrub_stamp":"2023-10-06T03:29:33.500929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T09:15:40.932270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.25","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.601918+0000","last_change":"2023-10-06T03:29:34.540446+0000","last_active":"2023-10-06T03:29:38.601918+0000","last_peered":"2023-10-06T03:29:38.601918+0000","last_clean":"2023-10-06T03:29:38.601918+0000","last_became_active":"2023-10-06T03:29:34.539827+0000","last_became_peered":"2023-10-06T03:29:34.539827+0000","last_unstale":"2023-10-06T03:29:38.601918+0000","last_undegraded":"2023-10-06T03:29:38.601918+0000","last_fullsized":"2023-10-06T03:29:38.601918+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_clean_scrub_stamp":"2023-10-06T03:29:33.500929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T07:28:42.661588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.26","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.609415+0000","last_change":"2023-10-06T03:29:34.538237+0000","last_active":"2023-10-06T03:29:38.609415+0000","last_peered":"2023-10-06T03:29:38.609415+0000","last_clean":"2023-10-06T03:29:38.609415+0000","last_became_active":"2023-10-06T03:29:34.537636+0000","last_became_peered":"2023-10-06T03:29:34.537636+0000","last_unstale":"2023-10-06T03:29:38.609415+0000","last_undegraded":"2023-10-06T03:29:38.609415+0000","last_fullsized":"2023-10-06T03:29:38.609415+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_clean_scrub_stamp":"2023-10-06T03:29:33.500929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T05:09:24.657371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.27","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.608601+0000","last_change":"2023-10-06T03:29:34.549127+0000","last_active":"2023-10-06T03:29:38.608601+0000","last_peered":"2023-10-06T03:29:38.608601+0000","last_clean":"2023-10-06T03:29:38.608601+0000","last_became_active":"2023-10-06T03:29:34.548839+0000","last_became_peered":"2023-10-06T03:29:34.548839+0000","last_unstale":"2023-10-06T03:29:38.608601+0000","last_undegraded":"2023-10-06T03:29:38.608601+0000","last_fullsized":"2023-10-06T03:29:38.608601+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_clean_scrub_stamp":"2023-10-06T03:29:33.500929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T08:03:23.482409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.28","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.603462+0000","last_change":"2023-10-06T03:29:34.544177+0000","last_active":"2023-10-06T03:29:38.603462+0000","last_peered":"2023-10-06T03:29:38.603462+0000","last_clean":"2023-10-06T03:29:38.603462+0000","last_became_active":"2023-10-06T03:29:34.543872+0000","last_became_peered":"2023-10-06T03:29:34.543872+0000","last_unstale":"2023-10-06T03:29:38.603462+0000","last_undegraded":"2023-10-06T03:29:38.603462+0000","last_fullsized":"2023-10-06T03:29:38.603462+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_clean_scrub_stamp":"2023-10-06T03:29:33.500929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T06:09:15.729486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.29","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.588861+0000","last_change":"2023-10-06T03:29:34.540282+0000","last_active":"2023-10-06T03:29:38.588861+0000","last_peered":"2023-10-06T03:29:38.588861+0000","last_clean":"2023-10-06T03:29:38.588861+0000","last_became_active":"2023-10-06T03:29:34.539988+0000","last_became_peered":"2023-10-06T03:29:34.539988+0000","last_unstale":"2023-10-06T03:29:38.588861+0000","last_undegraded":"2023-10-06T03:29:38.588861+0000","last_fullsized":"2023-10-06T03:29:38.588861+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_clean_scrub_stamp":"2023-10-06T03:29:33.500929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T07:36:17.771693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.2a","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.591640+0000","last_change":"2023-10-06T03:29:34.547360+0000","last_active":"2023-10-06T03:29:38.591640+0000","last_peered":"2023-10-06T03:29:38.591640+0000","last_clean":"2023-10-06T03:29:38.591640+0000","last_became_active":"2023-10-06T03:29:34.547086+0000","last_became_peered":"2023-10-06T03:29:34.547086+0000","last_unstale":"2023-10-06T03:29:38.591640+0000","last_undegraded":"2023-10-06T03:29:38.591640+0000","last_fullsized":"2023-10-06T03:29:38.591640+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_clean_scrub_stamp":"2023-10-06T03:29:33.500929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T07:25:41.452975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2b","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.604010+0000","last_change":"2023-10-06T03:29:34.544346+0000","last_active":"2023-10-06T03:29:38.604010+0000","last_peered":"2023-10-06T03:29:38.604010+0000","last_clean":"2023-10-06T03:29:38.604010+0000","last_became_active":"2023-10-06T03:29:34.543976+0000","last_became_peered":"2023-10-06T03:29:34.543976+0000","last_unstale":"2023-10-06T03:29:38.604010+0000","last_undegraded":"2023-10-06T03:29:38.604010+0000","last_fullsized":"2023-10-06T03:29:38.604010+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_clean_scrub_stamp":"2023-10-06T03:29:33.500929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T14:11:35.669872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.2c","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.593246+0000","last_change":"2023-10-06T03:29:34.549013+0000","last_active":"2023-10-06T03:29:38.593246+0000","last_peered":"2023-10-06T03:29:38.593246+0000","last_clean":"2023-10-06T03:29:38.593246+0000","last_became_active":"2023-10-06T03:29:34.548707+0000","last_became_peered":"2023-10-06T03:29:34.548707+0000","last_unstale":"2023-10-06T03:29:38.593246+0000","last_undegraded":"2023-10-06T03:29:38.593246+0000","last_fullsized":"2023-10-06T03:29:38.593246+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_clean_scrub_stamp":"2023-10-06T03:29:33.500929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T14:34:26.405025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2d","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.598606+0000","last_change":"2023-10-06T03:29:34.545639+0000","last_active":"2023-10-06T03:29:38.598606+0000","last_peered":"2023-10-06T03:29:38.598606+0000","last_clean":"2023-10-06T03:29:38.598606+0000","last_became_active":"2023-10-06T03:29:34.545316+0000","last_became_peered":"2023-10-06T03:29:34.545316+0000","last_unstale":"2023-10-06T03:29:38.598606+0000","last_undegraded":"2023-10-06T03:29:38.598606+0000","last_fullsized":"2023-10-06T03:29:38.598606+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_clean_scrub_stamp":"2023-10-06T03:29:33.500929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T11:58:54.143086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2e","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.592479+0000","last_change":"2023-10-06T03:29:34.532132+0000","last_active":"2023-10-06T03:29:38.592479+0000","last_peered":"2023-10-06T03:29:38.592479+0000","last_clean":"2023-10-06T03:29:38.592479+0000","last_became_active":"2023-10-06T03:29:34.531862+0000","last_became_peered":"2023-10-06T03:29:34.531862+0000","last_unstale":"2023-10-06T03:29:38.592479+0000","last_undegraded":"2023-10-06T03:29:38.592479+0000","last_fullsized":"2023-10-06T03:29:38.592479+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_clean_scrub_stamp":"2023-10-06T03:29:33.500929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T07:25:41.452975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2f","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.592223+0000","last_change":"2023-10-06T03:29:34.540499+0000","last_active":"2023-10-06T03:29:38.592223+0000","last_peered":"2023-10-06T03:29:38.592223+0000","last_clean":"2023-10-06T03:29:38.592223+0000","last_became_active":"2023-10-06T03:29:34.539154+0000","last_became_peered":"2023-10-06T03:29:34.539154+0000","last_unstale":"2023-10-06T03:29:38.592223+0000","last_undegraded":"2023-10-06T03:29:38.592223+0000","last_fullsized":"2023-10-06T03:29:38.592223+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_clean_scrub_stamp":"2023-10-06T03:29:33.500929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T15:20:45.655411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.30","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.594589+0000","last_change":"2023-10-06T03:29:34.532489+0000","last_active":"2023-10-06T03:29:38.594589+0000","last_peered":"2023-10-06T03:29:38.594589+0000","last_clean":"2023-10-06T03:29:38.594589+0000","last_became_active":"2023-10-06T03:29:34.532153+0000","last_became_peered":"2023-10-06T03:29:34.532153+0000","last_unstale":"2023-10-06T03:29:38.594589+0000","last_undegraded":"2023-10-06T03:29:38.594589+0000","last_fullsized":"2023-10-06T03:29:38.594589+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_clean_scrub_stamp":"2023-10-06T03:29:33.500929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T08:05:18.552294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.31","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.603291+0000","last_change":"2023-10-06T03:29:34.551157+0000","last_active":"2023-10-06T03:29:38.603291+0000","last_peered":"2023-10-06T03:29:38.603291+0000","last_clean":"2023-10-06T03:29:38.603291+0000","last_became_active":"2023-10-06T03:29:34.550852+0000","last_became_peered":"2023-10-06T03:29:34.550852+0000","last_unstale":"2023-10-06T03:29:38.603291+0000","last_undegraded":"2023-10-06T03:29:38.603291+0000","last_fullsized":"2023-10-06T03:29:38.603291+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_clean_scrub_stamp":"2023-10-06T03:29:33.500929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T08:29:43.429905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.32","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.600883+0000","last_change":"2023-10-06T03:29:34.539103+0000","last_active":"2023-10-06T03:29:38.600883+0000","last_peered":"2023-10-06T03:29:38.600883+0000","last_clean":"2023-10-06T03:29:38.600883+0000","last_became_active":"2023-10-06T03:29:34.538817+0000","last_became_peered":"2023-10-06T03:29:34.538817+0000","last_unstale":"2023-10-06T03:29:38.600883+0000","last_undegraded":"2023-10-06T03:29:38.600883+0000","last_fullsized":"2023-10-06T03:29:38.600883+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_clean_scrub_stamp":"2023-10-06T03:29:33.500929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T07:07:03.705326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.33","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.599036+0000","last_change":"2023-10-06T03:29:34.538671+0000","last_active":"2023-10-06T03:29:38.599036+0000","last_peered":"2023-10-06T03:29:38.599036+0000","last_clean":"2023-10-06T03:29:38.599036+0000","last_became_active":"2023-10-06T03:29:34.538375+0000","last_became_peered":"2023-10-06T03:29:34.538375+0000","last_unstale":"2023-10-06T03:29:38.599036+0000","last_undegraded":"2023-10-06T03:29:38.599036+0000","last_fullsized":"2023-10-06T03:29:38.599036+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_clean_scrub_stamp":"2023-10-06T03:29:33.500929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T07:14:03.653521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.34","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.600214+0000","last_change":"2023-10-06T03:29:34.535139+0000","last_active":"2023-10-06T03:29:38.600214+0000","last_peered":"2023-10-06T03:29:38.600214+0000","last_clean":"2023-10-06T03:29:38.600214+0000","last_became_active":"2023-10-06T03:29:34.534756+0000","last_became_peered":"2023-10-06T03:29:34.534756+0000","last_unstale":"2023-10-06T03:29:38.600214+0000","last_undegraded":"2023-10-06T03:29:38.600214+0000","last_fullsized":"2023-10-06T03:29:38.600214+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_clean_scrub_stamp":"2023-10-06T03:29:33.500929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T14:51:50.895276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.35","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.592690+0000","last_change":"2023-10-06T03:29:34.550626+0000","last_active":"2023-10-06T03:29:38.592690+0000","last_peered":"2023-10-06T03:29:38.592690+0000","last_clean":"2023-10-06T03:29:38.592690+0000","last_became_active":"2023-10-06T03:29:34.550289+0000","last_became_peered":"2023-10-06T03:29:34.550289+0000","last_unstale":"2023-10-06T03:29:38.592690+0000","last_undegraded":"2023-10-06T03:29:38.592690+0000","last_fullsized":"2023-10-06T03:29:38.592690+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_clean_scrub_stamp":"2023-10-06T03:29:33.500929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T12:44:47.758971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.36","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.602276+0000","last_change":"2023-10-06T03:29:34.541344+0000","last_active":"2023-10-06T03:29:38.602276+0000","last_peered":"2023-10-06T03:29:38.602276+0000","last_clean":"2023-10-06T03:29:38.602276+0000","last_became_active":"2023-10-06T03:29:34.540994+0000","last_became_peered":"2023-10-06T03:29:34.540994+0000","last_unstale":"2023-10-06T03:29:38.602276+0000","last_undegraded":"2023-10-06T03:29:38.602276+0000","last_fullsized":"2023-10-06T03:29:38.602276+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_clean_scrub_stamp":"2023-10-06T03:29:33.500929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T05:31:20.131568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.37","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.592106+0000","last_change":"2023-10-06T03:29:34.540530+0000","last_active":"2023-10-06T03:29:38.592106+0000","last_peered":"2023-10-06T03:29:38.592106+0000","last_clean":"2023-10-06T03:29:38.592106+0000","last_became_active":"2023-10-06T03:29:34.539518+0000","last_became_peered":"2023-10-06T03:29:34.539518+0000","last_unstale":"2023-10-06T03:29:38.592106+0000","last_undegraded":"2023-10-06T03:29:38.592106+0000","last_fullsized":"2023-10-06T03:29:38.592106+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_clean_scrub_stamp":"2023-10-06T03:29:33.500929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T14:34:26.405025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.38","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.600763+0000","last_change":"2023-10-06T03:29:34.537333+0000","last_active":"2023-10-06T03:29:38.600763+0000","last_peered":"2023-10-06T03:29:38.600763+0000","last_clean":"2023-10-06T03:29:38.600763+0000","last_became_active":"2023-10-06T03:29:34.536851+0000","last_became_peered":"2023-10-06T03:29:34.536851+0000","last_unstale":"2023-10-06T03:29:38.600763+0000","last_undegraded":"2023-10-06T03:29:38.600763+0000","last_fullsized":"2023-10-06T03:29:38.600763+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_clean_scrub_stamp":"2023-10-06T03:29:33.500929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T14:20:50.168977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.39","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.603186+0000","last_change":"2023-10-06T03:29:34.551594+0000","last_active":"2023-10-06T03:29:38.603186+0000","last_peered":"2023-10-06T03:29:38.603186+0000","last_clean":"2023-10-06T03:29:38.603186+0000","last_became_active":"2023-10-06T03:29:34.551142+0000","last_became_peered":"2023-10-06T03:29:34.551142+0000","last_unstale":"2023-10-06T03:29:38.603186+0000","last_undegraded":"2023-10-06T03:29:38.603186+0000","last_fullsized":"2023-10-06T03:29:38.603186+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_clean_scrub_stamp":"2023-10-06T03:29:33.500929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T07:28:42.661588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3a","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.592149+0000","last_change":"2023-10-06T03:29:34.542011+0000","last_active":"2023-10-06T03:29:38.592149+0000","last_peered":"2023-10-06T03:29:38.592149+0000","last_clean":"2023-10-06T03:29:38.592149+0000","last_became_active":"2023-10-06T03:29:34.540577+0000","last_became_peered":"2023-10-06T03:29:34.540577+0000","last_unstale":"2023-10-06T03:29:38.592149+0000","last_undegraded":"2023-10-06T03:29:38.592149+0000","last_fullsized":"2023-10-06T03:29:38.592149+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_clean_scrub_stamp":"2023-10-06T03:29:33.500929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T12:26:16.082642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.3b","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.593039+0000","last_change":"2023-10-06T03:29:34.538708+0000","last_active":"2023-10-06T03:29:38.593039+0000","last_peered":"2023-10-06T03:29:38.593039+0000","last_clean":"2023-10-06T03:29:38.593039+0000","last_became_active":"2023-10-06T03:29:34.538419+0000","last_became_peered":"2023-10-06T03:29:34.538419+0000","last_unstale":"2023-10-06T03:29:38.593039+0000","last_undegraded":"2023-10-06T03:29:38.593039+0000","last_fullsized":"2023-10-06T03:29:38.593039+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_clean_scrub_stamp":"2023-10-06T03:29:33.500929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T03:45:19.939590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3c","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.609565+0000","last_change":"2023-10-06T03:29:34.546290+0000","last_active":"2023-10-06T03:29:38.609565+0000","last_peered":"2023-10-06T03:29:38.609565+0000","last_clean":"2023-10-06T03:29:38.609565+0000","last_became_active":"2023-10-06T03:29:34.545947+0000","last_became_peered":"2023-10-06T03:29:34.545947+0000","last_unstale":"2023-10-06T03:29:38.609565+0000","last_undegraded":"2023-10-06T03:29:38.609565+0000","last_fullsized":"2023-10-06T03:29:38.609565+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_clean_scrub_stamp":"2023-10-06T03:29:33.500929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T12:25:11.771364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3d","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.592442+0000","last_change":"2023-10-06T03:29:34.541409+0000","last_active":"2023-10-06T03:29:38.592442+0000","last_peered":"2023-10-06T03:29:38.592442+0000","last_clean":"2023-10-06T03:29:38.592442+0000","last_became_active":"2023-10-06T03:29:34.541111+0000","last_became_peered":"2023-10-06T03:29:34.541111+0000","last_unstale":"2023-10-06T03:29:38.592442+0000","last_undegraded":"2023-10-06T03:29:38.592442+0000","last_fullsized":"2023-10-06T03:29:38.592442+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_clean_scrub_stamp":"2023-10-06T03:29:33.500929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T04:15:27.116813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.3e","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.602155+0000","last_change":"2023-10-06T03:29:34.549044+0000","last_active":"2023-10-06T03:29:38.602155+0000","last_peered":"2023-10-06T03:29:38.602155+0000","last_clean":"2023-10-06T03:29:38.602155+0000","last_became_active":"2023-10-06T03:29:34.548691+0000","last_became_peered":"2023-10-06T03:29:34.548691+0000","last_unstale":"2023-10-06T03:29:38.602155+0000","last_undegraded":"2023-10-06T03:29:38.602155+0000","last_fullsized":"2023-10-06T03:29:38.602155+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_clean_scrub_stamp":"2023-10-06T03:29:33.500929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T06:09:15.729486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.3f","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.598391+0000","last_change":"2023-10-06T03:29:34.538647+0000","last_active":"2023-10-06T03:29:38.598391+0000","last_peered":"2023-10-06T03:29:38.598391+0000","last_clean":"2023-10-06T03:29:38.598391+0000","last_became_active":"2023-10-06T03:29:34.538343+0000","last_became_peered":"2023-10-06T03:29:34.538343+0000","last_unstale":"2023-10-06T03:29:38.598391+0000","last_undegraded":"2023-10-06T03:29:38.598391+0000","last_fullsized":"2023-10-06T03:29:38.598391+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_clean_scrub_stamp":"2023-10-06T03:29:33.500929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T12:25:11.771364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":4,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":4484,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":48,"num_write_kb":17,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":8968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":64,"acting":64,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":6,"up_from":16,"seq":68719476746,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297572,"kb_used_data":284,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074268,"statfs":{"total":96636764160,"available":96332050432,"internally_reserved":0,"allocated":290816,"data_stored":120026,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0549999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97399999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92500000000000004}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.014}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2869999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.333}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3160000000000001}]}]},{"osd":5,"up_from":16,"seq":68719476746,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297580,"kb_used_data":292,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074260,"statfs":{"total":96636764160,"available":96332042240,"internally_reserved":0,"allocated":299008,"data_stored":120650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59699999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0720000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.99299999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85299999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81499999999999995}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.171}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.153}]}]},{"osd":4,"up_from":16,"seq":68719476745,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297592,"kb_used_data":304,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074248,"statfs":{"total":96636764160,"available":96332029952,"internally_reserved":0,"allocated":311296,"data_stored":123796,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64300000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59499999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1910000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.99199999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.91300000000000003}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95999999999999996}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2609999999999999}]}]},{"osd":7,"up_from":15,"seq":64424509449,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298180,"kb_used_data":892,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94073660,"statfs":{"total":96636764160,"available":96331427840,"internally_reserved":0,"allocated":913408,"data_stored":714324,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63100000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.875}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72599999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84699999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83199999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.139}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1759999999999999}]}]},{"osd":0,"up_from":15,"seq":64424509450,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302252,"kb_used_data":868,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94069588,"statfs":{"total":96636764160,"available":96327258112,"internally_reserved":0,"allocated":888832,"data_stored":710948,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0029999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.042}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.90200000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80700000000000005}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.026}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.427}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0960000000000001}]}]},{"osd":3,"up_from":14,"seq":60129542153,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":301672,"kb_used_data":288,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94070168,"statfs":{"total":96636764160,"available":96327852032,"internally_reserved":0,"allocated":294912,"data_stored":120116,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49399999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51000000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76200000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81399999999999995}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92400000000000004}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0660000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1479999999999999}]}]},{"osd":2,"up_from":14,"seq":60129542152,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":301668,"kb_used_data":284,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94070172,"statfs":{"total":96636764160,"available":96327856128,"internally_reserved":0,"allocated":290816,"data_stored":120026,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67200000000000004}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76500000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.872}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.077}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0129999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79300000000000004}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.496}]}]},{"osd":1,"up_from":14,"seq":60129542152,"num_pgs":51,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":301668,"kb_used_data":284,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94070172,"statfs":{"total":96636764160,"available":96327856128,"internally_reserved":0,"allocated":290816,"data_stored":120026,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56000000000000005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40999999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.628}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.75800000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82899999999999996}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69199999999999995}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86099999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":554,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":3770,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":3930,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-06T03:29:52.277 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-06T03:29:52.378 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:52.373+0000 7f8c4ea5c700 1 Processor -- start 2023-10-06T03:29:52.378 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:52.373+0000 7f8c4ea5c700 1 -- start start 2023-10-06T03:29:52.379 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:52.373+0000 7f8c4ea5c700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8c4815d420 0x7f8c480c7b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:52.379 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:52.373+0000 7f8c4ea5c700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f8c48069d70 0x7f8c480ca270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:52.379 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:52.373+0000 7f8c4ea5c700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c4806aef0 con 0x7f8c4815d7f0 2023-10-06T03:29:52.379 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:52.373+0000 7f8c4ea5c700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8c4806c7d0 con 0x7f8c48069d70 2023-10-06T03:29:52.379 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:52.373+0000 7f8c4ea5c700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c4806a2e0 con 0x7f8c4815d420 2023-10-06T03:29:52.379 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:52.373+0000 7f8c47fff700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8c4815d420 0x7f8c480c7b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:52.379 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:52.373+0000 7f8c47fff700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8c4815d420 0x7f8c480c7b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3301/0 says I am v2:172.21.15.110:34720/0 (socket says 172.21.15.110:34720) 2023-10-06T03:29:52.380 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:52.373+0000 7f8c47fff700 1 -- 172.21.15.110:0/2520608091 learned_addr learned my addr 172.21.15.110:0/2520608091 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:29:52.380 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:52.377+0000 7f8c3f7fe700 1 --2- 172.21.15.110:0/2520608091 >> v2:172.21.15.192:3300/0 conn(0x7f8c48069d70 0x7f8c480ca270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:52.381 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:52.377+0000 7f8c47fff700 1 -- 172.21.15.110:0/2520608091 >> v2:172.21.15.192:3300/0 conn(0x7f8c48069d70 msgr2=0x7f8c480ca270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:29:52.381 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:52.377+0000 7f8c47fff700 1 --2- 172.21.15.110:0/2520608091 >> v2:172.21.15.192:3300/0 conn(0x7f8c48069d70 0x7f8c480ca270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:52.381 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:52.377+0000 7f8c47fff700 1 -- 172.21.15.110:0/2520608091 >> v1:172.21.15.192:6789/0 conn(0x7f8c4815d7f0 legacy=0x7f8c48161250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:52.381 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:52.377+0000 7f8c477fe700 1 -- 172.21.15.110:0/2520608091 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 50630809 0 0) 0x7f8c4806aef0 con 0x7f8c4815d7f0 2023-10-06T03:29:52.381 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:52.377+0000 7f8c47fff700 1 -- 172.21.15.110:0/2520608091 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c4806a5c0 con 0x7f8c4815d420 2023-10-06T03:29:52.381 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:52.377+0000 7f8c47fff700 1 --2- 172.21.15.110:0/2520608091 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8c4815d420 0x7f8c480c7b90 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f8c380099b0 tx=0x7f8c38009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6d55e36d8e367e8c server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:29:52.381 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:52.377+0000 7f8c477fe700 1 -- 172.21.15.110:0/2520608091 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c38009ea0 con 0x7f8c4815d420 2023-10-06T03:29:52.382 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:52.377+0000 7f8c477fe700 1 -- 172.21.15.110:0/2520608091 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c3800b860 con 0x7f8c4815d420 2023-10-06T03:29:52.382 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:52.377+0000 7f8c477fe700 1 -- 172.21.15.110:0/2520608091 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c3800bc00 con 0x7f8c4815d420 2023-10-06T03:29:52.382 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:52.377+0000 7f8c4ea5c700 1 -- 172.21.15.110:0/2520608091 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8c4815d420 msgr2=0x7f8c480c7b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:52.382 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:52.377+0000 7f8c4ea5c700 1 --2- 172.21.15.110:0/2520608091 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8c4815d420 0x7f8c480c7b90 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f8c380099b0 tx=0x7f8c38009250 comp rx=0 tx=0).stop 2023-10-06T03:29:52.382 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:52.377+0000 7f8c4ea5c700 1 -- 172.21.15.110:0/2520608091 shutdown_connections 2023-10-06T03:29:52.382 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:52.377+0000 7f8c4ea5c700 1 --2- 172.21.15.110:0/2520608091 >> v2:172.21.15.192:3300/0 conn(0x7f8c48069d70 0x7f8c480ca270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:52.382 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:52.377+0000 7f8c4ea5c700 1 --2- 172.21.15.110:0/2520608091 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8c4815d420 0x7f8c480c7b90 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:52.382 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:52.377+0000 7f8c4ea5c700 1 -- 172.21.15.110:0/2520608091 >> 172.21.15.110:0/2520608091 conn(0x7f8c480cbd00 msgr2=0x7f8c480cc100 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:29:52.382 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:52.377+0000 7f8c4ea5c700 1 -- 172.21.15.110:0/2520608091 shutdown_connections 2023-10-06T03:29:52.382 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:52.377+0000 7f8c4ea5c700 1 -- 172.21.15.110:0/2520608091 wait complete. 2023-10-06T03:29:52.384 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:52.377+0000 7f8c4ea5c700 1 Processor -- start 2023-10-06T03:29:52.384 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:52.377+0000 7f8c4ea5c700 1 -- start start 2023-10-06T03:29:52.384 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:52.377+0000 7f8c4ea5c700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8c48069d70 0x7f8c48148160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:52.384 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:52.377+0000 7f8c4ea5c700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8c4815d420 0x7f8c4815b8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:52.384 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:52.377+0000 7f8c4ea5c700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8c4815d7f0 0x7f8c4815bdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:52.385 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:52.377+0000 7f8c4ea5c700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c4815e030 con 0x7f8c48069d70 2023-10-06T03:29:52.385 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:52.377+0000 7f8c4ea5c700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c4816e5e0 con 0x7f8c4815d7f0 2023-10-06T03:29:52.385 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:52.377+0000 7f8c4ea5c700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c481486a0 con 0x7f8c4815d420 2023-10-06T03:29:52.385 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:52.381+0000 7f8c47fff700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8c48069d70 0x7f8c48148160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:52.385 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:52.381+0000 7f8c47fff700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8c48069d70 0x7f8c48148160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:47650/0 (socket says 172.21.15.110:47650) 2023-10-06T03:29:52.385 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:52.381+0000 7f8c47fff700 1 -- 172.21.15.110:0/2992266786 learned_addr learned my addr 172.21.15.110:0/2992266786 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:29:52.385 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:52.381+0000 7f8c3f7fe700 1 --2- 172.21.15.110:0/2992266786 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8c4815d420 0x7f8c4815b8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:52.385 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:52.381+0000 7f8c4cff9700 1 --2- 172.21.15.110:0/2992266786 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8c4815d7f0 0x7f8c4815bdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:52.385 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:52.381+0000 7f8c47fff700 1 -- 172.21.15.110:0/2992266786 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8c4815d420 msgr2=0x7f8c4815b8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:52.385 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:52.381+0000 7f8c47fff700 1 --2- 172.21.15.110:0/2992266786 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8c4815d420 0x7f8c4815b8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:52.385 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:52.381+0000 7f8c47fff700 1 -- 172.21.15.110:0/2992266786 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8c4815d7f0 msgr2=0x7f8c4815bdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:52.385 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:52.381+0000 7f8c47fff700 1 --2- 172.21.15.110:0/2992266786 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8c4815d7f0 0x7f8c4815bdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:52.386 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:52.381+0000 7f8c47fff700 1 -- 172.21.15.110:0/2992266786 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c380094f0 con 0x7f8c48069d70 2023-10-06T03:29:52.386 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:52.381+0000 7f8c47fff700 1 --2- 172.21.15.110:0/2992266786 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8c48069d70 0x7f8c48148160 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f8c4806a420 tx=0x7f8c38017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:29:52.386 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:52.381+0000 7f8c44ff9700 1 -- 172.21.15.110:0/2992266786 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c3800bbe0 con 0x7f8c48069d70 2023-10-06T03:29:52.386 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:52.381+0000 7f8c44ff9700 1 -- 172.21.15.110:0/2992266786 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c38011420 con 0x7f8c48069d70 2023-10-06T03:29:52.386 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:52.381+0000 7f8c4ea5c700 1 -- 172.21.15.110:0/2992266786 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c4816e300 con 0x7f8c48069d70 2023-10-06T03:29:52.386 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:52.381+0000 7f8c44ff9700 1 -- 172.21.15.110:0/2992266786 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c380117c0 con 0x7f8c48069d70 2023-10-06T03:29:52.386 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:52.381+0000 7f8c4ea5c700 1 -- 172.21.15.110:0/2992266786 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c4816ec20 con 0x7f8c48069d70 2023-10-06T03:29:52.386 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:52.381+0000 7f8c4ea5c700 1 -- 172.21.15.110:0/2992266786 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c4806ff60 con 0x7f8c48069d70 2023-10-06T03:29:52.388 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:52.385+0000 7f8c44ff9700 1 -- 172.21.15.110:0/2992266786 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7f8c38011b30 con 0x7f8c48069d70 2023-10-06T03:29:52.388 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:52.385+0000 7f8c44ff9700 1 --2- 172.21.15.110:0/2992266786 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f8c28060550 0x7f8c28062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:52.388 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:52.385+0000 7f8c44ff9700 1 -- 172.21.15.110:0/2992266786 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6809+0+0 (secure 0 0 0) 0x7f8c3807e590 con 0x7f8c48069d70 2023-10-06T03:29:52.388 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:52.385+0000 7f8c3f7fe700 1 --2- 172.21.15.110:0/2992266786 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f8c28060550 0x7f8c28062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:52.389 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:52.385+0000 7f8c3f7fe700 1 --2- 172.21.15.110:0/2992266786 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f8c28060550 0x7f8c28062a00 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f8c4816b230 tx=0x7f8c30008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:29:52.395 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:52.393+0000 7f8c44ff9700 1 -- 172.21.15.110:0/2992266786 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f8c3804e730 con 0x7f8c48069d70 2023-10-06T03:29:52.544 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:52.541+0000 7f8c4ea5c700 1 -- 172.21.15.110:0/2992266786 --> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f8c480cb120 con 0x7f8c28060550 2023-10-06T03:29:52.562 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:52.557+0000 7f8c44ff9700 1 -- 172.21.15.110:0/2992266786 <== mgr.4110 v2:172.21.15.110:6832/33268 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+367265 (secure 0 0 0) 0x7f8c480cb120 con 0x7f8c28060550 2023-10-06T03:29:52.568 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-10-06T03:29:52.571 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:52.565+0000 7f8c4ea5c700 1 -- 172.21.15.110:0/2992266786 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f8c28060550 msgr2=0x7f8c28062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:52.571 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:52.565+0000 7f8c4ea5c700 1 --2- 172.21.15.110:0/2992266786 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f8c28060550 0x7f8c28062a00 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f8c4816b230 tx=0x7f8c30008040 comp rx=0 tx=0).stop 2023-10-06T03:29:52.571 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:52.565+0000 7f8c4ea5c700 1 -- 172.21.15.110:0/2992266786 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8c48069d70 msgr2=0x7f8c48148160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:52.571 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:52.565+0000 7f8c4ea5c700 1 --2- 172.21.15.110:0/2992266786 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8c48069d70 0x7f8c48148160 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f8c4806a420 tx=0x7f8c38017000 comp rx=0 tx=0).stop 2023-10-06T03:29:52.571 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:52.565+0000 7f8c4ea5c700 1 -- 172.21.15.110:0/2992266786 shutdown_connections 2023-10-06T03:29:52.572 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:52.565+0000 7f8c4ea5c700 1 --2- 172.21.15.110:0/2992266786 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f8c28060550 0x7f8c28062a00 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:52.572 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:52.565+0000 7f8c4ea5c700 1 --2- 172.21.15.110:0/2992266786 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8c4815d7f0 0x7f8c4815bdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:52.572 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:52.565+0000 7f8c4ea5c700 1 --2- 172.21.15.110:0/2992266786 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8c4815d420 0x7f8c4815b8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:52.572 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:52.565+0000 7f8c4ea5c700 1 --2- 172.21.15.110:0/2992266786 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8c48069d70 0x7f8c48148160 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:52.572 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:52.565+0000 7f8c4ea5c700 1 -- 172.21.15.110:0/2992266786 >> 172.21.15.110:0/2992266786 conn(0x7f8c480cbd00 msgr2=0x7f8c481611e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:29:52.572 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:52.565+0000 7f8c4ea5c700 1 -- 172.21.15.110:0/2992266786 shutdown_connections 2023-10-06T03:29:52.572 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:52.565+0000 7f8c4ea5c700 1 -- 172.21.15.110:0/2992266786 wait complete. 2023-10-06T03:29:52.572 INFO:teuthology.orchestra.run.smithi110.stderr:dumped all 2023-10-06T03:29:52.584 INFO:teuthology.orchestra.run.smithi110.stdout:{"pg_ready":true,"pg_map":{"version":36,"stamp":"2023-10-06T03:29:50.653327+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":594852,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":207,"num_write_kb":2433,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":159,"ondisk_log_size":159,"up":338,"acting":338,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":338,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2398184,"kb_used_data":3496,"kb_used_omap":0,"kb_used_meta":2394624,"kb_avail":752576536,"statfs":{"total":773094113280,"available":770638372864,"internally_reserved":0,"allocated":3579904,"data_stored":2149912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2452094976},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2178,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"11.037086"},"pg_stats":[{"pgid":"5.19","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.624659+0000","last_change":"2023-10-06T03:29:38.620919+0000","last_active":"2023-10-06T03:29:38.624659+0000","last_peered":"2023-10-06T03:29:38.624659+0000","last_clean":"2023-10-06T03:29:38.624659+0000","last_became_active":"2023-10-06T03:29:38.620293+0000","last_became_peered":"2023-10-06T03:29:38.620293+0000","last_unstale":"2023-10-06T03:29:38.624659+0000","last_undegraded":"2023-10-06T03:29:38.624659+0000","last_fullsized":"2023-10-06T03:29:38.624659+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:37.570693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:37.570693+0000","last_clean_scrub_stamp":"2023-10-06T03:29:37.570693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T13:27:03.828118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7,2],"acting":[1,5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"23'4","reported_seq":18,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.592373+0000","last_change":"2023-10-06T03:29:33.535641+0000","last_active":"2023-10-06T03:29:38.592373+0000","last_peered":"2023-10-06T03:29:38.592373+0000","last_clean":"2023-10-06T03:29:38.592373+0000","last_became_active":"2023-10-06T03:29:33.534410+0000","last_became_peered":"2023-10-06T03:29:33.534410+0000","last_unstale":"2023-10-06T03:29:38.592373+0000","last_undegraded":"2023-10-06T03:29:38.592373+0000","last_fullsized":"2023-10-06T03:29:38.592373+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:32.495616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:32.495616+0000","last_clean_scrub_stamp":"2023-10-06T03:29:32.495616+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T14:51:49.889964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":554,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.594781+0000","last_change":"2023-10-06T03:29:34.546243+0000","last_active":"2023-10-06T03:29:38.594781+0000","last_peered":"2023-10-06T03:29:38.594781+0000","last_clean":"2023-10-06T03:29:38.594781+0000","last_became_active":"2023-10-06T03:29:34.545923+0000","last_became_peered":"2023-10-06T03:29:34.545923+0000","last_unstale":"2023-10-06T03:29:38.594781+0000","last_undegraded":"2023-10-06T03:29:38.594781+0000","last_fullsized":"2023-10-06T03:29:38.594781+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_clean_scrub_stamp":"2023-10-06T03:29:33.500929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T15:20:45.655411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.617434+0000","last_change":"2023-10-06T03:29:38.608716+0000","last_active":"2023-10-06T03:29:38.617434+0000","last_peered":"2023-10-06T03:29:38.617434+0000","last_clean":"2023-10-06T03:29:38.617434+0000","last_became_active":"2023-10-06T03:29:38.607714+0000","last_became_peered":"2023-10-06T03:29:38.607714+0000","last_unstale":"2023-10-06T03:29:38.617434+0000","last_undegraded":"2023-10-06T03:29:38.617434+0000","last_fullsized":"2023-10-06T03:29:38.617434+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:37.570693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:37.570693+0000","last_clean_scrub_stamp":"2023-10-06T03:29:37.570693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T07:11:39.919440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1,2],"acting":[4,6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"23'1","reported_seq":15,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.591195+0000","last_change":"2023-10-06T03:29:33.538145+0000","last_active":"2023-10-06T03:29:38.591195+0000","last_peered":"2023-10-06T03:29:38.591195+0000","last_clean":"2023-10-06T03:29:38.591195+0000","last_became_active":"2023-10-06T03:29:33.536028+0000","last_became_peered":"2023-10-06T03:29:33.536028+0000","last_unstale":"2023-10-06T03:29:38.591195+0000","last_undegraded":"2023-10-06T03:29:38.591195+0000","last_fullsized":"2023-10-06T03:29:38.591195+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:32.495616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:32.495616+0000","last_clean_scrub_stamp":"2023-10-06T03:29:32.495616+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T06:15:00.105181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.589205+0000","last_change":"2023-10-06T03:29:34.546777+0000","last_active":"2023-10-06T03:29:38.589205+0000","last_peered":"2023-10-06T03:29:38.589205+0000","last_clean":"2023-10-06T03:29:38.589205+0000","last_became_active":"2023-10-06T03:29:34.545461+0000","last_became_peered":"2023-10-06T03:29:34.545461+0000","last_unstale":"2023-10-06T03:29:38.589205+0000","last_undegraded":"2023-10-06T03:29:38.589205+0000","last_fullsized":"2023-10-06T03:29:38.589205+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_clean_scrub_stamp":"2023-10-06T03:29:33.500929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T06:40:43.933850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.621795+0000","last_change":"2023-10-06T03:29:38.619453+0000","last_active":"2023-10-06T03:29:38.621795+0000","last_peered":"2023-10-06T03:29:38.621795+0000","last_clean":"2023-10-06T03:29:38.621795+0000","last_became_active":"2023-10-06T03:29:38.618690+0000","last_became_peered":"2023-10-06T03:29:38.618690+0000","last_unstale":"2023-10-06T03:29:38.621795+0000","last_undegraded":"2023-10-06T03:29:38.621795+0000","last_fullsized":"2023-10-06T03:29:38.621795+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:37.570693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:37.570693+0000","last_clean_scrub_stamp":"2023-10-06T03:29:37.570693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T06:00:28.537215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7,4],"acting":[5,0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"23'2","reported_seq":16,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.602095+0000","last_change":"2023-10-06T03:29:33.542309+0000","last_active":"2023-10-06T03:29:38.602095+0000","last_peered":"2023-10-06T03:29:38.602095+0000","last_clean":"2023-10-06T03:29:38.602095+0000","last_became_active":"2023-10-06T03:29:33.542038+0000","last_became_peered":"2023-10-06T03:29:33.542038+0000","last_unstale":"2023-10-06T03:29:38.602095+0000","last_undegraded":"2023-10-06T03:29:38.602095+0000","last_fullsized":"2023-10-06T03:29:38.602095+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:32.495616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:32.495616+0000","last_clean_scrub_stamp":"2023-10-06T03:29:32.495616+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T08:04:27.115289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.592351+0000","last_change":"2023-10-06T03:29:34.540591+0000","last_active":"2023-10-06T03:29:38.592351+0000","last_peered":"2023-10-06T03:29:38.592351+0000","last_clean":"2023-10-06T03:29:38.592351+0000","last_became_active":"2023-10-06T03:29:34.540219+0000","last_became_peered":"2023-10-06T03:29:34.540219+0000","last_unstale":"2023-10-06T03:29:38.592351+0000","last_undegraded":"2023-10-06T03:29:38.592351+0000","last_fullsized":"2023-10-06T03:29:38.592351+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_clean_scrub_stamp":"2023-10-06T03:29:33.500929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T11:28:48.945575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.626642+0000","last_change":"2023-10-06T03:29:38.624230+0000","last_active":"2023-10-06T03:29:38.626642+0000","last_peered":"2023-10-06T03:29:38.626642+0000","last_clean":"2023-10-06T03:29:38.626642+0000","last_became_active":"2023-10-06T03:29:38.623638+0000","last_became_peered":"2023-10-06T03:29:38.623638+0000","last_unstale":"2023-10-06T03:29:38.626642+0000","last_undegraded":"2023-10-06T03:29:38.626642+0000","last_fullsized":"2023-10-06T03:29:38.626642+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:37.570693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:37.570693+0000","last_clean_scrub_stamp":"2023-10-06T03:29:37.570693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T12:11:50.851770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1,6],"acting":[7,4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"23'2","reported_seq":16,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.600846+0000","last_change":"2023-10-06T03:29:33.540179+0000","last_active":"2023-10-06T03:29:38.600846+0000","last_peered":"2023-10-06T03:29:38.600846+0000","last_clean":"2023-10-06T03:29:38.600846+0000","last_became_active":"2023-10-06T03:29:33.539825+0000","last_became_peered":"2023-10-06T03:29:33.539825+0000","last_unstale":"2023-10-06T03:29:38.600846+0000","last_undegraded":"2023-10-06T03:29:38.600846+0000","last_fullsized":"2023-10-06T03:29:38.600846+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:32.495616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:32.495616+0000","last_clean_scrub_stamp":"2023-10-06T03:29:32.495616+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T11:58:53.137774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.593620+0000","last_change":"2023-10-06T03:29:34.540676+0000","last_active":"2023-10-06T03:29:38.593620+0000","last_peered":"2023-10-06T03:29:38.593620+0000","last_clean":"2023-10-06T03:29:38.593620+0000","last_became_active":"2023-10-06T03:29:34.540358+0000","last_became_peered":"2023-10-06T03:29:34.540358+0000","last_unstale":"2023-10-06T03:29:38.593620+0000","last_undegraded":"2023-10-06T03:29:38.593620+0000","last_fullsized":"2023-10-06T03:29:38.593620+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_clean_scrub_stamp":"2023-10-06T03:29:33.500929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T07:51:21.462969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.624627+0000","last_change":"2023-10-06T03:29:38.620564+0000","last_active":"2023-10-06T03:29:38.624627+0000","last_peered":"2023-10-06T03:29:38.624627+0000","last_clean":"2023-10-06T03:29:38.624627+0000","last_became_active":"2023-10-06T03:29:38.620134+0000","last_became_peered":"2023-10-06T03:29:38.620134+0000","last_unstale":"2023-10-06T03:29:38.624627+0000","last_undegraded":"2023-10-06T03:29:38.624627+0000","last_fullsized":"2023-10-06T03:29:38.624627+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:37.570693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:37.570693+0000","last_clean_scrub_stamp":"2023-10-06T03:29:37.570693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T04:19:16.510300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4,0],"acting":[1,3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"23'1","reported_seq":15,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.593175+0000","last_change":"2023-10-06T03:29:33.542267+0000","last_active":"2023-10-06T03:29:38.593175+0000","last_peered":"2023-10-06T03:29:38.593175+0000","last_clean":"2023-10-06T03:29:38.593175+0000","last_became_active":"2023-10-06T03:29:33.541987+0000","last_became_peered":"2023-10-06T03:29:33.541987+0000","last_unstale":"2023-10-06T03:29:38.593175+0000","last_undegraded":"2023-10-06T03:29:38.593175+0000","last_fullsized":"2023-10-06T03:29:38.593175+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:32.495616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:32.495616+0000","last_clean_scrub_stamp":"2023-10-06T03:29:32.495616+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T12:25:10.766051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.600288+0000","last_change":"2023-10-06T03:29:34.547283+0000","last_active":"2023-10-06T03:29:38.600288+0000","last_peered":"2023-10-06T03:29:38.600288+0000","last_clean":"2023-10-06T03:29:38.600288+0000","last_became_active":"2023-10-06T03:29:34.546966+0000","last_became_peered":"2023-10-06T03:29:34.546966+0000","last_unstale":"2023-10-06T03:29:38.600288+0000","last_undegraded":"2023-10-06T03:29:38.600288+0000","last_fullsized":"2023-10-06T03:29:38.600288+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_clean_scrub_stamp":"2023-10-06T03:29:33.500929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T07:55:08.792414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.621924+0000","last_change":"2023-10-06T03:29:38.614364+0000","last_active":"2023-10-06T03:29:38.621924+0000","last_peered":"2023-10-06T03:29:38.621924+0000","last_clean":"2023-10-06T03:29:38.621924+0000","last_became_active":"2023-10-06T03:29:38.613892+0000","last_became_peered":"2023-10-06T03:29:38.613892+0000","last_unstale":"2023-10-06T03:29:38.621924+0000","last_undegraded":"2023-10-06T03:29:38.621924+0000","last_fullsized":"2023-10-06T03:29:38.621924+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:37.570693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:37.570693+0000","last_clean_scrub_stamp":"2023-10-06T03:29:37.570693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T13:00:21.693396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2,0],"acting":[4,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"23'1","reported_seq":15,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.592761+0000","last_change":"2023-10-06T03:29:33.546282+0000","last_active":"2023-10-06T03:29:38.592761+0000","last_peered":"2023-10-06T03:29:38.592761+0000","last_clean":"2023-10-06T03:29:38.592761+0000","last_became_active":"2023-10-06T03:29:33.546039+0000","last_became_peered":"2023-10-06T03:29:33.546039+0000","last_unstale":"2023-10-06T03:29:38.592761+0000","last_undegraded":"2023-10-06T03:29:38.592761+0000","last_fullsized":"2023-10-06T03:29:38.592761+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:32.495616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:32.495616+0000","last_clean_scrub_stamp":"2023-10-06T03:29:32.495616+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T12:01:27.570873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.589771+0000","last_change":"2023-10-06T03:29:34.550953+0000","last_active":"2023-10-06T03:29:38.589771+0000","last_peered":"2023-10-06T03:29:38.589771+0000","last_clean":"2023-10-06T03:29:38.589771+0000","last_became_active":"2023-10-06T03:29:34.550614+0000","last_became_peered":"2023-10-06T03:29:34.550614+0000","last_unstale":"2023-10-06T03:29:38.589771+0000","last_undegraded":"2023-10-06T03:29:38.589771+0000","last_fullsized":"2023-10-06T03:29:38.589771+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_clean_scrub_stamp":"2023-10-06T03:29:33.500929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T11:59:02.611216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.620922+0000","last_change":"2023-10-06T03:29:38.613494+0000","last_active":"2023-10-06T03:29:38.620922+0000","last_peered":"2023-10-06T03:29:38.620922+0000","last_clean":"2023-10-06T03:29:38.620922+0000","last_became_active":"2023-10-06T03:29:38.613121+0000","last_became_peered":"2023-10-06T03:29:38.613121+0000","last_unstale":"2023-10-06T03:29:38.620922+0000","last_undegraded":"2023-10-06T03:29:38.620922+0000","last_fullsized":"2023-10-06T03:29:38.620922+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:37.570693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:37.570693+0000","last_clean_scrub_stamp":"2023-10-06T03:29:37.570693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T03:45:24.009354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3,7],"acting":[6,4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.608215+0000","last_change":"2023-10-06T03:29:33.542907+0000","last_active":"2023-10-06T03:29:38.608215+0000","last_peered":"2023-10-06T03:29:38.608215+0000","last_clean":"2023-10-06T03:29:38.608215+0000","last_became_active":"2023-10-06T03:29:33.542556+0000","last_became_peered":"2023-10-06T03:29:33.542556+0000","last_unstale":"2023-10-06T03:29:38.608215+0000","last_undegraded":"2023-10-06T03:29:38.608215+0000","last_fullsized":"2023-10-06T03:29:38.608215+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:32.495616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:32.495616+0000","last_clean_scrub_stamp":"2023-10-06T03:29:32.495616+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T03:54:50.215488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.599642+0000","last_change":"2023-10-06T03:29:34.541680+0000","last_active":"2023-10-06T03:29:38.599642+0000","last_peered":"2023-10-06T03:29:38.599642+0000","last_clean":"2023-10-06T03:29:38.599642+0000","last_became_active":"2023-10-06T03:29:34.541220+0000","last_became_peered":"2023-10-06T03:29:34.541220+0000","last_unstale":"2023-10-06T03:29:38.599642+0000","last_undegraded":"2023-10-06T03:29:38.599642+0000","last_fullsized":"2023-10-06T03:29:38.599642+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_clean_scrub_stamp":"2023-10-06T03:29:33.500929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T06:09:15.729486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.627293+0000","last_change":"2023-10-06T03:29:38.623302+0000","last_active":"2023-10-06T03:29:38.627293+0000","last_peered":"2023-10-06T03:29:38.627293+0000","last_clean":"2023-10-06T03:29:38.627293+0000","last_became_active":"2023-10-06T03:29:38.622782+0000","last_became_peered":"2023-10-06T03:29:38.622782+0000","last_unstale":"2023-10-06T03:29:38.627293+0000","last_undegraded":"2023-10-06T03:29:38.627293+0000","last_fullsized":"2023-10-06T03:29:38.627293+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:37.570693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:37.570693+0000","last_clean_scrub_stamp":"2023-10-06T03:29:37.570693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T06:00:28.537215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1,2],"acting":[0,7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":14,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.589343+0000","last_change":"2023-10-06T03:29:33.543906+0000","last_active":"2023-10-06T03:29:38.589343+0000","last_peered":"2023-10-06T03:29:38.589343+0000","last_clean":"2023-10-06T03:29:38.589343+0000","last_became_active":"2023-10-06T03:29:33.543551+0000","last_became_peered":"2023-10-06T03:29:33.543551+0000","last_unstale":"2023-10-06T03:29:38.589343+0000","last_undegraded":"2023-10-06T03:29:38.589343+0000","last_fullsized":"2023-10-06T03:29:38.589343+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:32.495616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:32.495616+0000","last_clean_scrub_stamp":"2023-10-06T03:29:32.495616+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T03:34:32.799922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.609821+0000","last_change":"2023-10-06T03:29:34.549763+0000","last_active":"2023-10-06T03:29:38.609821+0000","last_peered":"2023-10-06T03:29:38.609821+0000","last_clean":"2023-10-06T03:29:38.609821+0000","last_became_active":"2023-10-06T03:29:34.549402+0000","last_became_peered":"2023-10-06T03:29:34.549402+0000","last_unstale":"2023-10-06T03:29:38.609821+0000","last_undegraded":"2023-10-06T03:29:38.609821+0000","last_fullsized":"2023-10-06T03:29:38.609821+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_clean_scrub_stamp":"2023-10-06T03:29:33.500929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T07:14:03.653521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.588889+0000","last_change":"2023-10-06T03:29:34.548269+0000","last_active":"2023-10-06T03:29:38.588889+0000","last_peered":"2023-10-06T03:29:38.588889+0000","last_clean":"2023-10-06T03:29:38.588889+0000","last_became_active":"2023-10-06T03:29:34.547982+0000","last_became_peered":"2023-10-06T03:29:34.547982+0000","last_unstale":"2023-10-06T03:29:38.588889+0000","last_undegraded":"2023-10-06T03:29:38.588889+0000","last_fullsized":"2023-10-06T03:29:38.588889+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_clean_scrub_stamp":"2023-10-06T03:29:33.500929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T03:54:50.291053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":14,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.596539+0000","last_change":"2023-10-06T03:29:33.542538+0000","last_active":"2023-10-06T03:29:38.596539+0000","last_peered":"2023-10-06T03:29:38.596539+0000","last_clean":"2023-10-06T03:29:38.596539+0000","last_became_active":"2023-10-06T03:29:33.536222+0000","last_became_peered":"2023-10-06T03:29:33.536222+0000","last_unstale":"2023-10-06T03:29:38.596539+0000","last_undegraded":"2023-10-06T03:29:38.596539+0000","last_fullsized":"2023-10-06T03:29:38.596539+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:32.495616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:32.495616+0000","last_clean_scrub_stamp":"2023-10-06T03:29:32.495616+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T07:14:02.648208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.620952+0000","last_change":"2023-10-06T03:29:38.613462+0000","last_active":"2023-10-06T03:29:38.620952+0000","last_peered":"2023-10-06T03:29:38.620952+0000","last_clean":"2023-10-06T03:29:38.620952+0000","last_became_active":"2023-10-06T03:29:38.613096+0000","last_became_peered":"2023-10-06T03:29:38.613096+0000","last_unstale":"2023-10-06T03:29:38.620952+0000","last_undegraded":"2023-10-06T03:29:38.620952+0000","last_fullsized":"2023-10-06T03:29:38.620952+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:37.570693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:37.570693+0000","last_clean_scrub_stamp":"2023-10-06T03:29:37.570693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T11:28:53.015339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7,1],"acting":[6,4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.589101+0000","last_change":"2023-10-06T03:29:34.546990+0000","last_active":"2023-10-06T03:29:38.589101+0000","last_peered":"2023-10-06T03:29:38.589101+0000","last_clean":"2023-10-06T03:29:38.589101+0000","last_became_active":"2023-10-06T03:29:34.544561+0000","last_became_peered":"2023-10-06T03:29:34.544561+0000","last_unstale":"2023-10-06T03:29:38.589101+0000","last_undegraded":"2023-10-06T03:29:38.589101+0000","last_fullsized":"2023-10-06T03:29:38.589101+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_clean_scrub_stamp":"2023-10-06T03:29:33.500929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T09:13:47.732421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":14,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.602311+0000","last_change":"2023-10-06T03:29:33.536276+0000","last_active":"2023-10-06T03:29:38.602311+0000","last_peered":"2023-10-06T03:29:38.602311+0000","last_clean":"2023-10-06T03:29:38.602311+0000","last_became_active":"2023-10-06T03:29:33.535859+0000","last_became_peered":"2023-10-06T03:29:33.535859+0000","last_unstale":"2023-10-06T03:29:38.602311+0000","last_undegraded":"2023-10-06T03:29:38.602311+0000","last_fullsized":"2023-10-06T03:29:38.602311+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:32.495616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:32.495616+0000","last_clean_scrub_stamp":"2023-10-06T03:29:32.495616+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T07:55:07.787102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.623243+0000","last_change":"2023-10-06T03:29:38.618743+0000","last_active":"2023-10-06T03:29:38.623243+0000","last_peered":"2023-10-06T03:29:38.623243+0000","last_clean":"2023-10-06T03:29:38.623243+0000","last_became_active":"2023-10-06T03:29:38.618260+0000","last_became_peered":"2023-10-06T03:29:38.618260+0000","last_unstale":"2023-10-06T03:29:38.623243+0000","last_undegraded":"2023-10-06T03:29:38.623243+0000","last_fullsized":"2023-10-06T03:29:38.623243+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:37.570693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:37.570693+0000","last_clean_scrub_stamp":"2023-10-06T03:29:37.570693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T04:14:12.838199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6,0],"acting":[7,4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.15","version":"23'3","reported_seq":16,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.606179+0000","last_change":"2023-10-06T03:29:33.541547+0000","last_active":"2023-10-06T03:29:38.606179+0000","last_peered":"2023-10-06T03:29:38.606179+0000","last_clean":"2023-10-06T03:29:38.606179+0000","last_became_active":"2023-10-06T03:29:33.541267+0000","last_became_peered":"2023-10-06T03:29:33.541267+0000","last_unstale":"2023-10-06T03:29:38.606179+0000","last_undegraded":"2023-10-06T03:29:38.606179+0000","last_fullsized":"2023-10-06T03:29:38.606179+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:32.495616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:32.495616+0000","last_clean_scrub_stamp":"2023-10-06T03:29:32.495616+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T12:25:10.766051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.608031+0000","last_change":"2023-10-06T03:29:34.550174+0000","last_active":"2023-10-06T03:29:38.608031+0000","last_peered":"2023-10-06T03:29:38.608031+0000","last_clean":"2023-10-06T03:29:38.608031+0000","last_became_active":"2023-10-06T03:29:34.549630+0000","last_became_peered":"2023-10-06T03:29:34.549630+0000","last_unstale":"2023-10-06T03:29:38.608031+0000","last_undegraded":"2023-10-06T03:29:38.608031+0000","last_fullsized":"2023-10-06T03:29:38.608031+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_clean_scrub_stamp":"2023-10-06T03:29:33.500929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T13:02:33.356490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.620450+0000","last_change":"2023-10-06T03:29:38.614643+0000","last_active":"2023-10-06T03:29:38.620450+0000","last_peered":"2023-10-06T03:29:38.620450+0000","last_clean":"2023-10-06T03:29:38.620450+0000","last_became_active":"2023-10-06T03:29:38.614242+0000","last_became_peered":"2023-10-06T03:29:38.614242+0000","last_unstale":"2023-10-06T03:29:38.620450+0000","last_undegraded":"2023-10-06T03:29:38.620450+0000","last_fullsized":"2023-10-06T03:29:38.620450+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:37.570693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:37.570693+0000","last_clean_scrub_stamp":"2023-10-06T03:29:37.570693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T06:29:58.268896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,5],"acting":[3,0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.14","version":"28'9","reported_seq":23,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T03:29:44.383979+0000","last_change":"2023-10-06T03:29:33.538661+0000","last_active":"2023-10-06T03:29:44.383979+0000","last_peered":"2023-10-06T03:29:44.383979+0000","last_clean":"2023-10-06T03:29:44.383979+0000","last_became_active":"2023-10-06T03:29:33.538296+0000","last_became_peered":"2023-10-06T03:29:33.538296+0000","last_unstale":"2023-10-06T03:29:44.383979+0000","last_undegraded":"2023-10-06T03:29:44.383979+0000","last_fullsized":"2023-10-06T03:29:44.383979+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:32.495616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:32.495616+0000","last_clean_scrub_stamp":"2023-10-06T03:29:32.495616+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T10:47:52.547584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3714,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.593839+0000","last_change":"2023-10-06T03:29:34.549928+0000","last_active":"2023-10-06T03:29:38.593839+0000","last_peered":"2023-10-06T03:29:38.593839+0000","last_clean":"2023-10-06T03:29:38.593839+0000","last_became_active":"2023-10-06T03:29:34.549421+0000","last_became_peered":"2023-10-06T03:29:34.549421+0000","last_unstale":"2023-10-06T03:29:38.593839+0000","last_undegraded":"2023-10-06T03:29:38.593839+0000","last_fullsized":"2023-10-06T03:29:38.593839+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_clean_scrub_stamp":"2023-10-06T03:29:33.500929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T03:45:19.939590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.624617+0000","last_change":"2023-10-06T03:29:38.620526+0000","last_active":"2023-10-06T03:29:38.624617+0000","last_peered":"2023-10-06T03:29:38.624617+0000","last_clean":"2023-10-06T03:29:38.624617+0000","last_became_active":"2023-10-06T03:29:38.620095+0000","last_became_peered":"2023-10-06T03:29:38.620095+0000","last_unstale":"2023-10-06T03:29:38.624617+0000","last_undegraded":"2023-10-06T03:29:38.624617+0000","last_fullsized":"2023-10-06T03:29:38.624617+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:37.570693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:37.570693+0000","last_clean_scrub_stamp":"2023-10-06T03:29:37.570693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T04:00:40.252544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3,7],"acting":[1,5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.594637+0000","last_change":"2023-10-06T03:29:34.538055+0000","last_active":"2023-10-06T03:29:38.594637+0000","last_peered":"2023-10-06T03:29:38.594637+0000","last_clean":"2023-10-06T03:29:38.594637+0000","last_became_active":"2023-10-06T03:29:34.537755+0000","last_became_peered":"2023-10-06T03:29:34.537755+0000","last_unstale":"2023-10-06T03:29:38.594637+0000","last_undegraded":"2023-10-06T03:29:38.594637+0000","last_fullsized":"2023-10-06T03:29:38.594637+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_clean_scrub_stamp":"2023-10-06T03:29:33.500929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T08:21:53.296989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":16,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.610021+0000","last_change":"2023-10-06T03:29:29.387402+0000","last_active":"2023-10-06T03:29:38.610021+0000","last_peered":"2023-10-06T03:29:38.610021+0000","last_clean":"2023-10-06T03:29:38.610021+0000","last_became_active":"2023-10-06T03:29:29.386812+0000","last_became_peered":"2023-10-06T03:29:29.386812+0000","last_unstale":"2023-10-06T03:29:38.610021+0000","last_undegraded":"2023-10-06T03:29:38.610021+0000","last_fullsized":"2023-10-06T03:29:38.610021+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:28.357506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:28.357506+0000","last_clean_scrub_stamp":"2023-10-06T03:29:28.357506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T12:09:43.895810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"27'1","reported_seq":15,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:39.494514+0000","last_change":"2023-10-06T03:29:33.542307+0000","last_active":"2023-10-06T03:29:39.494514+0000","last_peered":"2023-10-06T03:29:39.494514+0000","last_clean":"2023-10-06T03:29:39.494514+0000","last_became_active":"2023-10-06T03:29:33.541670+0000","last_became_peered":"2023-10-06T03:29:33.541670+0000","last_unstale":"2023-10-06T03:29:39.494514+0000","last_undegraded":"2023-10-06T03:29:39.494514+0000","last_fullsized":"2023-10-06T03:29:39.494514+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:32.495616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:32.495616+0000","last_clean_scrub_stamp":"2023-10-06T03:29:32.495616+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T05:03:47.111820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.620435+0000","last_change":"2023-10-06T03:29:38.613923+0000","last_active":"2023-10-06T03:29:38.620435+0000","last_peered":"2023-10-06T03:29:38.620435+0000","last_clean":"2023-10-06T03:29:38.620435+0000","last_became_active":"2023-10-06T03:29:38.612452+0000","last_became_peered":"2023-10-06T03:29:38.612452+0000","last_unstale":"2023-10-06T03:29:38.620435+0000","last_undegraded":"2023-10-06T03:29:38.620435+0000","last_fullsized":"2023-10-06T03:29:38.620435+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:37.570693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:37.570693+0000","last_clean_scrub_stamp":"2023-10-06T03:29:37.570693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T05:52:46.994576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4,0],"acting":[3,1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.589121+0000","last_change":"2023-10-06T03:29:34.539246+0000","last_active":"2023-10-06T03:29:38.589121+0000","last_peered":"2023-10-06T03:29:38.589121+0000","last_clean":"2023-10-06T03:29:38.589121+0000","last_became_active":"2023-10-06T03:29:34.538925+0000","last_became_peered":"2023-10-06T03:29:34.538925+0000","last_unstale":"2023-10-06T03:29:38.589121+0000","last_undegraded":"2023-10-06T03:29:38.589121+0000","last_fullsized":"2023-10-06T03:29:38.589121+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_clean_scrub_stamp":"2023-10-06T03:29:33.500929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T03:59:12.193877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":16,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.609252+0000","last_change":"2023-10-06T03:29:29.386238+0000","last_active":"2023-10-06T03:29:38.609252+0000","last_peered":"2023-10-06T03:29:38.609252+0000","last_clean":"2023-10-06T03:29:38.609252+0000","last_became_active":"2023-10-06T03:29:29.385865+0000","last_became_peered":"2023-10-06T03:29:29.385865+0000","last_unstale":"2023-10-06T03:29:38.609252+0000","last_undegraded":"2023-10-06T03:29:38.609252+0000","last_fullsized":"2023-10-06T03:29:38.609252+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:28.357506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:28.357506+0000","last_clean_scrub_stamp":"2023-10-06T03:29:28.357506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T12:43:34.714730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"23'2","reported_seq":16,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.593260+0000","last_change":"2023-10-06T03:29:33.539135+0000","last_active":"2023-10-06T03:29:38.593260+0000","last_peered":"2023-10-06T03:29:38.593260+0000","last_clean":"2023-10-06T03:29:38.593260+0000","last_became_active":"2023-10-06T03:29:33.538818+0000","last_became_peered":"2023-10-06T03:29:33.538818+0000","last_unstale":"2023-10-06T03:29:38.593260+0000","last_undegraded":"2023-10-06T03:29:38.593260+0000","last_fullsized":"2023-10-06T03:29:38.593260+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:32.495616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:32.495616+0000","last_clean_scrub_stamp":"2023-10-06T03:29:32.495616+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T03:54:50.215488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.622166+0000","last_change":"2023-10-06T03:29:38.614527+0000","last_active":"2023-10-06T03:29:38.622166+0000","last_peered":"2023-10-06T03:29:38.622166+0000","last_clean":"2023-10-06T03:29:38.622166+0000","last_became_active":"2023-10-06T03:29:38.614090+0000","last_became_peered":"2023-10-06T03:29:38.614090+0000","last_unstale":"2023-10-06T03:29:38.622166+0000","last_undegraded":"2023-10-06T03:29:38.622166+0000","last_fullsized":"2023-10-06T03:29:38.622166+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:37.570693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:37.570693+0000","last_clean_scrub_stamp":"2023-10-06T03:29:37.570693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T05:47:25.729169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7,0],"acting":[4,3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.592223+0000","last_change":"2023-10-06T03:29:34.537232+0000","last_active":"2023-10-06T03:29:38.592223+0000","last_peered":"2023-10-06T03:29:38.592223+0000","last_clean":"2023-10-06T03:29:38.592223+0000","last_became_active":"2023-10-06T03:29:34.536984+0000","last_became_peered":"2023-10-06T03:29:34.536984+0000","last_unstale":"2023-10-06T03:29:38.592223+0000","last_undegraded":"2023-10-06T03:29:38.592223+0000","last_fullsized":"2023-10-06T03:29:38.592223+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_clean_scrub_stamp":"2023-10-06T03:29:33.500929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T03:42:52.876376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":16,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.605996+0000","last_change":"2023-10-06T03:29:29.384279+0000","last_active":"2023-10-06T03:29:38.605996+0000","last_peered":"2023-10-06T03:29:38.605996+0000","last_clean":"2023-10-06T03:29:38.605996+0000","last_became_active":"2023-10-06T03:29:29.383533+0000","last_became_peered":"2023-10-06T03:29:29.383533+0000","last_unstale":"2023-10-06T03:29:38.605996+0000","last_undegraded":"2023-10-06T03:29:38.605996+0000","last_fullsized":"2023-10-06T03:29:38.605996+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:28.357506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:28.357506+0000","last_clean_scrub_stamp":"2023-10-06T03:29:28.357506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T08:34:19.641525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.609261+0000","last_change":"2023-10-06T03:29:33.542909+0000","last_active":"2023-10-06T03:29:38.609261+0000","last_peered":"2023-10-06T03:29:38.609261+0000","last_clean":"2023-10-06T03:29:38.609261+0000","last_became_active":"2023-10-06T03:29:33.542573+0000","last_became_peered":"2023-10-06T03:29:33.542573+0000","last_unstale":"2023-10-06T03:29:38.609261+0000","last_undegraded":"2023-10-06T03:29:38.609261+0000","last_fullsized":"2023-10-06T03:29:38.609261+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:32.495616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:32.495616+0000","last_clean_scrub_stamp":"2023-10-06T03:29:32.495616+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T14:26:49.262597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.620982+0000","last_change":"2023-10-06T03:29:38.615475+0000","last_active":"2023-10-06T03:29:38.620982+0000","last_peered":"2023-10-06T03:29:38.620982+0000","last_clean":"2023-10-06T03:29:38.620982+0000","last_became_active":"2023-10-06T03:29:38.615049+0000","last_became_peered":"2023-10-06T03:29:38.615049+0000","last_unstale":"2023-10-06T03:29:38.620982+0000","last_undegraded":"2023-10-06T03:29:38.620982+0000","last_fullsized":"2023-10-06T03:29:38.620982+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:37.570693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:37.570693+0000","last_clean_scrub_stamp":"2023-10-06T03:29:37.570693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T14:34:30.474789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5,2],"acting":[6,0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.608125+0000","last_change":"2023-10-06T03:29:34.543257+0000","last_active":"2023-10-06T03:29:38.608125+0000","last_peered":"2023-10-06T03:29:38.608125+0000","last_clean":"2023-10-06T03:29:38.608125+0000","last_became_active":"2023-10-06T03:29:34.537574+0000","last_became_peered":"2023-10-06T03:29:34.537574+0000","last_unstale":"2023-10-06T03:29:38.608125+0000","last_undegraded":"2023-10-06T03:29:38.608125+0000","last_fullsized":"2023-10-06T03:29:38.608125+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_clean_scrub_stamp":"2023-10-06T03:29:33.500929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T13:34:24.699976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":16,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.609348+0000","last_change":"2023-10-06T03:29:29.382183+0000","last_active":"2023-10-06T03:29:38.609348+0000","last_peered":"2023-10-06T03:29:38.609348+0000","last_clean":"2023-10-06T03:29:38.609348+0000","last_became_active":"2023-10-06T03:29:29.381918+0000","last_became_peered":"2023-10-06T03:29:29.381918+0000","last_unstale":"2023-10-06T03:29:38.609348+0000","last_undegraded":"2023-10-06T03:29:38.609348+0000","last_fullsized":"2023-10-06T03:29:38.609348+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:28.357506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:28.357506+0000","last_clean_scrub_stamp":"2023-10-06T03:29:28.357506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T11:56:26.013716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":14,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.602189+0000","last_change":"2023-10-06T03:29:33.540375+0000","last_active":"2023-10-06T03:29:38.602189+0000","last_peered":"2023-10-06T03:29:38.602189+0000","last_clean":"2023-10-06T03:29:38.602189+0000","last_became_active":"2023-10-06T03:29:33.540016+0000","last_became_peered":"2023-10-06T03:29:33.540016+0000","last_unstale":"2023-10-06T03:29:38.602189+0000","last_undegraded":"2023-10-06T03:29:38.602189+0000","last_fullsized":"2023-10-06T03:29:38.602189+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:32.495616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:32.495616+0000","last_clean_scrub_stamp":"2023-10-06T03:29:32.495616+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T10:47:52.547584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.623995+0000","last_change":"2023-10-06T03:29:38.613181+0000","last_active":"2023-10-06T03:29:38.623995+0000","last_peered":"2023-10-06T03:29:38.623995+0000","last_clean":"2023-10-06T03:29:38.623995+0000","last_became_active":"2023-10-06T03:29:38.612787+0000","last_became_peered":"2023-10-06T03:29:38.612787+0000","last_unstale":"2023-10-06T03:29:38.623995+0000","last_undegraded":"2023-10-06T03:29:38.623995+0000","last_fullsized":"2023-10-06T03:29:38.623995+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:37.570693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:37.570693+0000","last_clean_scrub_stamp":"2023-10-06T03:29:37.570693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T07:38:31.118869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6,5],"acting":[0,1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.609205+0000","last_change":"2023-10-06T03:29:34.543207+0000","last_active":"2023-10-06T03:29:38.609205+0000","last_peered":"2023-10-06T03:29:38.609205+0000","last_clean":"2023-10-06T03:29:38.609205+0000","last_became_active":"2023-10-06T03:29:34.537291+0000","last_became_peered":"2023-10-06T03:29:34.537291+0000","last_unstale":"2023-10-06T03:29:38.609205+0000","last_undegraded":"2023-10-06T03:29:38.609205+0000","last_fullsized":"2023-10-06T03:29:38.609205+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_clean_scrub_stamp":"2023-10-06T03:29:33.500929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T06:40:43.933850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":14,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.594240+0000","last_change":"2023-10-06T03:29:33.543412+0000","last_active":"2023-10-06T03:29:38.594240+0000","last_peered":"2023-10-06T03:29:38.594240+0000","last_clean":"2023-10-06T03:29:38.594240+0000","last_became_active":"2023-10-06T03:29:33.543114+0000","last_became_peered":"2023-10-06T03:29:33.543114+0000","last_unstale":"2023-10-06T03:29:38.594240+0000","last_undegraded":"2023-10-06T03:29:38.594240+0000","last_fullsized":"2023-10-06T03:29:38.594240+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:32.495616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:32.495616+0000","last_clean_scrub_stamp":"2023-10-06T03:29:32.495616+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T15:12:57.750941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":17,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.600966+0000","last_change":"2023-10-06T03:29:29.384415+0000","last_active":"2023-10-06T03:29:38.600966+0000","last_peered":"2023-10-06T03:29:38.600966+0000","last_clean":"2023-10-06T03:29:38.600966+0000","last_became_active":"2023-10-06T03:29:29.383894+0000","last_became_peered":"2023-10-06T03:29:29.383894+0000","last_unstale":"2023-10-06T03:29:38.600966+0000","last_undegraded":"2023-10-06T03:29:38.600966+0000","last_fullsized":"2023-10-06T03:29:38.600966+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:28.357506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:28.357506+0000","last_clean_scrub_stamp":"2023-10-06T03:29:28.357506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T13:09:00.657107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.622959+0000","last_change":"2023-10-06T03:29:38.613076+0000","last_active":"2023-10-06T03:29:38.622959+0000","last_peered":"2023-10-06T03:29:38.622959+0000","last_clean":"2023-10-06T03:29:38.622959+0000","last_became_active":"2023-10-06T03:29:38.612646+0000","last_became_peered":"2023-10-06T03:29:38.612646+0000","last_unstale":"2023-10-06T03:29:38.622959+0000","last_undegraded":"2023-10-06T03:29:38.622959+0000","last_fullsized":"2023-10-06T03:29:38.622959+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:37.570693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:37.570693+0000","last_clean_scrub_stamp":"2023-10-06T03:29:37.570693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T06:07:31.871068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4,6],"acting":[0,1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.609278+0000","last_change":"2023-10-06T03:29:34.543346+0000","last_active":"2023-10-06T03:29:38.609278+0000","last_peered":"2023-10-06T03:29:38.609278+0000","last_clean":"2023-10-06T03:29:38.609278+0000","last_became_active":"2023-10-06T03:29:34.537438+0000","last_became_peered":"2023-10-06T03:29:34.537438+0000","last_unstale":"2023-10-06T03:29:38.609278+0000","last_undegraded":"2023-10-06T03:29:38.609278+0000","last_fullsized":"2023-10-06T03:29:38.609278+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_clean_scrub_stamp":"2023-10-06T03:29:33.500929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T09:13:47.732421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.609403+0000","last_change":"2023-10-06T03:29:33.544242+0000","last_active":"2023-10-06T03:29:38.609403+0000","last_peered":"2023-10-06T03:29:38.609403+0000","last_clean":"2023-10-06T03:29:38.609403+0000","last_became_active":"2023-10-06T03:29:33.543938+0000","last_became_peered":"2023-10-06T03:29:33.543938+0000","last_unstale":"2023-10-06T03:29:38.609403+0000","last_undegraded":"2023-10-06T03:29:38.609403+0000","last_fullsized":"2023-10-06T03:29:38.609403+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:32.495616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:32.495616+0000","last_clean_scrub_stamp":"2023-10-06T03:29:32.495616+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T10:09:45.061520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":16,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.599964+0000","last_change":"2023-10-06T03:29:29.606195+0000","last_active":"2023-10-06T03:29:38.599964+0000","last_peered":"2023-10-06T03:29:38.599964+0000","last_clean":"2023-10-06T03:29:38.599964+0000","last_became_active":"2023-10-06T03:29:29.605489+0000","last_became_peered":"2023-10-06T03:29:29.605489+0000","last_unstale":"2023-10-06T03:29:38.599964+0000","last_undegraded":"2023-10-06T03:29:38.599964+0000","last_fullsized":"2023-10-06T03:29:38.599964+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:28.357506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:28.357506+0000","last_clean_scrub_stamp":"2023-10-06T03:29:28.357506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T10:21:35.864895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.626333+0000","last_change":"2023-10-06T03:29:38.624434+0000","last_active":"2023-10-06T03:29:38.626333+0000","last_peered":"2023-10-06T03:29:38.626333+0000","last_clean":"2023-10-06T03:29:38.626333+0000","last_became_active":"2023-10-06T03:29:38.623676+0000","last_became_peered":"2023-10-06T03:29:38.623676+0000","last_unstale":"2023-10-06T03:29:38.626333+0000","last_undegraded":"2023-10-06T03:29:38.626333+0000","last_fullsized":"2023-10-06T03:29:38.626333+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:37.570693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:37.570693+0000","last_clean_scrub_stamp":"2023-10-06T03:29:37.570693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T12:26:20.152406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,1,7],"acting":[2,5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.599731+0000","last_change":"2023-10-06T03:29:34.550793+0000","last_active":"2023-10-06T03:29:38.599731+0000","last_peered":"2023-10-06T03:29:38.599731+0000","last_clean":"2023-10-06T03:29:38.599731+0000","last_became_active":"2023-10-06T03:29:34.550345+0000","last_became_peered":"2023-10-06T03:29:34.550345+0000","last_unstale":"2023-10-06T03:29:38.599731+0000","last_undegraded":"2023-10-06T03:29:38.599731+0000","last_fullsized":"2023-10-06T03:29:38.599731+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_clean_scrub_stamp":"2023-10-06T03:29:33.500929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T11:28:48.945575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":14,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.591494+0000","last_change":"2023-10-06T03:29:33.543304+0000","last_active":"2023-10-06T03:29:38.591494+0000","last_peered":"2023-10-06T03:29:38.591494+0000","last_clean":"2023-10-06T03:29:38.591494+0000","last_became_active":"2023-10-06T03:29:33.542483+0000","last_became_peered":"2023-10-06T03:29:33.542483+0000","last_unstale":"2023-10-06T03:29:38.591494+0000","last_undegraded":"2023-10-06T03:29:38.591494+0000","last_fullsized":"2023-10-06T03:29:38.591494+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:32.495616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:32.495616+0000","last_clean_scrub_stamp":"2023-10-06T03:29:32.495616+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T11:58:53.137774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":16,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.603498+0000","last_change":"2023-10-06T03:29:29.384353+0000","last_active":"2023-10-06T03:29:38.603498+0000","last_peered":"2023-10-06T03:29:38.603498+0000","last_clean":"2023-10-06T03:29:38.603498+0000","last_became_active":"2023-10-06T03:29:29.383884+0000","last_became_peered":"2023-10-06T03:29:29.383884+0000","last_unstale":"2023-10-06T03:29:38.603498+0000","last_undegraded":"2023-10-06T03:29:38.603498+0000","last_fullsized":"2023-10-06T03:29:38.603498+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:28.357506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:28.357506+0000","last_clean_scrub_stamp":"2023-10-06T03:29:28.357506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T14:43:09.107140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.629976+0000","last_change":"2023-10-06T03:29:38.626568+0000","last_active":"2023-10-06T03:29:38.629976+0000","last_peered":"2023-10-06T03:29:38.629976+0000","last_clean":"2023-10-06T03:29:38.629976+0000","last_became_active":"2023-10-06T03:29:38.625323+0000","last_became_peered":"2023-10-06T03:29:38.625323+0000","last_unstale":"2023-10-06T03:29:38.629976+0000","last_undegraded":"2023-10-06T03:29:38.629976+0000","last_fullsized":"2023-10-06T03:29:38.629976+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:37.570693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:37.570693+0000","last_clean_scrub_stamp":"2023-10-06T03:29:37.570693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T04:51:55.443872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0,4],"acting":[5,1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.609554+0000","last_change":"2023-10-06T03:29:34.549281+0000","last_active":"2023-10-06T03:29:38.609554+0000","last_peered":"2023-10-06T03:29:38.609554+0000","last_clean":"2023-10-06T03:29:38.609554+0000","last_became_active":"2023-10-06T03:29:34.548900+0000","last_became_peered":"2023-10-06T03:29:34.548900+0000","last_unstale":"2023-10-06T03:29:38.609554+0000","last_undegraded":"2023-10-06T03:29:38.609554+0000","last_fullsized":"2023-10-06T03:29:38.609554+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_clean_scrub_stamp":"2023-10-06T03:29:33.500929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T09:06:08.844003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":14,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.589353+0000","last_change":"2023-10-06T03:29:33.534969+0000","last_active":"2023-10-06T03:29:38.589353+0000","last_peered":"2023-10-06T03:29:38.589353+0000","last_clean":"2023-10-06T03:29:38.589353+0000","last_became_active":"2023-10-06T03:29:33.534402+0000","last_became_peered":"2023-10-06T03:29:33.534402+0000","last_unstale":"2023-10-06T03:29:38.589353+0000","last_undegraded":"2023-10-06T03:29:38.589353+0000","last_fullsized":"2023-10-06T03:29:38.589353+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:32.495616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:32.495616+0000","last_clean_scrub_stamp":"2023-10-06T03:29:32.495616+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T11:00:52.451461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":17,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.601254+0000","last_change":"2023-10-06T03:29:29.386120+0000","last_active":"2023-10-06T03:29:38.601254+0000","last_peered":"2023-10-06T03:29:38.601254+0000","last_clean":"2023-10-06T03:29:38.601254+0000","last_became_active":"2023-10-06T03:29:29.385692+0000","last_became_peered":"2023-10-06T03:29:29.385692+0000","last_unstale":"2023-10-06T03:29:38.601254+0000","last_undegraded":"2023-10-06T03:29:38.601254+0000","last_fullsized":"2023-10-06T03:29:38.601254+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:28.357506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:28.357506+0000","last_clean_scrub_stamp":"2023-10-06T03:29:28.357506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T10:21:35.864895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":169,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.603558+0000","last_change":"2023-10-06T03:29:21.978311+0000","last_active":"2023-10-06T03:29:38.603558+0000","last_peered":"2023-10-06T03:29:38.603558+0000","last_clean":"2023-10-06T03:29:38.603558+0000","last_became_active":"2023-10-06T03:29:21.978017+0000","last_became_peered":"2023-10-06T03:29:21.978017+0000","last_unstale":"2023-10-06T03:29:38.603558+0000","last_undegraded":"2023-10-06T03:29:38.603558+0000","last_fullsized":"2023-10-06T03:29:38.603558+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:20.957955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:20.957955+0000","last_clean_scrub_stamp":"2023-10-06T03:29:20.957955+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T07:33:10.429084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.614806+0000","last_change":"2023-10-06T03:29:38.612549+0000","last_active":"2023-10-06T03:29:38.614806+0000","last_peered":"2023-10-06T03:29:38.614806+0000","last_clean":"2023-10-06T03:29:38.614806+0000","last_became_active":"2023-10-06T03:29:38.611790+0000","last_became_peered":"2023-10-06T03:29:38.611790+0000","last_unstale":"2023-10-06T03:29:38.614806+0000","last_undegraded":"2023-10-06T03:29:38.614806+0000","last_fullsized":"2023-10-06T03:29:38.614806+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:37.570693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:37.570693+0000","last_clean_scrub_stamp":"2023-10-06T03:29:37.570693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T12:17:41.605048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5,3],"acting":[7,2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.608679+0000","last_change":"2023-10-06T03:29:34.545006+0000","last_active":"2023-10-06T03:29:38.608679+0000","last_peered":"2023-10-06T03:29:38.608679+0000","last_clean":"2023-10-06T03:29:38.608679+0000","last_became_active":"2023-10-06T03:29:34.544682+0000","last_became_peered":"2023-10-06T03:29:34.544682+0000","last_unstale":"2023-10-06T03:29:38.608679+0000","last_undegraded":"2023-10-06T03:29:38.608679+0000","last_fullsized":"2023-10-06T03:29:38.608679+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_clean_scrub_stamp":"2023-10-06T03:29:33.500929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T03:59:12.193877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":14,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.589559+0000","last_change":"2023-10-06T03:29:33.545053+0000","last_active":"2023-10-06T03:29:38.589559+0000","last_peered":"2023-10-06T03:29:38.589559+0000","last_clean":"2023-10-06T03:29:38.589559+0000","last_became_active":"2023-10-06T03:29:33.544749+0000","last_became_peered":"2023-10-06T03:29:33.544749+0000","last_unstale":"2023-10-06T03:29:38.589559+0000","last_undegraded":"2023-10-06T03:29:38.589559+0000","last_fullsized":"2023-10-06T03:29:38.589559+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:32.495616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:32.495616+0000","last_clean_scrub_stamp":"2023-10-06T03:29:32.495616+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T14:26:49.262597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.621981+0000","last_change":"2023-10-06T03:29:38.614480+0000","last_active":"2023-10-06T03:29:38.621981+0000","last_peered":"2023-10-06T03:29:38.621981+0000","last_clean":"2023-10-06T03:29:38.621981+0000","last_became_active":"2023-10-06T03:29:38.613976+0000","last_became_peered":"2023-10-06T03:29:38.613976+0000","last_unstale":"2023-10-06T03:29:38.621981+0000","last_undegraded":"2023-10-06T03:29:38.621981+0000","last_fullsized":"2023-10-06T03:29:38.621981+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:37.570693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:37.570693+0000","last_clean_scrub_stamp":"2023-10-06T03:29:37.570693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T09:04:13.741349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0,2],"acting":[4,5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.592722+0000","last_change":"2023-10-06T03:29:34.551311+0000","last_active":"2023-10-06T03:29:38.592722+0000","last_peered":"2023-10-06T03:29:38.592722+0000","last_clean":"2023-10-06T03:29:38.592722+0000","last_became_active":"2023-10-06T03:29:34.550961+0000","last_became_peered":"2023-10-06T03:29:34.550961+0000","last_unstale":"2023-10-06T03:29:38.592722+0000","last_undegraded":"2023-10-06T03:29:38.592722+0000","last_fullsized":"2023-10-06T03:29:38.592722+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_clean_scrub_stamp":"2023-10-06T03:29:33.500929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T14:20:50.168977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":14,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.592869+0000","last_change":"2023-10-06T03:29:33.538732+0000","last_active":"2023-10-06T03:29:38.592869+0000","last_peered":"2023-10-06T03:29:38.592869+0000","last_clean":"2023-10-06T03:29:38.592869+0000","last_became_active":"2023-10-06T03:29:33.538430+0000","last_became_peered":"2023-10-06T03:29:33.538430+0000","last_unstale":"2023-10-06T03:29:38.592869+0000","last_undegraded":"2023-10-06T03:29:38.592869+0000","last_fullsized":"2023-10-06T03:29:38.592869+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:32.495616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:32.495616+0000","last_clean_scrub_stamp":"2023-10-06T03:29:32.495616+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T12:57:50.614907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.621727+0000","last_change":"2023-10-06T03:29:38.617352+0000","last_active":"2023-10-06T03:29:38.621727+0000","last_peered":"2023-10-06T03:29:38.621727+0000","last_clean":"2023-10-06T03:29:38.621727+0000","last_became_active":"2023-10-06T03:29:38.616718+0000","last_became_peered":"2023-10-06T03:29:38.616718+0000","last_unstale":"2023-10-06T03:29:38.621727+0000","last_undegraded":"2023-10-06T03:29:38.621727+0000","last_fullsized":"2023-10-06T03:29:38.621727+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:37.570693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:37.570693+0000","last_clean_scrub_stamp":"2023-10-06T03:29:37.570693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T11:16:01.323266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6,2],"acting":[5,4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.592923+0000","last_change":"2023-10-06T03:29:34.541404+0000","last_active":"2023-10-06T03:29:38.592923+0000","last_peered":"2023-10-06T03:29:38.592923+0000","last_clean":"2023-10-06T03:29:38.592923+0000","last_became_active":"2023-10-06T03:29:34.541037+0000","last_became_peered":"2023-10-06T03:29:34.541037+0000","last_unstale":"2023-10-06T03:29:38.592923+0000","last_undegraded":"2023-10-06T03:29:38.592923+0000","last_fullsized":"2023-10-06T03:29:38.592923+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_clean_scrub_stamp":"2023-10-06T03:29:33.500929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T08:57:07.064845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.610030+0000","last_change":"2023-10-06T03:29:33.542889+0000","last_active":"2023-10-06T03:29:38.610030+0000","last_peered":"2023-10-06T03:29:38.610030+0000","last_clean":"2023-10-06T03:29:38.610030+0000","last_became_active":"2023-10-06T03:29:33.542527+0000","last_became_peered":"2023-10-06T03:29:33.542527+0000","last_unstale":"2023-10-06T03:29:38.610030+0000","last_undegraded":"2023-10-06T03:29:38.610030+0000","last_fullsized":"2023-10-06T03:29:38.610030+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:32.495616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:32.495616+0000","last_clean_scrub_stamp":"2023-10-06T03:29:32.495616+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T09:16:59.387601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.624663+0000","last_change":"2023-10-06T03:29:38.616847+0000","last_active":"2023-10-06T03:29:38.624663+0000","last_peered":"2023-10-06T03:29:38.624663+0000","last_clean":"2023-10-06T03:29:38.624663+0000","last_became_active":"2023-10-06T03:29:38.616359+0000","last_became_peered":"2023-10-06T03:29:38.616359+0000","last_unstale":"2023-10-06T03:29:38.624663+0000","last_undegraded":"2023-10-06T03:29:38.624663+0000","last_fullsized":"2023-10-06T03:29:38.624663+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:37.570693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:37.570693+0000","last_clean_scrub_stamp":"2023-10-06T03:29:37.570693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T10:33:46.594949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0,6],"acting":[1,4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.593466+0000","last_change":"2023-10-06T03:29:34.542358+0000","last_active":"2023-10-06T03:29:38.593466+0000","last_peered":"2023-10-06T03:29:38.593466+0000","last_clean":"2023-10-06T03:29:38.593466+0000","last_became_active":"2023-10-06T03:29:34.542038+0000","last_became_peered":"2023-10-06T03:29:34.542038+0000","last_unstale":"2023-10-06T03:29:38.593466+0000","last_undegraded":"2023-10-06T03:29:38.593466+0000","last_fullsized":"2023-10-06T03:29:38.593466+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_clean_scrub_stamp":"2023-10-06T03:29:33.500929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T11:19:26.387732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"23'2","reported_seq":16,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.590740+0000","last_change":"2023-10-06T03:29:33.544626+0000","last_active":"2023-10-06T03:29:38.590740+0000","last_peered":"2023-10-06T03:29:38.590740+0000","last_clean":"2023-10-06T03:29:38.590740+0000","last_became_active":"2023-10-06T03:29:33.544301+0000","last_became_peered":"2023-10-06T03:29:33.544301+0000","last_unstale":"2023-10-06T03:29:38.590740+0000","last_undegraded":"2023-10-06T03:29:38.590740+0000","last_fullsized":"2023-10-06T03:29:38.590740+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:32.495616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:32.495616+0000","last_clean_scrub_stamp":"2023-10-06T03:29:32.495616+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T09:13:04.451900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.613447+0000","last_change":"2023-10-06T03:29:38.610963+0000","last_active":"2023-10-06T03:29:38.613447+0000","last_peered":"2023-10-06T03:29:38.613447+0000","last_clean":"2023-10-06T03:29:38.613447+0000","last_became_active":"2023-10-06T03:29:38.610492+0000","last_became_peered":"2023-10-06T03:29:38.610492+0000","last_unstale":"2023-10-06T03:29:38.613447+0000","last_undegraded":"2023-10-06T03:29:38.613447+0000","last_fullsized":"2023-10-06T03:29:38.613447+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:37.570693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:37.570693+0000","last_clean_scrub_stamp":"2023-10-06T03:29:37.570693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T04:15:31.186577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5,6],"acting":[2,7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.592107+0000","last_change":"2023-10-06T03:29:34.547048+0000","last_active":"2023-10-06T03:29:38.592107+0000","last_peered":"2023-10-06T03:29:38.592107+0000","last_clean":"2023-10-06T03:29:38.592107+0000","last_became_active":"2023-10-06T03:29:34.546757+0000","last_became_peered":"2023-10-06T03:29:34.546757+0000","last_unstale":"2023-10-06T03:29:38.592107+0000","last_undegraded":"2023-10-06T03:29:38.592107+0000","last_fullsized":"2023-10-06T03:29:38.592107+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_clean_scrub_stamp":"2023-10-06T03:29:33.500929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T14:20:50.168977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":14,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.600978+0000","last_change":"2023-10-06T03:29:33.537618+0000","last_active":"2023-10-06T03:29:38.600978+0000","last_peered":"2023-10-06T03:29:38.600978+0000","last_clean":"2023-10-06T03:29:38.600978+0000","last_became_active":"2023-10-06T03:29:33.537140+0000","last_became_peered":"2023-10-06T03:29:33.537140+0000","last_unstale":"2023-10-06T03:29:38.600978+0000","last_undegraded":"2023-10-06T03:29:38.600978+0000","last_fullsized":"2023-10-06T03:29:38.600978+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:32.495616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:32.495616+0000","last_clean_scrub_stamp":"2023-10-06T03:29:32.495616+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T12:25:10.766051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.623146+0000","last_change":"2023-10-06T03:29:38.616303+0000","last_active":"2023-10-06T03:29:38.623146+0000","last_peered":"2023-10-06T03:29:38.623146+0000","last_clean":"2023-10-06T03:29:38.623146+0000","last_became_active":"2023-10-06T03:29:38.615882+0000","last_became_peered":"2023-10-06T03:29:38.615882+0000","last_unstale":"2023-10-06T03:29:38.623146+0000","last_undegraded":"2023-10-06T03:29:38.623146+0000","last_fullsized":"2023-10-06T03:29:38.623146+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:37.570693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:37.570693+0000","last_clean_scrub_stamp":"2023-10-06T03:29:37.570693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T11:19:30.457497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3,6],"acting":[2,4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.610059+0000","last_change":"2023-10-06T03:29:34.546395+0000","last_active":"2023-10-06T03:29:38.610059+0000","last_peered":"2023-10-06T03:29:38.610059+0000","last_clean":"2023-10-06T03:29:38.610059+0000","last_became_active":"2023-10-06T03:29:34.546016+0000","last_became_peered":"2023-10-06T03:29:34.546016+0000","last_unstale":"2023-10-06T03:29:38.610059+0000","last_undegraded":"2023-10-06T03:29:38.610059+0000","last_fullsized":"2023-10-06T03:29:38.610059+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_clean_scrub_stamp":"2023-10-06T03:29:33.500929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T14:51:50.895276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"23'1","reported_seq":14,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.605833+0000","last_change":"2023-10-06T03:29:33.536994+0000","last_active":"2023-10-06T03:29:38.605833+0000","last_peered":"2023-10-06T03:29:38.605833+0000","last_clean":"2023-10-06T03:29:38.605833+0000","last_became_active":"2023-10-06T03:29:33.535768+0000","last_became_peered":"2023-10-06T03:29:33.535768+0000","last_unstale":"2023-10-06T03:29:38.605833+0000","last_undegraded":"2023-10-06T03:29:38.605833+0000","last_fullsized":"2023-10-06T03:29:38.605833+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:32.495616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:32.495616+0000","last_clean_scrub_stamp":"2023-10-06T03:29:32.495616+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T06:53:50.776266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.615680+0000","last_change":"2023-10-06T03:29:38.614617+0000","last_active":"2023-10-06T03:29:38.615680+0000","last_peered":"2023-10-06T03:29:38.615680+0000","last_clean":"2023-10-06T03:29:38.615680+0000","last_became_active":"2023-10-06T03:29:38.614134+0000","last_became_peered":"2023-10-06T03:29:38.614134+0000","last_unstale":"2023-10-06T03:29:38.615680+0000","last_undegraded":"2023-10-06T03:29:38.615680+0000","last_fullsized":"2023-10-06T03:29:38.615680+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:37.570693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:37.570693+0000","last_clean_scrub_stamp":"2023-10-06T03:29:37.570693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T07:51:25.532733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5,6],"acting":[2,0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.592039+0000","last_change":"2023-10-06T03:29:34.535355+0000","last_active":"2023-10-06T03:29:38.592039+0000","last_peered":"2023-10-06T03:29:38.592039+0000","last_clean":"2023-10-06T03:29:38.592039+0000","last_became_active":"2023-10-06T03:29:34.534944+0000","last_became_peered":"2023-10-06T03:29:34.534944+0000","last_unstale":"2023-10-06T03:29:38.592039+0000","last_undegraded":"2023-10-06T03:29:38.592039+0000","last_fullsized":"2023-10-06T03:29:38.592039+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_clean_scrub_stamp":"2023-10-06T03:29:33.500929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T09:39:33.168497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"23'2","reported_seq":15,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.605956+0000","last_change":"2023-10-06T03:29:33.545575+0000","last_active":"2023-10-06T03:29:38.605956+0000","last_peered":"2023-10-06T03:29:38.605956+0000","last_clean":"2023-10-06T03:29:38.605956+0000","last_became_active":"2023-10-06T03:29:33.545110+0000","last_became_peered":"2023-10-06T03:29:33.545110+0000","last_unstale":"2023-10-06T03:29:38.605956+0000","last_undegraded":"2023-10-06T03:29:38.605956+0000","last_fullsized":"2023-10-06T03:29:38.605956+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:32.495616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:32.495616+0000","last_clean_scrub_stamp":"2023-10-06T03:29:32.495616+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T14:51:49.889964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.623767+0000","last_change":"2023-10-06T03:29:38.620856+0000","last_active":"2023-10-06T03:29:38.623767+0000","last_peered":"2023-10-06T03:29:38.623767+0000","last_clean":"2023-10-06T03:29:38.623767+0000","last_became_active":"2023-10-06T03:29:38.620374+0000","last_became_peered":"2023-10-06T03:29:38.620374+0000","last_unstale":"2023-10-06T03:29:38.623767+0000","last_undegraded":"2023-10-06T03:29:38.623767+0000","last_fullsized":"2023-10-06T03:29:38.623767+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:37.570693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:37.570693+0000","last_clean_scrub_stamp":"2023-10-06T03:29:37.570693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T08:05:22.622058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1,5],"acting":[2,0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.600871+0000","last_change":"2023-10-06T03:29:34.537767+0000","last_active":"2023-10-06T03:29:38.600871+0000","last_peered":"2023-10-06T03:29:38.600871+0000","last_clean":"2023-10-06T03:29:38.600871+0000","last_became_active":"2023-10-06T03:29:34.537414+0000","last_became_peered":"2023-10-06T03:29:34.537414+0000","last_unstale":"2023-10-06T03:29:38.600871+0000","last_undegraded":"2023-10-06T03:29:38.600871+0000","last_fullsized":"2023-10-06T03:29:38.600871+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_clean_scrub_stamp":"2023-10-06T03:29:33.500929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T08:29:43.429905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"23'1","reported_seq":14,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.606271+0000","last_change":"2023-10-06T03:29:33.543577+0000","last_active":"2023-10-06T03:29:38.606271+0000","last_peered":"2023-10-06T03:29:38.606271+0000","last_clean":"2023-10-06T03:29:38.606271+0000","last_became_active":"2023-10-06T03:29:33.543265+0000","last_became_peered":"2023-10-06T03:29:33.543265+0000","last_unstale":"2023-10-06T03:29:38.606271+0000","last_undegraded":"2023-10-06T03:29:38.606271+0000","last_fullsized":"2023-10-06T03:29:38.606271+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:32.495616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:32.495616+0000","last_clean_scrub_stamp":"2023-10-06T03:29:32.495616+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T09:06:07.838690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.623296+0000","last_change":"2023-10-06T03:29:38.619910+0000","last_active":"2023-10-06T03:29:38.623296+0000","last_peered":"2023-10-06T03:29:38.623296+0000","last_clean":"2023-10-06T03:29:38.623296+0000","last_became_active":"2023-10-06T03:29:38.619517+0000","last_became_peered":"2023-10-06T03:29:38.619517+0000","last_unstale":"2023-10-06T03:29:38.623296+0000","last_undegraded":"2023-10-06T03:29:38.623296+0000","last_fullsized":"2023-10-06T03:29:38.623296+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:37.570693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:37.570693+0000","last_clean_scrub_stamp":"2023-10-06T03:29:37.570693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T04:29:21.981307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3,4],"acting":[7,6,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.627322+0000","last_change":"2023-10-06T03:29:38.623954+0000","last_active":"2023-10-06T03:29:38.627322+0000","last_peered":"2023-10-06T03:29:38.627322+0000","last_clean":"2023-10-06T03:29:38.627322+0000","last_became_active":"2023-10-06T03:29:38.623537+0000","last_became_peered":"2023-10-06T03:29:38.623537+0000","last_unstale":"2023-10-06T03:29:38.627322+0000","last_undegraded":"2023-10-06T03:29:38.627322+0000","last_fullsized":"2023-10-06T03:29:38.627322+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:37.570693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:37.570693+0000","last_clean_scrub_stamp":"2023-10-06T03:29:37.570693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T07:38:31.118869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1,6],"acting":[5,3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"23'8","reported_seq":21,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.609831+0000","last_change":"2023-10-06T03:29:33.538136+0000","last_active":"2023-10-06T03:29:38.609831+0000","last_peered":"2023-10-06T03:29:38.609831+0000","last_clean":"2023-10-06T03:29:38.609831+0000","last_became_active":"2023-10-06T03:29:33.537656+0000","last_became_peered":"2023-10-06T03:29:33.537656+0000","last_unstale":"2023-10-06T03:29:38.609831+0000","last_undegraded":"2023-10-06T03:29:38.609831+0000","last_fullsized":"2023-10-06T03:29:38.609831+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:32.495616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:32.495616+0000","last_clean_scrub_stamp":"2023-10-06T03:29:32.495616+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T07:23:17.430374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.592883+0000","last_change":"2023-10-06T03:29:34.546204+0000","last_active":"2023-10-06T03:29:38.592883+0000","last_peered":"2023-10-06T03:29:38.592883+0000","last_clean":"2023-10-06T03:29:38.592883+0000","last_became_active":"2023-10-06T03:29:34.545585+0000","last_became_peered":"2023-10-06T03:29:34.545585+0000","last_unstale":"2023-10-06T03:29:38.592883+0000","last_undegraded":"2023-10-06T03:29:38.592883+0000","last_fullsized":"2023-10-06T03:29:38.592883+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_clean_scrub_stamp":"2023-10-06T03:29:33.500929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T05:40:45.388611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.620710+0000","last_change":"2023-10-06T03:29:38.612955+0000","last_active":"2023-10-06T03:29:38.620710+0000","last_peered":"2023-10-06T03:29:38.620710+0000","last_clean":"2023-10-06T03:29:38.620710+0000","last_became_active":"2023-10-06T03:29:38.612404+0000","last_became_peered":"2023-10-06T03:29:38.612404+0000","last_unstale":"2023-10-06T03:29:38.620710+0000","last_undegraded":"2023-10-06T03:29:38.620710+0000","last_fullsized":"2023-10-06T03:29:38.620710+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:37.570693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:37.570693+0000","last_clean_scrub_stamp":"2023-10-06T03:29:37.570693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T08:25:07.069848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7,0],"acting":[3,1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.603401+0000","last_change":"2023-10-06T03:29:33.545324+0000","last_active":"2023-10-06T03:29:38.603401+0000","last_peered":"2023-10-06T03:29:38.603401+0000","last_clean":"2023-10-06T03:29:38.603401+0000","last_became_active":"2023-10-06T03:29:33.545029+0000","last_became_peered":"2023-10-06T03:29:33.545029+0000","last_unstale":"2023-10-06T03:29:38.603401+0000","last_undegraded":"2023-10-06T03:29:38.603401+0000","last_fullsized":"2023-10-06T03:29:38.603401+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:32.495616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:32.495616+0000","last_clean_scrub_stamp":"2023-10-06T03:29:32.495616+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T07:14:02.648208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.599807+0000","last_change":"2023-10-06T03:29:34.535716+0000","last_active":"2023-10-06T03:29:38.599807+0000","last_peered":"2023-10-06T03:29:38.599807+0000","last_clean":"2023-10-06T03:29:38.599807+0000","last_became_active":"2023-10-06T03:29:34.535107+0000","last_became_peered":"2023-10-06T03:29:34.535107+0000","last_unstale":"2023-10-06T03:29:38.599807+0000","last_undegraded":"2023-10-06T03:29:38.599807+0000","last_fullsized":"2023-10-06T03:29:38.599807+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_clean_scrub_stamp":"2023-10-06T03:29:33.500929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T11:19:26.387732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":14,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.591766+0000","last_change":"2023-10-06T03:29:33.541792+0000","last_active":"2023-10-06T03:29:38.591766+0000","last_peered":"2023-10-06T03:29:38.591766+0000","last_clean":"2023-10-06T03:29:38.591766+0000","last_became_active":"2023-10-06T03:29:33.535547+0000","last_became_peered":"2023-10-06T03:29:33.535547+0000","last_unstale":"2023-10-06T03:29:38.591766+0000","last_undegraded":"2023-10-06T03:29:38.591766+0000","last_fullsized":"2023-10-06T03:29:38.591766+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:32.495616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:32.495616+0000","last_clean_scrub_stamp":"2023-10-06T03:29:32.495616+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T07:55:07.787102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.602006+0000","last_change":"2023-10-06T03:29:34.540666+0000","last_active":"2023-10-06T03:29:38.602006+0000","last_peered":"2023-10-06T03:29:38.602006+0000","last_clean":"2023-10-06T03:29:38.602006+0000","last_became_active":"2023-10-06T03:29:34.540176+0000","last_became_peered":"2023-10-06T03:29:34.540176+0000","last_unstale":"2023-10-06T03:29:38.602006+0000","last_undegraded":"2023-10-06T03:29:38.602006+0000","last_fullsized":"2023-10-06T03:29:38.602006+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_clean_scrub_stamp":"2023-10-06T03:29:33.500929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T14:11:35.669872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.617535+0000","last_change":"2023-10-06T03:29:38.610899+0000","last_active":"2023-10-06T03:29:38.617535+0000","last_peered":"2023-10-06T03:29:38.617535+0000","last_clean":"2023-10-06T03:29:38.617535+0000","last_became_active":"2023-10-06T03:29:38.610494+0000","last_became_peered":"2023-10-06T03:29:38.610494+0000","last_unstale":"2023-10-06T03:29:38.617535+0000","last_undegraded":"2023-10-06T03:29:38.617535+0000","last_fullsized":"2023-10-06T03:29:38.617535+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:37.570693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:37.570693+0000","last_clean_scrub_stamp":"2023-10-06T03:29:37.570693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T08:19:47.139035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2,1],"acting":[3,7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"23'2","reported_seq":15,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.604109+0000","last_change":"2023-10-06T03:29:33.544328+0000","last_active":"2023-10-06T03:29:38.604109+0000","last_peered":"2023-10-06T03:29:38.604109+0000","last_clean":"2023-10-06T03:29:38.604109+0000","last_became_active":"2023-10-06T03:29:33.544010+0000","last_became_peered":"2023-10-06T03:29:33.544010+0000","last_unstale":"2023-10-06T03:29:38.604109+0000","last_undegraded":"2023-10-06T03:29:38.604109+0000","last_fullsized":"2023-10-06T03:29:38.604109+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:32.495616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:32.495616+0000","last_clean_scrub_stamp":"2023-10-06T03:29:32.495616+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T11:17:52.119337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.592119+0000","last_change":"2023-10-06T03:29:34.551765+0000","last_active":"2023-10-06T03:29:38.592119+0000","last_peered":"2023-10-06T03:29:38.592119+0000","last_clean":"2023-10-06T03:29:38.592119+0000","last_became_active":"2023-10-06T03:29:34.551437+0000","last_became_peered":"2023-10-06T03:29:34.551437+0000","last_unstale":"2023-10-06T03:29:38.592119+0000","last_undegraded":"2023-10-06T03:29:38.592119+0000","last_fullsized":"2023-10-06T03:29:38.592119+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_clean_scrub_stamp":"2023-10-06T03:29:33.500929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T11:47:20.603758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.629984+0000","last_change":"2023-10-06T03:29:38.626781+0000","last_active":"2023-10-06T03:29:38.629984+0000","last_peered":"2023-10-06T03:29:38.629984+0000","last_clean":"2023-10-06T03:29:38.629984+0000","last_became_active":"2023-10-06T03:29:38.625489+0000","last_became_peered":"2023-10-06T03:29:38.625489+0000","last_unstale":"2023-10-06T03:29:38.629984+0000","last_undegraded":"2023-10-06T03:29:38.629984+0000","last_fullsized":"2023-10-06T03:29:38.629984+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:37.570693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:37.570693+0000","last_clean_scrub_stamp":"2023-10-06T03:29:37.570693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T06:13:47.912662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3,0],"acting":[5,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.20","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.609684+0000","last_change":"2023-10-06T03:29:34.546226+0000","last_active":"2023-10-06T03:29:38.609684+0000","last_peered":"2023-10-06T03:29:38.609684+0000","last_clean":"2023-10-06T03:29:38.609684+0000","last_became_active":"2023-10-06T03:29:34.545400+0000","last_became_peered":"2023-10-06T03:29:34.545400+0000","last_unstale":"2023-10-06T03:29:38.609684+0000","last_undegraded":"2023-10-06T03:29:38.609684+0000","last_fullsized":"2023-10-06T03:29:38.609684+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_clean_scrub_stamp":"2023-10-06T03:29:33.500929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T07:55:08.792414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.21","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.588987+0000","last_change":"2023-10-06T03:29:34.551672+0000","last_active":"2023-10-06T03:29:38.588987+0000","last_peered":"2023-10-06T03:29:38.588987+0000","last_clean":"2023-10-06T03:29:38.588987+0000","last_became_active":"2023-10-06T03:29:34.551337+0000","last_became_peered":"2023-10-06T03:29:34.551337+0000","last_unstale":"2023-10-06T03:29:38.588987+0000","last_undegraded":"2023-10-06T03:29:38.588987+0000","last_fullsized":"2023-10-06T03:29:38.588987+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_clean_scrub_stamp":"2023-10-06T03:29:33.500929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T05:15:29.367282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.22","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.592542+0000","last_change":"2023-10-06T03:29:34.549749+0000","last_active":"2023-10-06T03:29:38.592542+0000","last_peered":"2023-10-06T03:29:38.592542+0000","last_clean":"2023-10-06T03:29:38.592542+0000","last_became_active":"2023-10-06T03:29:34.549468+0000","last_became_peered":"2023-10-06T03:29:34.549468+0000","last_unstale":"2023-10-06T03:29:38.592542+0000","last_undegraded":"2023-10-06T03:29:38.592542+0000","last_fullsized":"2023-10-06T03:29:38.592542+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_clean_scrub_stamp":"2023-10-06T03:29:33.500929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T14:11:35.669872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.23","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.590075+0000","last_change":"2023-10-06T03:29:34.545511+0000","last_active":"2023-10-06T03:29:38.590075+0000","last_peered":"2023-10-06T03:29:38.590075+0000","last_clean":"2023-10-06T03:29:38.590075+0000","last_became_active":"2023-10-06T03:29:34.545183+0000","last_became_peered":"2023-10-06T03:29:34.545183+0000","last_unstale":"2023-10-06T03:29:38.590075+0000","last_undegraded":"2023-10-06T03:29:38.590075+0000","last_fullsized":"2023-10-06T03:29:38.590075+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_clean_scrub_stamp":"2023-10-06T03:29:33.500929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T07:54:17.345077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.24","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.609286+0000","last_change":"2023-10-06T03:29:34.543928+0000","last_active":"2023-10-06T03:29:38.609286+0000","last_peered":"2023-10-06T03:29:38.609286+0000","last_clean":"2023-10-06T03:29:38.609286+0000","last_became_active":"2023-10-06T03:29:34.543604+0000","last_became_peered":"2023-10-06T03:29:34.543604+0000","last_unstale":"2023-10-06T03:29:38.609286+0000","last_undegraded":"2023-10-06T03:29:38.609286+0000","last_fullsized":"2023-10-06T03:29:38.609286+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_clean_scrub_stamp":"2023-10-06T03:29:33.500929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T09:15:40.932270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.25","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.601918+0000","last_change":"2023-10-06T03:29:34.540446+0000","last_active":"2023-10-06T03:29:38.601918+0000","last_peered":"2023-10-06T03:29:38.601918+0000","last_clean":"2023-10-06T03:29:38.601918+0000","last_became_active":"2023-10-06T03:29:34.539827+0000","last_became_peered":"2023-10-06T03:29:34.539827+0000","last_unstale":"2023-10-06T03:29:38.601918+0000","last_undegraded":"2023-10-06T03:29:38.601918+0000","last_fullsized":"2023-10-06T03:29:38.601918+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_clean_scrub_stamp":"2023-10-06T03:29:33.500929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T07:28:42.661588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.26","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.609415+0000","last_change":"2023-10-06T03:29:34.538237+0000","last_active":"2023-10-06T03:29:38.609415+0000","last_peered":"2023-10-06T03:29:38.609415+0000","last_clean":"2023-10-06T03:29:38.609415+0000","last_became_active":"2023-10-06T03:29:34.537636+0000","last_became_peered":"2023-10-06T03:29:34.537636+0000","last_unstale":"2023-10-06T03:29:38.609415+0000","last_undegraded":"2023-10-06T03:29:38.609415+0000","last_fullsized":"2023-10-06T03:29:38.609415+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_clean_scrub_stamp":"2023-10-06T03:29:33.500929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T05:09:24.657371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.27","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.608601+0000","last_change":"2023-10-06T03:29:34.549127+0000","last_active":"2023-10-06T03:29:38.608601+0000","last_peered":"2023-10-06T03:29:38.608601+0000","last_clean":"2023-10-06T03:29:38.608601+0000","last_became_active":"2023-10-06T03:29:34.548839+0000","last_became_peered":"2023-10-06T03:29:34.548839+0000","last_unstale":"2023-10-06T03:29:38.608601+0000","last_undegraded":"2023-10-06T03:29:38.608601+0000","last_fullsized":"2023-10-06T03:29:38.608601+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_clean_scrub_stamp":"2023-10-06T03:29:33.500929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T08:03:23.482409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.28","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.603462+0000","last_change":"2023-10-06T03:29:34.544177+0000","last_active":"2023-10-06T03:29:38.603462+0000","last_peered":"2023-10-06T03:29:38.603462+0000","last_clean":"2023-10-06T03:29:38.603462+0000","last_became_active":"2023-10-06T03:29:34.543872+0000","last_became_peered":"2023-10-06T03:29:34.543872+0000","last_unstale":"2023-10-06T03:29:38.603462+0000","last_undegraded":"2023-10-06T03:29:38.603462+0000","last_fullsized":"2023-10-06T03:29:38.603462+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_clean_scrub_stamp":"2023-10-06T03:29:33.500929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T06:09:15.729486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.29","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.588861+0000","last_change":"2023-10-06T03:29:34.540282+0000","last_active":"2023-10-06T03:29:38.588861+0000","last_peered":"2023-10-06T03:29:38.588861+0000","last_clean":"2023-10-06T03:29:38.588861+0000","last_became_active":"2023-10-06T03:29:34.539988+0000","last_became_peered":"2023-10-06T03:29:34.539988+0000","last_unstale":"2023-10-06T03:29:38.588861+0000","last_undegraded":"2023-10-06T03:29:38.588861+0000","last_fullsized":"2023-10-06T03:29:38.588861+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_clean_scrub_stamp":"2023-10-06T03:29:33.500929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T07:36:17.771693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.2a","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.591640+0000","last_change":"2023-10-06T03:29:34.547360+0000","last_active":"2023-10-06T03:29:38.591640+0000","last_peered":"2023-10-06T03:29:38.591640+0000","last_clean":"2023-10-06T03:29:38.591640+0000","last_became_active":"2023-10-06T03:29:34.547086+0000","last_became_peered":"2023-10-06T03:29:34.547086+0000","last_unstale":"2023-10-06T03:29:38.591640+0000","last_undegraded":"2023-10-06T03:29:38.591640+0000","last_fullsized":"2023-10-06T03:29:38.591640+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_clean_scrub_stamp":"2023-10-06T03:29:33.500929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T07:25:41.452975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2b","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.604010+0000","last_change":"2023-10-06T03:29:34.544346+0000","last_active":"2023-10-06T03:29:38.604010+0000","last_peered":"2023-10-06T03:29:38.604010+0000","last_clean":"2023-10-06T03:29:38.604010+0000","last_became_active":"2023-10-06T03:29:34.543976+0000","last_became_peered":"2023-10-06T03:29:34.543976+0000","last_unstale":"2023-10-06T03:29:38.604010+0000","last_undegraded":"2023-10-06T03:29:38.604010+0000","last_fullsized":"2023-10-06T03:29:38.604010+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_clean_scrub_stamp":"2023-10-06T03:29:33.500929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T14:11:35.669872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.2c","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.593246+0000","last_change":"2023-10-06T03:29:34.549013+0000","last_active":"2023-10-06T03:29:38.593246+0000","last_peered":"2023-10-06T03:29:38.593246+0000","last_clean":"2023-10-06T03:29:38.593246+0000","last_became_active":"2023-10-06T03:29:34.548707+0000","last_became_peered":"2023-10-06T03:29:34.548707+0000","last_unstale":"2023-10-06T03:29:38.593246+0000","last_undegraded":"2023-10-06T03:29:38.593246+0000","last_fullsized":"2023-10-06T03:29:38.593246+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_clean_scrub_stamp":"2023-10-06T03:29:33.500929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T14:34:26.405025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2d","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.598606+0000","last_change":"2023-10-06T03:29:34.545639+0000","last_active":"2023-10-06T03:29:38.598606+0000","last_peered":"2023-10-06T03:29:38.598606+0000","last_clean":"2023-10-06T03:29:38.598606+0000","last_became_active":"2023-10-06T03:29:34.545316+0000","last_became_peered":"2023-10-06T03:29:34.545316+0000","last_unstale":"2023-10-06T03:29:38.598606+0000","last_undegraded":"2023-10-06T03:29:38.598606+0000","last_fullsized":"2023-10-06T03:29:38.598606+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_clean_scrub_stamp":"2023-10-06T03:29:33.500929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T11:58:54.143086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2e","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.592479+0000","last_change":"2023-10-06T03:29:34.532132+0000","last_active":"2023-10-06T03:29:38.592479+0000","last_peered":"2023-10-06T03:29:38.592479+0000","last_clean":"2023-10-06T03:29:38.592479+0000","last_became_active":"2023-10-06T03:29:34.531862+0000","last_became_peered":"2023-10-06T03:29:34.531862+0000","last_unstale":"2023-10-06T03:29:38.592479+0000","last_undegraded":"2023-10-06T03:29:38.592479+0000","last_fullsized":"2023-10-06T03:29:38.592479+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_clean_scrub_stamp":"2023-10-06T03:29:33.500929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T07:25:41.452975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2f","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.592223+0000","last_change":"2023-10-06T03:29:34.540499+0000","last_active":"2023-10-06T03:29:38.592223+0000","last_peered":"2023-10-06T03:29:38.592223+0000","last_clean":"2023-10-06T03:29:38.592223+0000","last_became_active":"2023-10-06T03:29:34.539154+0000","last_became_peered":"2023-10-06T03:29:34.539154+0000","last_unstale":"2023-10-06T03:29:38.592223+0000","last_undegraded":"2023-10-06T03:29:38.592223+0000","last_fullsized":"2023-10-06T03:29:38.592223+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_clean_scrub_stamp":"2023-10-06T03:29:33.500929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T15:20:45.655411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.30","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.594589+0000","last_change":"2023-10-06T03:29:34.532489+0000","last_active":"2023-10-06T03:29:38.594589+0000","last_peered":"2023-10-06T03:29:38.594589+0000","last_clean":"2023-10-06T03:29:38.594589+0000","last_became_active":"2023-10-06T03:29:34.532153+0000","last_became_peered":"2023-10-06T03:29:34.532153+0000","last_unstale":"2023-10-06T03:29:38.594589+0000","last_undegraded":"2023-10-06T03:29:38.594589+0000","last_fullsized":"2023-10-06T03:29:38.594589+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_clean_scrub_stamp":"2023-10-06T03:29:33.500929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T08:05:18.552294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.31","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.603291+0000","last_change":"2023-10-06T03:29:34.551157+0000","last_active":"2023-10-06T03:29:38.603291+0000","last_peered":"2023-10-06T03:29:38.603291+0000","last_clean":"2023-10-06T03:29:38.603291+0000","last_became_active":"2023-10-06T03:29:34.550852+0000","last_became_peered":"2023-10-06T03:29:34.550852+0000","last_unstale":"2023-10-06T03:29:38.603291+0000","last_undegraded":"2023-10-06T03:29:38.603291+0000","last_fullsized":"2023-10-06T03:29:38.603291+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_clean_scrub_stamp":"2023-10-06T03:29:33.500929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T08:29:43.429905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.32","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.600883+0000","last_change":"2023-10-06T03:29:34.539103+0000","last_active":"2023-10-06T03:29:38.600883+0000","last_peered":"2023-10-06T03:29:38.600883+0000","last_clean":"2023-10-06T03:29:38.600883+0000","last_became_active":"2023-10-06T03:29:34.538817+0000","last_became_peered":"2023-10-06T03:29:34.538817+0000","last_unstale":"2023-10-06T03:29:38.600883+0000","last_undegraded":"2023-10-06T03:29:38.600883+0000","last_fullsized":"2023-10-06T03:29:38.600883+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_clean_scrub_stamp":"2023-10-06T03:29:33.500929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T07:07:03.705326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.33","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.599036+0000","last_change":"2023-10-06T03:29:34.538671+0000","last_active":"2023-10-06T03:29:38.599036+0000","last_peered":"2023-10-06T03:29:38.599036+0000","last_clean":"2023-10-06T03:29:38.599036+0000","last_became_active":"2023-10-06T03:29:34.538375+0000","last_became_peered":"2023-10-06T03:29:34.538375+0000","last_unstale":"2023-10-06T03:29:38.599036+0000","last_undegraded":"2023-10-06T03:29:38.599036+0000","last_fullsized":"2023-10-06T03:29:38.599036+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_clean_scrub_stamp":"2023-10-06T03:29:33.500929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T07:14:03.653521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.34","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.600214+0000","last_change":"2023-10-06T03:29:34.535139+0000","last_active":"2023-10-06T03:29:38.600214+0000","last_peered":"2023-10-06T03:29:38.600214+0000","last_clean":"2023-10-06T03:29:38.600214+0000","last_became_active":"2023-10-06T03:29:34.534756+0000","last_became_peered":"2023-10-06T03:29:34.534756+0000","last_unstale":"2023-10-06T03:29:38.600214+0000","last_undegraded":"2023-10-06T03:29:38.600214+0000","last_fullsized":"2023-10-06T03:29:38.600214+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_clean_scrub_stamp":"2023-10-06T03:29:33.500929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T14:51:50.895276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.35","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.592690+0000","last_change":"2023-10-06T03:29:34.550626+0000","last_active":"2023-10-06T03:29:38.592690+0000","last_peered":"2023-10-06T03:29:38.592690+0000","last_clean":"2023-10-06T03:29:38.592690+0000","last_became_active":"2023-10-06T03:29:34.550289+0000","last_became_peered":"2023-10-06T03:29:34.550289+0000","last_unstale":"2023-10-06T03:29:38.592690+0000","last_undegraded":"2023-10-06T03:29:38.592690+0000","last_fullsized":"2023-10-06T03:29:38.592690+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_clean_scrub_stamp":"2023-10-06T03:29:33.500929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T12:44:47.758971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.36","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.602276+0000","last_change":"2023-10-06T03:29:34.541344+0000","last_active":"2023-10-06T03:29:38.602276+0000","last_peered":"2023-10-06T03:29:38.602276+0000","last_clean":"2023-10-06T03:29:38.602276+0000","last_became_active":"2023-10-06T03:29:34.540994+0000","last_became_peered":"2023-10-06T03:29:34.540994+0000","last_unstale":"2023-10-06T03:29:38.602276+0000","last_undegraded":"2023-10-06T03:29:38.602276+0000","last_fullsized":"2023-10-06T03:29:38.602276+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_clean_scrub_stamp":"2023-10-06T03:29:33.500929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T05:31:20.131568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.37","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.592106+0000","last_change":"2023-10-06T03:29:34.540530+0000","last_active":"2023-10-06T03:29:38.592106+0000","last_peered":"2023-10-06T03:29:38.592106+0000","last_clean":"2023-10-06T03:29:38.592106+0000","last_became_active":"2023-10-06T03:29:34.539518+0000","last_became_peered":"2023-10-06T03:29:34.539518+0000","last_unstale":"2023-10-06T03:29:38.592106+0000","last_undegraded":"2023-10-06T03:29:38.592106+0000","last_fullsized":"2023-10-06T03:29:38.592106+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_clean_scrub_stamp":"2023-10-06T03:29:33.500929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T14:34:26.405025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.38","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.600763+0000","last_change":"2023-10-06T03:29:34.537333+0000","last_active":"2023-10-06T03:29:38.600763+0000","last_peered":"2023-10-06T03:29:38.600763+0000","last_clean":"2023-10-06T03:29:38.600763+0000","last_became_active":"2023-10-06T03:29:34.536851+0000","last_became_peered":"2023-10-06T03:29:34.536851+0000","last_unstale":"2023-10-06T03:29:38.600763+0000","last_undegraded":"2023-10-06T03:29:38.600763+0000","last_fullsized":"2023-10-06T03:29:38.600763+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_clean_scrub_stamp":"2023-10-06T03:29:33.500929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T14:20:50.168977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.39","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.603186+0000","last_change":"2023-10-06T03:29:34.551594+0000","last_active":"2023-10-06T03:29:38.603186+0000","last_peered":"2023-10-06T03:29:38.603186+0000","last_clean":"2023-10-06T03:29:38.603186+0000","last_became_active":"2023-10-06T03:29:34.551142+0000","last_became_peered":"2023-10-06T03:29:34.551142+0000","last_unstale":"2023-10-06T03:29:38.603186+0000","last_undegraded":"2023-10-06T03:29:38.603186+0000","last_fullsized":"2023-10-06T03:29:38.603186+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_clean_scrub_stamp":"2023-10-06T03:29:33.500929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T07:28:42.661588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3a","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.592149+0000","last_change":"2023-10-06T03:29:34.542011+0000","last_active":"2023-10-06T03:29:38.592149+0000","last_peered":"2023-10-06T03:29:38.592149+0000","last_clean":"2023-10-06T03:29:38.592149+0000","last_became_active":"2023-10-06T03:29:34.540577+0000","last_became_peered":"2023-10-06T03:29:34.540577+0000","last_unstale":"2023-10-06T03:29:38.592149+0000","last_undegraded":"2023-10-06T03:29:38.592149+0000","last_fullsized":"2023-10-06T03:29:38.592149+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_clean_scrub_stamp":"2023-10-06T03:29:33.500929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T12:26:16.082642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.3b","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.593039+0000","last_change":"2023-10-06T03:29:34.538708+0000","last_active":"2023-10-06T03:29:38.593039+0000","last_peered":"2023-10-06T03:29:38.593039+0000","last_clean":"2023-10-06T03:29:38.593039+0000","last_became_active":"2023-10-06T03:29:34.538419+0000","last_became_peered":"2023-10-06T03:29:34.538419+0000","last_unstale":"2023-10-06T03:29:38.593039+0000","last_undegraded":"2023-10-06T03:29:38.593039+0000","last_fullsized":"2023-10-06T03:29:38.593039+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_clean_scrub_stamp":"2023-10-06T03:29:33.500929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T03:45:19.939590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3c","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.609565+0000","last_change":"2023-10-06T03:29:34.546290+0000","last_active":"2023-10-06T03:29:38.609565+0000","last_peered":"2023-10-06T03:29:38.609565+0000","last_clean":"2023-10-06T03:29:38.609565+0000","last_became_active":"2023-10-06T03:29:34.545947+0000","last_became_peered":"2023-10-06T03:29:34.545947+0000","last_unstale":"2023-10-06T03:29:38.609565+0000","last_undegraded":"2023-10-06T03:29:38.609565+0000","last_fullsized":"2023-10-06T03:29:38.609565+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_clean_scrub_stamp":"2023-10-06T03:29:33.500929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T12:25:11.771364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3d","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.592442+0000","last_change":"2023-10-06T03:29:34.541409+0000","last_active":"2023-10-06T03:29:38.592442+0000","last_peered":"2023-10-06T03:29:38.592442+0000","last_clean":"2023-10-06T03:29:38.592442+0000","last_became_active":"2023-10-06T03:29:34.541111+0000","last_became_peered":"2023-10-06T03:29:34.541111+0000","last_unstale":"2023-10-06T03:29:38.592442+0000","last_undegraded":"2023-10-06T03:29:38.592442+0000","last_fullsized":"2023-10-06T03:29:38.592442+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_clean_scrub_stamp":"2023-10-06T03:29:33.500929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T04:15:27.116813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.3e","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.602155+0000","last_change":"2023-10-06T03:29:34.549044+0000","last_active":"2023-10-06T03:29:38.602155+0000","last_peered":"2023-10-06T03:29:38.602155+0000","last_clean":"2023-10-06T03:29:38.602155+0000","last_became_active":"2023-10-06T03:29:34.548691+0000","last_became_peered":"2023-10-06T03:29:34.548691+0000","last_unstale":"2023-10-06T03:29:38.602155+0000","last_undegraded":"2023-10-06T03:29:38.602155+0000","last_fullsized":"2023-10-06T03:29:38.602155+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_clean_scrub_stamp":"2023-10-06T03:29:33.500929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T06:09:15.729486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.3f","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-06T03:29:38.598391+0000","last_change":"2023-10-06T03:29:34.538647+0000","last_active":"2023-10-06T03:29:38.598391+0000","last_peered":"2023-10-06T03:29:38.598391+0000","last_clean":"2023-10-06T03:29:38.598391+0000","last_became_active":"2023-10-06T03:29:34.538343+0000","last_became_peered":"2023-10-06T03:29:34.538343+0000","last_unstale":"2023-10-06T03:29:38.598391+0000","last_undegraded":"2023-10-06T03:29:38.598391+0000","last_fullsized":"2023-10-06T03:29:38.598391+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:33.500929+0000","last_clean_scrub_stamp":"2023-10-06T03:29:33.500929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T12:25:11.771364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":4,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":4484,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":48,"num_write_kb":17,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":8968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":64,"acting":64,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":6,"up_from":16,"seq":68719476746,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297572,"kb_used_data":284,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074268,"statfs":{"total":96636764160,"available":96332050432,"internally_reserved":0,"allocated":290816,"data_stored":120026,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0549999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97399999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92500000000000004}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.014}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2869999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.333}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3160000000000001}]}]},{"osd":5,"up_from":16,"seq":68719476746,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297580,"kb_used_data":292,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074260,"statfs":{"total":96636764160,"available":96332042240,"internally_reserved":0,"allocated":299008,"data_stored":120650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59699999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0720000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.99299999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85299999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81499999999999995}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.171}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.153}]}]},{"osd":4,"up_from":16,"seq":68719476745,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297592,"kb_used_data":304,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074248,"statfs":{"total":96636764160,"available":96332029952,"internally_reserved":0,"allocated":311296,"data_stored":123796,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64300000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59499999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1910000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.99199999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.91300000000000003}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95999999999999996}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2609999999999999}]}]},{"osd":7,"up_from":15,"seq":64424509449,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298180,"kb_used_data":892,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94073660,"statfs":{"total":96636764160,"available":96331427840,"internally_reserved":0,"allocated":913408,"data_stored":714324,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63100000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.875}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72599999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84699999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83199999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.139}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1759999999999999}]}]},{"osd":0,"up_from":15,"seq":64424509450,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302252,"kb_used_data":868,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94069588,"statfs":{"total":96636764160,"available":96327258112,"internally_reserved":0,"allocated":888832,"data_stored":710948,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0029999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.042}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.90200000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80700000000000005}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.026}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.427}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0960000000000001}]}]},{"osd":3,"up_from":14,"seq":60129542153,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":301672,"kb_used_data":288,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94070168,"statfs":{"total":96636764160,"available":96327852032,"internally_reserved":0,"allocated":294912,"data_stored":120116,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49399999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51000000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76200000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81399999999999995}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92400000000000004}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0660000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1479999999999999}]}]},{"osd":2,"up_from":14,"seq":60129542152,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":301668,"kb_used_data":284,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94070172,"statfs":{"total":96636764160,"available":96327856128,"internally_reserved":0,"allocated":290816,"data_stored":120026,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67200000000000004}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76500000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.872}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.077}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0129999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79300000000000004}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.496}]}]},{"osd":1,"up_from":14,"seq":60129542152,"num_pgs":51,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":301668,"kb_used_data":284,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94070172,"statfs":{"total":96636764160,"available":96327856128,"internally_reserved":0,"allocated":290816,"data_stored":120026,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56000000000000005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40999999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.628}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.75800000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82899999999999996}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69199999999999995}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86099999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":554,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":3770,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":3930,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-06T03:29:52.594 INFO:tasks.ceph.ceph_manager.ceph:clean! 2023-10-06T03:29:52.594 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2023-10-06T03:29:52.594 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2023-10-06T03:29:52.594 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-10-06T03:29:52.694 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:52.689+0000 7fcd4b3d0700 1 Processor -- start 2023-10-06T03:29:52.694 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:52.689+0000 7fcd4b3d0700 1 -- start start 2023-10-06T03:29:52.695 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:52.689+0000 7fcd4b3d0700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fcd44158d30 0x7fcd44159100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:52.695 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:52.689+0000 7fcd4b3d0700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fcd4415d5c0 0x7fcd440693e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:52.695 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:52.689+0000 7fcd4b3d0700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fcd440c9e30 0x7fcd440ca2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:52.695 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:52.689+0000 7fcd4b3d0700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcd440ca7e0 con 0x7fcd44158d30 2023-10-06T03:29:52.695 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:52.689+0000 7fcd4b3d0700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcd4406a6f0 con 0x7fcd440c9e30 2023-10-06T03:29:52.696 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:52.689+0000 7fcd4b3d0700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd4415da10 con 0x7fcd4415d5c0 2023-10-06T03:29:52.696 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:52.693+0000 7fcd4a3ce700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fcd44158d30 0x7fcd44159100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:52.696 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:52.693+0000 7fcd4a3ce700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fcd44158d30 0x7fcd44159100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:47652/0 (socket says 172.21.15.110:47652) 2023-10-06T03:29:52.696 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:52.693+0000 7fcd4a3ce700 1 -- 172.21.15.110:0/4023943833 learned_addr learned my addr 172.21.15.110:0/4023943833 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:29:52.696 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:52.693+0000 7fcd49bcd700 1 --2- 172.21.15.110:0/4023943833 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fcd4415d5c0 0x7fcd440693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:52.696 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:52.693+0000 7fcd4abcf700 1 --2- 172.21.15.110:0/4023943833 >> v2:172.21.15.192:3300/0 conn(0x7fcd440c9e30 0x7fcd440ca2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:52.697 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:52.693+0000 7fcd4a3ce700 1 -- 172.21.15.110:0/4023943833 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fcd4415d5c0 msgr2=0x7fcd440693e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:29:52.698 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:52.693+0000 7fcd4a3ce700 1 --2- 172.21.15.110:0/4023943833 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fcd4415d5c0 0x7fcd440693e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:52.698 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:52.693+0000 7fcd4a3ce700 1 -- 172.21.15.110:0/4023943833 >> v2:172.21.15.192:3300/0 conn(0x7fcd440c9e30 msgr2=0x7fcd440ca2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:29:52.698 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:52.693+0000 7fcd4a3ce700 1 --2- 172.21.15.110:0/4023943833 >> v2:172.21.15.192:3300/0 conn(0x7fcd440c9e30 0x7fcd440ca2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:52.698 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:52.693+0000 7fcd4a3ce700 1 -- 172.21.15.110:0/4023943833 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd44069ae0 con 0x7fcd44158d30 2023-10-06T03:29:52.698 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:52.693+0000 7fcd4abcf700 1 --2- 172.21.15.110:0/4023943833 >> v2:172.21.15.192:3300/0 conn(0x7fcd440c9e30 0x7fcd440ca2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-06T03:29:52.698 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:52.693+0000 7fcd4a3ce700 1 --2- 172.21.15.110:0/4023943833 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcd44158d30 0x7fcd44159100 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fcd4415caf0 tx=0x7fcd40009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e4ee57d123d91260 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:29:52.698 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:52.693+0000 7fcd493cc700 1 -- 172.21.15.110:0/4023943833 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd4000b640 con 0x7fcd44158d30 2023-10-06T03:29:52.698 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:52.693+0000 7fcd493cc700 1 -- 172.21.15.110:0/4023943833 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd4000b7a0 con 0x7fcd44158d30 2023-10-06T03:29:52.698 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:52.693+0000 7fcd493cc700 1 -- 172.21.15.110:0/4023943833 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd4000bb60 con 0x7fcd44158d30 2023-10-06T03:29:52.698 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:52.693+0000 7fcd4b3d0700 1 -- 172.21.15.110:0/4023943833 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcd44158d30 msgr2=0x7fcd44159100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:52.698 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:52.693+0000 7fcd4b3d0700 1 --2- 172.21.15.110:0/4023943833 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcd44158d30 0x7fcd44159100 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fcd4415caf0 tx=0x7fcd40009250 comp rx=0 tx=0).stop 2023-10-06T03:29:52.699 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:52.693+0000 7fcd4b3d0700 1 -- 172.21.15.110:0/4023943833 shutdown_connections 2023-10-06T03:29:52.699 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:52.693+0000 7fcd4b3d0700 1 --2- 172.21.15.110:0/4023943833 >> v2:172.21.15.192:3300/0 conn(0x7fcd440c9e30 0x7fcd440ca2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:52.699 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:52.693+0000 7fcd4b3d0700 1 --2- 172.21.15.110:0/4023943833 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fcd4415d5c0 0x7fcd440693e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:52.699 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:52.693+0000 7fcd4b3d0700 1 --2- 172.21.15.110:0/4023943833 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcd44158d30 0x7fcd44159100 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:52.699 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:52.693+0000 7fcd4b3d0700 1 -- 172.21.15.110:0/4023943833 >> 172.21.15.110:0/4023943833 conn(0x7fcd440cc4e0 msgr2=0x7fcd440c8480 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:29:52.699 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:52.693+0000 7fcd4b3d0700 1 -- 172.21.15.110:0/4023943833 shutdown_connections 2023-10-06T03:29:52.699 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:52.693+0000 7fcd4b3d0700 1 -- 172.21.15.110:0/4023943833 wait complete. 2023-10-06T03:29:52.701 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:52.693+0000 7fcd4b3d0700 1 Processor -- start 2023-10-06T03:29:52.701 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:52.693+0000 7fcd4b3d0700 1 -- start start 2023-10-06T03:29:52.701 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:52.697+0000 7fcd4b3d0700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fcd440c9e30 0x7fcd4415f930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:52.701 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:52.697+0000 7fcd4b3d0700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcd44158d30 0x7fcd440ae8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:52.701 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:52.697+0000 7fcd4b3d0700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcd4415d5c0 0x7fcd441527b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:52.702 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:52.697+0000 7fcd4b3d0700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd440aee20 con 0x7fcd4415d5c0 2023-10-06T03:29:52.702 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:52.697+0000 7fcd4b3d0700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd4415a410 con 0x7fcd44158d30 2023-10-06T03:29:52.702 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:52.697+0000 7fcd4b3d0700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd440af050 con 0x7fcd440c9e30 2023-10-06T03:29:52.702 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:52.697+0000 7fcd4abcf700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcd4415d5c0 0x7fcd441527b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:52.702 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:52.697+0000 7fcd4abcf700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcd4415d5c0 0x7fcd441527b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:47662/0 (socket says 172.21.15.110:47662) 2023-10-06T03:29:52.702 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:52.697+0000 7fcd4abcf700 1 -- 172.21.15.110:0/34227388 learned_addr learned my addr 172.21.15.110:0/34227388 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:29:52.702 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:52.697+0000 7fcd4a3ce700 1 --2- 172.21.15.110:0/34227388 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fcd440c9e30 0x7fcd4415f930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:52.702 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:52.697+0000 7fcd49bcd700 1 --2- 172.21.15.110:0/34227388 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcd44158d30 0x7fcd440ae8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:52.703 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:52.697+0000 7fcd4abcf700 1 -- 172.21.15.110:0/34227388 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fcd440c9e30 msgr2=0x7fcd4415f930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:52.703 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:52.697+0000 7fcd4abcf700 1 --2- 172.21.15.110:0/34227388 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fcd440c9e30 0x7fcd4415f930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:52.703 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:52.697+0000 7fcd4abcf700 1 -- 172.21.15.110:0/34227388 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcd44158d30 msgr2=0x7fcd440ae8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:52.703 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:52.697+0000 7fcd4abcf700 1 --2- 172.21.15.110:0/34227388 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcd44158d30 0x7fcd440ae8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:52.703 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:52.697+0000 7fcd4abcf700 1 -- 172.21.15.110:0/34227388 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd3c009a20 con 0x7fcd4415d5c0 2023-10-06T03:29:52.703 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:52.697+0000 7fcd4abcf700 1 --2- 172.21.15.110:0/34227388 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcd4415d5c0 0x7fcd441527b0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fcd3c009fc0 tx=0x7fcd3c00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:29:52.703 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:52.697+0000 7fcd3a7fc700 1 -- 172.21.15.110:0/34227388 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd3c00cbd0 con 0x7fcd4415d5c0 2023-10-06T03:29:52.703 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:52.697+0000 7fcd3a7fc700 1 -- 172.21.15.110:0/34227388 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd3c00f040 con 0x7fcd4415d5c0 2023-10-06T03:29:52.703 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:52.697+0000 7fcd3a7fc700 1 -- 172.21.15.110:0/34227388 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd3c014480 con 0x7fcd4415d5c0 2023-10-06T03:29:52.704 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:52.697+0000 7fcd4b3d0700 1 -- 172.21.15.110:0/34227388 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd400094f0 con 0x7fcd4415d5c0 2023-10-06T03:29:52.704 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:52.697+0000 7fcd4b3d0700 1 -- 172.21.15.110:0/34227388 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd440af610 con 0x7fcd4415d5c0 2023-10-06T03:29:52.707 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:52.697+0000 7fcd4b3d0700 1 -- 172.21.15.110:0/34227388 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd4415a1e0 con 0x7fcd4415d5c0 2023-10-06T03:29:52.707 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:52.701+0000 7fcd3a7fc700 1 -- 172.21.15.110:0/34227388 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7fcd3c003020 con 0x7fcd4415d5c0 2023-10-06T03:29:52.707 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:52.701+0000 7fcd3a7fc700 1 --2- 172.21.15.110:0/34227388 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fcd28060580 0x7fcd28062a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:52.707 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:52.701+0000 7fcd3a7fc700 1 -- 172.21.15.110:0/34227388 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6809+0+0 (secure 0 0 0) 0x7fcd3c07d590 con 0x7fcd4415d5c0 2023-10-06T03:29:52.707 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:52.701+0000 7fcd4a3ce700 1 --2- 172.21.15.110:0/34227388 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fcd28060580 0x7fcd28062a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:52.707 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:52.701+0000 7fcd4a3ce700 1 --2- 172.21.15.110:0/34227388 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fcd28060580 0x7fcd28062a30 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fcd4415caf0 tx=0x7fcd40017000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:29:52.712 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:52.709+0000 7fcd3a7fc700 1 -- 172.21.15.110:0/34227388 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fcd4415a1e0 con 0x7fcd4415d5c0 2023-10-06T03:29:52.888 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:52.885+0000 7fcd4b3d0700 1 -- 172.21.15.110:0/34227388 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fcd4415a1e0 con 0x7fcd4415d5c0 2023-10-06T03:29:52.888 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:52.885+0000 7fcd3a7fc700 1 -- 172.21.15.110:0/34227388 <== mon.0 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (secure 0 0 0) 0x7fcd4415a1e0 con 0x7fcd4415d5c0 2023-10-06T03:29:52.889 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-10-06T03:29:52.889 INFO:teuthology.orchestra.run.smithi110.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2023-10-06T03:29:52.892 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:52.885+0000 7fcd4b3d0700 1 -- 172.21.15.110:0/34227388 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fcd28060580 msgr2=0x7fcd28062a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:52.892 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:52.885+0000 7fcd4b3d0700 1 --2- 172.21.15.110:0/34227388 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fcd28060580 0x7fcd28062a30 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fcd4415caf0 tx=0x7fcd40017000 comp rx=0 tx=0).stop 2023-10-06T03:29:52.892 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:52.885+0000 7fcd4b3d0700 1 -- 172.21.15.110:0/34227388 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcd4415d5c0 msgr2=0x7fcd441527b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:52.892 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:52.885+0000 7fcd4b3d0700 1 --2- 172.21.15.110:0/34227388 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcd4415d5c0 0x7fcd441527b0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fcd3c009fc0 tx=0x7fcd3c00b040 comp rx=0 tx=0).stop 2023-10-06T03:29:52.892 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:52.885+0000 7fcd4b3d0700 1 -- 172.21.15.110:0/34227388 shutdown_connections 2023-10-06T03:29:52.892 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:52.885+0000 7fcd4b3d0700 1 --2- 172.21.15.110:0/34227388 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fcd28060580 0x7fcd28062a30 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:52.892 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:52.885+0000 7fcd4b3d0700 1 --2- 172.21.15.110:0/34227388 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcd4415d5c0 0x7fcd441527b0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:52.893 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:52.885+0000 7fcd4b3d0700 1 --2- 172.21.15.110:0/34227388 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcd44158d30 0x7fcd440ae8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:52.893 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:52.885+0000 7fcd4b3d0700 1 --2- 172.21.15.110:0/34227388 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fcd440c9e30 0x7fcd4415f930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:52.893 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:52.885+0000 7fcd4b3d0700 1 -- 172.21.15.110:0/34227388 >> 172.21.15.110:0/34227388 conn(0x7fcd440cc4e0 msgr2=0x7fcd440c65e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:29:52.893 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:52.885+0000 7fcd4b3d0700 1 -- 172.21.15.110:0/34227388 shutdown_connections 2023-10-06T03:29:52.893 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:52.885+0000 7fcd4b3d0700 1 -- 172.21.15.110:0/34227388 wait complete. 2023-10-06T03:29:52.902 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2023-10-06T03:29:52.902 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-06T03:29:53.006 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.001+0000 7f31509b7700 1 Processor -- start 2023-10-06T03:29:53.006 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.001+0000 7f31509b7700 1 -- start start 2023-10-06T03:29:53.009 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.001+0000 7f31509b7700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f314c158c30 0x7f314c159000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:53.009 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.001+0000 7f31509b7700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f314c0c7710 con 0x7f314c0c7ba0 2023-10-06T03:29:53.009 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.001+0000 7f31509b7700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f314c06cf80 con 0x7f314c06c180 2023-10-06T03:29:53.009 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.001+0000 7f31509b7700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f314c06cb20 con 0x7f314c158c30 2023-10-06T03:29:53.010 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.001+0000 7f314bfff700 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f314c0c7ba0 0x7f314c15cac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:40132/0 (socket says 172.21.15.110:40132) 2023-10-06T03:29:53.010 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.001+0000 7f314bfff700 1 -- 172.21.15.110:0/39279608 learned_addr learned my addr 172.21.15.110:0/39279608 (peer_addr_for_me v1:172.21.15.110:0/0) 2023-10-06T03:29:53.010 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.001+0000 7f314a7fc700 1 -- 172.21.15.110:0/39279608 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1734672236 0 0) 0x7f314c0c7710 con 0x7f314c0c7ba0 2023-10-06T03:29:53.010 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.001+0000 7f314a7fc700 1 -- 172.21.15.110:0/39279608 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3138003570 con 0x7f314c0c7ba0 2023-10-06T03:29:53.010 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.001+0000 7f314b7fe700 1 --2- 172.21.15.110:0/39279608 >> v2:172.21.15.192:3300/0 conn(0x7f314c158c30 0x7f314c159000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:53.010 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.005+0000 7f314a7fc700 1 -- 172.21.15.110:0/39279608 <== mon.0 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 993122818 0 0) 0x7f3138003570 con 0x7f314c0c7ba0 2023-10-06T03:29:53.010 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.005+0000 7f314a7fc700 1 -- 172.21.15.110:0/39279608 >> v2:172.21.15.192:3300/0 conn(0x7f314c158c30 msgr2=0x7f314c159000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:29:53.010 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.005+0000 7f314a7fc700 1 --2- 172.21.15.110:0/39279608 >> v2:172.21.15.192:3300/0 conn(0x7f314c158c30 0x7f314c159000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:53.010 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.005+0000 7f314a7fc700 1 -- 172.21.15.110:0/39279608 >> v1:172.21.15.192:6789/0 conn(0x7f314c06c180 legacy=0x7f314c0670a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:53.010 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.005+0000 7f314a7fc700 1 -- 172.21.15.110:0/39279608 --> v1:172.21.15.110:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f314c06c5d0 con 0x7f314c0c7ba0 2023-10-06T03:29:53.011 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.005+0000 7f314a7fc700 1 -- 172.21.15.110:0/39279608 <== mon.0 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 118603703 0 0) 0x7f31440017f0 con 0x7f314c0c7ba0 2023-10-06T03:29:53.011 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.005+0000 7f314a7fc700 1 -- 172.21.15.110:0/39279608 >> v1:172.21.15.110:6789/0 conn(0x7f314c0c7ba0 legacy=0x7f314c15cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:53.011 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.005+0000 7f314a7fc700 1 --2- 172.21.15.110:0/39279608 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3138003a60 0x7f3138003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:53.011 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.005+0000 7f314a7fc700 1 --2- 172.21.15.110:0/39279608 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3138004810 0x7f3138004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:53.011 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.005+0000 7f314a7fc700 1 --2- 172.21.15.110:0/39279608 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f31380052e0 0x7f313800ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:53.011 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.005+0000 7f314a7fc700 1 -- 172.21.15.110:0/39279608 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f314c06cb20 con 0x7f3138003a60 2023-10-06T03:29:53.011 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.005+0000 7f314a7fc700 1 -- 172.21.15.110:0/39279608 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f313800c230 con 0x7f31380052e0 2023-10-06T03:29:53.011 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.005+0000 7f314a7fc700 1 -- 172.21.15.110:0/39279608 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f313800c3f0 con 0x7f3138004810 2023-10-06T03:29:53.011 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.005+0000 7f314b7fe700 1 --2- 172.21.15.110:0/39279608 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3138003a60 0x7f3138003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:53.011 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.005+0000 7f314bfff700 1 --2- 172.21.15.110:0/39279608 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f31380052e0 0x7f313800ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:53.012 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.005+0000 7f314affd700 1 --2- 172.21.15.110:0/39279608 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3138004810 0x7f3138004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:53.012 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.005+0000 7f314b7fe700 1 -- 172.21.15.110:0/39279608 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3138004810 msgr2=0x7f3138004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:29:53.012 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.005+0000 7f314b7fe700 1 --2- 172.21.15.110:0/39279608 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3138004810 0x7f3138004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:53.012 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.005+0000 7f314b7fe700 1 -- 172.21.15.110:0/39279608 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f31380052e0 msgr2=0x7f313800ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:29:53.012 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.005+0000 7f314b7fe700 1 --2- 172.21.15.110:0/39279608 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f31380052e0 0x7f313800ba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:53.012 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.005+0000 7f314b7fe700 1 -- 172.21.15.110:0/39279608 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3138003570 con 0x7f3138003a60 2023-10-06T03:29:53.012 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.005+0000 7f314bfff700 1 --2- 172.21.15.110:0/39279608 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f31380052e0 0x7f313800ba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-06T03:29:53.012 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.005+0000 7f314bfff700 1 -- 172.21.15.110:0/39279608 reap_dead start 2023-10-06T03:29:53.013 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.005+0000 7f314b7fe700 1 --2- 172.21.15.110:0/39279608 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3138003a60 0x7f3138003e70 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f313c00bb70 tx=0x7f313c008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aba66b44c701cc8f server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:29:53.013 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.005+0000 7f314a7fc700 1 -- 172.21.15.110:0/39279608 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f313c005a00 con 0x7f3138003a60 2023-10-06T03:29:53.013 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.005+0000 7f314a7fc700 1 -- 172.21.15.110:0/39279608 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f313c005b60 con 0x7f3138003a60 2023-10-06T03:29:53.013 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.005+0000 7f314a7fc700 1 -- 172.21.15.110:0/39279608 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f313c00d5d0 con 0x7f3138003a60 2023-10-06T03:29:53.013 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.005+0000 7f31509b7700 1 -- 172.21.15.110:0/39279608 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3138003a60 msgr2=0x7f3138003e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:53.013 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.005+0000 7f31509b7700 1 --2- 172.21.15.110:0/39279608 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3138003a60 0x7f3138003e70 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f313c00bb70 tx=0x7f313c008040 comp rx=0 tx=0).stop 2023-10-06T03:29:53.013 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.005+0000 7f31509b7700 1 -- 172.21.15.110:0/39279608 shutdown_connections 2023-10-06T03:29:53.013 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.005+0000 7f31509b7700 1 --2- 172.21.15.110:0/39279608 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3138003a60 0x7f3138003e70 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:53.013 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.005+0000 7f31509b7700 1 -- 172.21.15.110:0/39279608 >> 172.21.15.110:0/39279608 conn(0x7f314c0c8d00 msgr2=0x7f314c0c9100 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:29:53.014 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.005+0000 7f31509b7700 1 -- 172.21.15.110:0/39279608 shutdown_connections 2023-10-06T03:29:53.014 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.005+0000 7f31509b7700 1 -- 172.21.15.110:0/39279608 wait complete. 2023-10-06T03:29:53.016 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.005+0000 7f31509b7700 1 Processor -- start 2023-10-06T03:29:53.016 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.005+0000 7f31509b7700 1 -- start start 2023-10-06T03:29:53.016 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.009+0000 7f31509b7700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3138003a60 0x7f314c1543e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:53.016 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.009+0000 7f31509b7700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f314c0ac4f0 0x7f314c0ac920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:53.016 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.009+0000 7f31509b7700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f314c0a9a10 0x7f314c0a9e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:53.016 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.009+0000 7f31509b7700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f314c0a9580 con 0x7f314c0a9a10 2023-10-06T03:29:53.017 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.009+0000 7f31509b7700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f314c0a8970 con 0x7f314c0ac4f0 2023-10-06T03:29:53.017 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.009+0000 7f31509b7700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f314c0a8c50 con 0x7f3138003a60 2023-10-06T03:29:53.017 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.009+0000 7f314bfff700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f314c0a9a10 0x7f314c0a9e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:53.017 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.009+0000 7f314bfff700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f314c0a9a10 0x7f314c0a9e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:47684/0 (socket says 172.21.15.110:47684) 2023-10-06T03:29:53.017 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.009+0000 7f314bfff700 1 -- 172.21.15.110:0/172889601 learned_addr learned my addr 172.21.15.110:0/172889601 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:29:53.017 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.009+0000 7f314b7fe700 1 --2- 172.21.15.110:0/172889601 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3138003a60 0x7f314c1543e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:53.017 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.009+0000 7f314affd700 1 --2- 172.21.15.110:0/172889601 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f314c0ac4f0 0x7f314c0ac920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:53.017 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.009+0000 7f314bfff700 1 -- 172.21.15.110:0/172889601 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3138003a60 msgr2=0x7f314c1543e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:53.017 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.009+0000 7f314bfff700 1 --2- 172.21.15.110:0/172889601 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3138003a60 0x7f314c1543e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:53.017 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.009+0000 7f314bfff700 1 -- 172.21.15.110:0/172889601 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f314c0ac4f0 msgr2=0x7f314c0ac920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:53.017 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.009+0000 7f314bfff700 1 --2- 172.21.15.110:0/172889601 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f314c0ac4f0 0x7f314c0ac920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:53.018 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.009+0000 7f314bfff700 1 -- 172.21.15.110:0/172889601 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f314c0a8f30 con 0x7f314c0a9a10 2023-10-06T03:29:53.018 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.009+0000 7f314bfff700 1 --2- 172.21.15.110:0/172889601 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f314c0a9a10 0x7f314c0a9e60 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f314c06f7a0 tx=0x7f3144001f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:29:53.018 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.009+0000 7f31437fe700 1 -- 172.21.15.110:0/172889601 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f314400eab0 con 0x7f314c0a9a10 2023-10-06T03:29:53.018 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.009+0000 7f31437fe700 1 -- 172.21.15.110:0/172889601 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f314400ec30 con 0x7f314c0a9a10 2023-10-06T03:29:53.018 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.009+0000 7f31509b7700 1 -- 172.21.15.110:0/172889601 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f314c0a9210 con 0x7f314c0a9a10 2023-10-06T03:29:53.018 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.009+0000 7f31437fe700 1 -- 172.21.15.110:0/172889601 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3144013420 con 0x7f314c0a9a10 2023-10-06T03:29:53.018 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.009+0000 7f31509b7700 1 -- 172.21.15.110:0/172889601 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f314c154920 con 0x7f314c0a9a10 2023-10-06T03:29:53.018 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.009+0000 7f31509b7700 1 -- 172.21.15.110:0/172889601 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f314c070ff0 con 0x7f314c0a9a10 2023-10-06T03:29:53.019 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.013+0000 7f31437fe700 1 -- 172.21.15.110:0/172889601 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7f3144013790 con 0x7f314c0a9a10 2023-10-06T03:29:53.019 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.013+0000 7f31437fe700 1 --2- 172.21.15.110:0/172889601 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f312c060550 0x7f312c062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:53.019 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.013+0000 7f31437fe700 1 -- 172.21.15.110:0/172889601 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6809+0+0 (secure 0 0 0) 0x7f314407dbb0 con 0x7f314c0a9a10 2023-10-06T03:29:53.019 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.013+0000 7f314b7fe700 1 --2- 172.21.15.110:0/172889601 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f312c060550 0x7f312c062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:53.019 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.013+0000 7f314b7fe700 1 --2- 172.21.15.110:0/172889601 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f312c060550 0x7f312c062a00 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f313c00b5e0 tx=0x7f313c01a000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:29:53.023 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.021+0000 7f31437fe700 1 -- 172.21.15.110:0/172889601 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f314404ddd0 con 0x7f314c0a9a10 2023-10-06T03:29:53.178 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.173+0000 7f31509b7700 1 -- 172.21.15.110:0/172889601 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f314c0a96c0 con 0x7f314c0a9a10 2023-10-06T03:29:53.179 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.177+0000 7f31437fe700 1 -- 172.21.15.110:0/172889601 <== mon.0 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v28) v1 ==== 74+0+18755 (secure 0 0 0) 0x7f31440513f0 con 0x7f314c0a9a10 2023-10-06T03:29:53.180 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-10-06T03:29:53.180 INFO:teuthology.orchestra.run.smithi110.stdout:{"epoch":28,"fsid":"93b8f6ac-4c92-4b45-b8e4-b59617024aa2","created":"2023-10-06T03:29:10.519583+0000","modified":"2023-10-06T03:29:38.951740+0000","last_up_change":"2023-10-06T03:29:21.960511+0000","last_in_change":"2023-10-06T03:29:12.560436+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-06T03:29:20.703743+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2023-10-06T03:29:28.307281+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2023-10-06T03:29:32.247251+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data","create_time":"2023-10-06T03:29:32.880086+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}}},{"pool":5,"pool_name":"cephfs_data_ec","create_time":"2023-10-06T03:29:36.613779+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"28","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"5866eb6c-2adc-46e4-be7a-e5bcda0f4065","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6824","nonce":33394},{"type":"v1","addr":"172.21.15.110:6825","nonce":33394}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6826","nonce":33394},{"type":"v1","addr":"172.21.15.110:6827","nonce":33394}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6830","nonce":33394},{"type":"v1","addr":"172.21.15.110:6831","nonce":33394}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6828","nonce":33394},{"type":"v1","addr":"172.21.15.110:6829","nonce":33394}]},"public_addr":"172.21.15.110:6825/33394","cluster_addr":"172.21.15.110:6827/33394","heartbeat_back_addr":"172.21.15.110:6831/33394","heartbeat_front_addr":"172.21.15.110:6829/33394","state":["exists","up"]},{"osd":1,"uuid":"a38de118-062b-4a05-9782-2ceddf6ff4df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6800","nonce":33396},{"type":"v1","addr":"172.21.15.110:6801","nonce":33396}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6802","nonce":33396},{"type":"v1","addr":"172.21.15.110:6803","nonce":33396}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6806","nonce":33396},{"type":"v1","addr":"172.21.15.110:6807","nonce":33396}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6804","nonce":33396},{"type":"v1","addr":"172.21.15.110:6805","nonce":33396}]},"public_addr":"172.21.15.110:6801/33396","cluster_addr":"172.21.15.110:6803/33396","heartbeat_back_addr":"172.21.15.110:6807/33396","heartbeat_front_addr":"172.21.15.110:6805/33396","state":["exists","up"]},{"osd":2,"uuid":"56e39be2-3407-4b69-877e-cf16617b06d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6816","nonce":33397},{"type":"v1","addr":"172.21.15.110:6817","nonce":33397}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6818","nonce":33397},{"type":"v1","addr":"172.21.15.110:6819","nonce":33397}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6822","nonce":33397},{"type":"v1","addr":"172.21.15.110:6823","nonce":33397}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6820","nonce":33397},{"type":"v1","addr":"172.21.15.110:6821","nonce":33397}]},"public_addr":"172.21.15.110:6817/33397","cluster_addr":"172.21.15.110:6819/33397","heartbeat_back_addr":"172.21.15.110:6823/33397","heartbeat_front_addr":"172.21.15.110:6821/33397","state":["exists","up"]},{"osd":3,"uuid":"01c82d13-c1d4-4bde-b55e-3e0089b40115","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6808","nonce":33395},{"type":"v1","addr":"172.21.15.110:6809","nonce":33395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6810","nonce":33395},{"type":"v1","addr":"172.21.15.110:6811","nonce":33395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6814","nonce":33395},{"type":"v1","addr":"172.21.15.110:6815","nonce":33395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6812","nonce":33395},{"type":"v1","addr":"172.21.15.110:6813","nonce":33395}]},"public_addr":"172.21.15.110:6809/33395","cluster_addr":"172.21.15.110:6811/33395","heartbeat_back_addr":"172.21.15.110:6815/33395","heartbeat_front_addr":"172.21.15.110:6813/33395","state":["exists","up"]},{"osd":4,"uuid":"86983430-5fa3-4d17-998f-03e2cd843237","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":33683},{"type":"v1","addr":"172.21.15.192:6817","nonce":33683}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":33683},{"type":"v1","addr":"172.21.15.192:6819","nonce":33683}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":33683},{"type":"v1","addr":"172.21.15.192:6823","nonce":33683}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6820","nonce":33683},{"type":"v1","addr":"172.21.15.192:6821","nonce":33683}]},"public_addr":"172.21.15.192:6817/33683","cluster_addr":"172.21.15.192:6819/33683","heartbeat_back_addr":"172.21.15.192:6823/33683","heartbeat_front_addr":"172.21.15.192:6821/33683","state":["exists","up"]},{"osd":5,"uuid":"6945257f-821d-4619-a63f-ebe857eba6f4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":33684},{"type":"v1","addr":"172.21.15.192:6801","nonce":33684}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":33684},{"type":"v1","addr":"172.21.15.192:6803","nonce":33684}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":33684},{"type":"v1","addr":"172.21.15.192:6807","nonce":33684}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6804","nonce":33684},{"type":"v1","addr":"172.21.15.192:6805","nonce":33684}]},"public_addr":"172.21.15.192:6801/33684","cluster_addr":"172.21.15.192:6803/33684","heartbeat_back_addr":"172.21.15.192:6807/33684","heartbeat_front_addr":"172.21.15.192:6805/33684","state":["exists","up"]},{"osd":6,"uuid":"7854d1ae-7060-4780-8f97-73cea5e7a12f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6824","nonce":33686},{"type":"v1","addr":"172.21.15.192:6825","nonce":33686}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":33686},{"type":"v1","addr":"172.21.15.192:6827","nonce":33686}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":33686},{"type":"v1","addr":"172.21.15.192:6831","nonce":33686}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6828","nonce":33686},{"type":"v1","addr":"172.21.15.192:6829","nonce":33686}]},"public_addr":"172.21.15.192:6825/33686","cluster_addr":"172.21.15.192:6827/33686","heartbeat_back_addr":"172.21.15.192:6831/33686","heartbeat_front_addr":"172.21.15.192:6829/33686","state":["exists","up"]},{"osd":7,"uuid":"f1ebdfd5-b859-482b-9f97-c18c4a26f445","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6808","nonce":33685},{"type":"v1","addr":"172.21.15.192:6809","nonce":33685}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":33685},{"type":"v1","addr":"172.21.15.192:6811","nonce":33685}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":33685},{"type":"v1","addr":"172.21.15.192:6815","nonce":33685}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6812","nonce":33685},{"type":"v1","addr":"172.21.15.192:6813","nonce":33685}]},"public_addr":"172.21.15.192:6809/33685","cluster_addr":"172.21.15.192:6811/33685","heartbeat_back_addr":"172.21.15.192:6815/33685","heartbeat_front_addr":"172.21.15.192:6813/33685","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T03:29:18.002048+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T03:29:17.478576+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T03:29:18.034944+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T03:29:17.628513+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T03:29:18.047338+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T03:29:17.772349+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T03:29:17.883958+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T03:29:18.151838+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-06T03:29:53.182 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.177+0000 7f31509b7700 1 -- 172.21.15.110:0/172889601 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f312c060550 msgr2=0x7f312c062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:53.182 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.177+0000 7f31509b7700 1 --2- 172.21.15.110:0/172889601 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f312c060550 0x7f312c062a00 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f313c00b5e0 tx=0x7f313c01a000 comp rx=0 tx=0).stop 2023-10-06T03:29:53.182 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.177+0000 7f31509b7700 1 -- 172.21.15.110:0/172889601 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f314c0a9a10 msgr2=0x7f314c0a9e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:53.182 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.177+0000 7f31509b7700 1 --2- 172.21.15.110:0/172889601 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f314c0a9a10 0x7f314c0a9e60 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f314c06f7a0 tx=0x7f3144001f20 comp rx=0 tx=0).stop 2023-10-06T03:29:53.183 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.177+0000 7f31509b7700 1 -- 172.21.15.110:0/172889601 shutdown_connections 2023-10-06T03:29:53.183 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.177+0000 7f31509b7700 1 --2- 172.21.15.110:0/172889601 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f312c060550 0x7f312c062a00 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:53.183 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.177+0000 7f31509b7700 1 --2- 172.21.15.110:0/172889601 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f314c0a9a10 0x7f314c0a9e60 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:53.183 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.177+0000 7f31509b7700 1 --2- 172.21.15.110:0/172889601 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f314c0ac4f0 0x7f314c0ac920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:53.183 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.177+0000 7f31509b7700 1 --2- 172.21.15.110:0/172889601 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3138003a60 0x7f314c1543e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:53.183 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.177+0000 7f31509b7700 1 -- 172.21.15.110:0/172889601 >> 172.21.15.110:0/172889601 conn(0x7f314c0c8d00 msgr2=0x7f314c0c90d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:29:53.183 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.177+0000 7f31509b7700 1 -- 172.21.15.110:0/172889601 shutdown_connections 2023-10-06T03:29:53.183 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.177+0000 7f31509b7700 1 -- 172.21.15.110:0/172889601 wait complete. 2023-10-06T03:29:53.198 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-06T03:29:20.703743+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-10-06T03:29:28.307281+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 3, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-06T03:29:32.247251+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 4, 'pool_name': 'cephfs_data', 'create_time': '2023-10-06T03:29:32.880086+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}, {'pool': 5, 'pool_name': 'cephfs_data_ec', 'create_time': '2023-10-06T03:29:36.613779+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '28', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-10-06T03:29:53.199 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-06T03:29:53.313 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.309+0000 7f4b7359e700 1 Processor -- start 2023-10-06T03:29:53.313 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.309+0000 7f4b7359e700 1 -- start start 2023-10-06T03:29:53.314 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.309+0000 7f4b7359e700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4b740cb1b0 0x7f4b7415cb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:53.314 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.309+0000 7f4b7359e700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f4b7415d040 0x7f4b740afa60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:53.314 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.309+0000 7f4b7359e700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b740cf5f0 con 0x7f4b740aef40 2023-10-06T03:29:53.315 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.309+0000 7f4b7359e700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4b7415d490 con 0x7f4b7415d040 2023-10-06T03:29:53.315 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.309+0000 7f4b7359e700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b740ce9e0 con 0x7f4b740cb1b0 2023-10-06T03:29:53.315 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.309+0000 7f4b7259c700 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f4b740aef40 0x7f4b740ce200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:40146/0 (socket says 172.21.15.110:40146) 2023-10-06T03:29:53.315 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.309+0000 7f4b7259c700 1 -- 172.21.15.110:0/1851948257 learned_addr learned my addr 172.21.15.110:0/1851948257 (peer_addr_for_me v1:172.21.15.110:0/0) 2023-10-06T03:29:53.315 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.309+0000 7f4b72d9d700 1 --2- 172.21.15.110:0/1851948257 >> v2:172.21.15.110:3300/0 conn(0x7f4b7415d040 0x7f4b740afa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:53.315 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.309+0000 7f4b71d9b700 1 --2- 172.21.15.110:0/1851948257 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4b740cb1b0 0x7f4b7415cb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:53.316 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.309+0000 7f4b72d9d700 1 -- 172.21.15.110:0/1851948257 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4b740cb1b0 msgr2=0x7f4b7415cb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:29:53.316 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.309+0000 7f4b72d9d700 1 --2- 172.21.15.110:0/1851948257 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4b740cb1b0 0x7f4b7415cb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:53.317 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.309+0000 7f4b72d9d700 1 -- 172.21.15.110:0/1851948257 >> v1:172.21.15.110:6789/0 conn(0x7f4b740aef40 legacy=0x7f4b740ce200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:53.317 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.309+0000 7f4b72d9d700 1 -- 172.21.15.110:0/1851948257 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b740cecc0 con 0x7f4b7415d040 2023-10-06T03:29:53.317 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.313+0000 7f4b72d9d700 1 --2- 172.21.15.110:0/1851948257 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4b7415d040 0x7f4b740afa60 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f4b6c009e60 tx=0x7f4b6c009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d3b3cf81b52ceb39 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:29:53.317 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.313+0000 7f4b7159a700 1 -- 172.21.15.110:0/1851948257 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b6c00e070 con 0x7f4b7415d040 2023-10-06T03:29:53.317 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.313+0000 7f4b7159a700 1 -- 172.21.15.110:0/1851948257 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b6c00bb60 con 0x7f4b7415d040 2023-10-06T03:29:53.317 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.313+0000 7f4b7159a700 1 -- 172.21.15.110:0/1851948257 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b6c012430 con 0x7f4b7415d040 2023-10-06T03:29:53.317 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.313+0000 7f4b7359e700 1 -- 172.21.15.110:0/1851948257 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4b7415d040 msgr2=0x7f4b740afa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:53.317 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.313+0000 7f4b7359e700 1 --2- 172.21.15.110:0/1851948257 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4b7415d040 0x7f4b740afa60 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f4b6c009e60 tx=0x7f4b6c009780 comp rx=0 tx=0).stop 2023-10-06T03:29:53.317 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.313+0000 7f4b7359e700 1 -- 172.21.15.110:0/1851948257 shutdown_connections 2023-10-06T03:29:53.318 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.313+0000 7f4b7359e700 1 --2- 172.21.15.110:0/1851948257 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4b7415d040 0x7f4b740afa60 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:53.318 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.313+0000 7f4b7359e700 1 --2- 172.21.15.110:0/1851948257 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4b740cb1b0 0x7f4b7415cb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:53.318 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.313+0000 7f4b7359e700 1 -- 172.21.15.110:0/1851948257 >> 172.21.15.110:0/1851948257 conn(0x7f4b740cfec0 msgr2=0x7f4b740d02c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:29:53.318 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.313+0000 7f4b7359e700 1 -- 172.21.15.110:0/1851948257 shutdown_connections 2023-10-06T03:29:53.318 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.313+0000 7f4b7359e700 1 -- 172.21.15.110:0/1851948257 wait complete. 2023-10-06T03:29:53.319 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.317+0000 7f4b7359e700 1 Processor -- start 2023-10-06T03:29:53.319 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.317+0000 7f4b7359e700 1 -- start start 2023-10-06T03:29:53.323 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.317+0000 7f4b7359e700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4b740aef40 0x7f4b74143dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:53.323 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.317+0000 7f4b7359e700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4b740cb1b0 0x7f4b740c9080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:53.323 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.317+0000 7f4b7359e700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4b7415d040 0x7f4b740c6f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:53.324 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.317+0000 7f4b7359e700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b740ca770 con 0x7f4b7415d040 2023-10-06T03:29:53.324 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.317+0000 7f4b7359e700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b740c9b60 con 0x7f4b740aef40 2023-10-06T03:29:53.324 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.317+0000 7f4b7359e700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b740c9e40 con 0x7f4b740cb1b0 2023-10-06T03:29:53.324 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.317+0000 7f4b72d9d700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4b7415d040 0x7f4b740c6f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:53.324 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.317+0000 7f4b72d9d700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4b7415d040 0x7f4b740c6f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:47704/0 (socket says 172.21.15.110:47704) 2023-10-06T03:29:53.324 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.317+0000 7f4b72d9d700 1 -- 172.21.15.110:0/3789081228 learned_addr learned my addr 172.21.15.110:0/3789081228 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:29:53.324 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.317+0000 7f4b7259c700 1 --2- 172.21.15.110:0/3789081228 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4b740aef40 0x7f4b74143dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:53.325 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.317+0000 7f4b71d9b700 1 --2- 172.21.15.110:0/3789081228 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4b740cb1b0 0x7f4b740c9080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:53.325 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.317+0000 7f4b72d9d700 1 -- 172.21.15.110:0/3789081228 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4b740cb1b0 msgr2=0x7f4b740c9080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:53.325 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.317+0000 7f4b72d9d700 1 --2- 172.21.15.110:0/3789081228 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4b740cb1b0 0x7f4b740c9080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:53.325 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.317+0000 7f4b72d9d700 1 -- 172.21.15.110:0/3789081228 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4b740aef40 msgr2=0x7f4b74143dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:53.325 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.317+0000 7f4b72d9d700 1 --2- 172.21.15.110:0/3789081228 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4b740aef40 0x7f4b74143dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:53.326 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.317+0000 7f4b72d9d700 1 -- 172.21.15.110:0/3789081228 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b6c009a20 con 0x7f4b7415d040 2023-10-06T03:29:53.326 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.317+0000 7f4b72d9d700 1 --2- 172.21.15.110:0/3789081228 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4b7415d040 0x7f4b740c6f70 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f4b6c00b6c0 tx=0x7f4b6c019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:29:53.326 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.317+0000 7f4b627fc700 1 -- 172.21.15.110:0/3789081228 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b6c007000 con 0x7f4b7415d040 2023-10-06T03:29:53.326 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.317+0000 7f4b627fc700 1 -- 172.21.15.110:0/3789081228 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b6c025440 con 0x7f4b7415d040 2023-10-06T03:29:53.326 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.317+0000 7f4b7359e700 1 -- 172.21.15.110:0/3789081228 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b740ca120 con 0x7f4b7415d040 2023-10-06T03:29:53.327 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.317+0000 7f4b627fc700 1 -- 172.21.15.110:0/3789081228 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b6c025800 con 0x7f4b7415d040 2023-10-06T03:29:53.327 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.317+0000 7f4b7359e700 1 -- 172.21.15.110:0/3789081228 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b74144310 con 0x7f4b7415d040 2023-10-06T03:29:53.327 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.321+0000 7f4b7359e700 1 -- 172.21.15.110:0/3789081228 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b740ca920 con 0x7f4b7415d040 2023-10-06T03:29:53.328 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.321+0000 7f4b627fc700 1 -- 172.21.15.110:0/3789081228 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7f4b6c003050 con 0x7f4b7415d040 2023-10-06T03:29:53.328 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.321+0000 7f4b627fc700 1 --2- 172.21.15.110:0/3789081228 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f4b54060550 0x7f4b54062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:53.328 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.321+0000 7f4b7259c700 1 --2- 172.21.15.110:0/3789081228 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f4b54060550 0x7f4b54062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:53.328 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.321+0000 7f4b627fc700 1 -- 172.21.15.110:0/3789081228 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6809+0+0 (secure 0 0 0) 0x7f4b6c07ea90 con 0x7f4b7415d040 2023-10-06T03:29:53.329 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.321+0000 7f4b7259c700 1 --2- 172.21.15.110:0/3789081228 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f4b54060550 0x7f4b54062a00 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f4b68000c00 tx=0x7f4b68009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:29:53.332 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.329+0000 7f4b627fc700 1 -- 172.21.15.110:0/3789081228 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f4b6c04ec10 con 0x7f4b7415d040 2023-10-06T03:29:53.489 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.485+0000 7f4b7359e700 1 -- 172.21.15.110:0/3789081228 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f4b74070ff0 con 0x7f4b7415d040 2023-10-06T03:29:53.490 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.485+0000 7f4b627fc700 1 -- 172.21.15.110:0/3789081228 <== mon.0 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v28) v1 ==== 93+0+10 (secure 0 0 0) 0x7f4b6c052230 con 0x7f4b7415d040 2023-10-06T03:29:53.490 INFO:teuthology.orchestra.run.smithi110.stdout:pg_num: 1 2023-10-06T03:29:53.492 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.489+0000 7f4b7359e700 1 -- 172.21.15.110:0/3789081228 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f4b54060550 msgr2=0x7f4b54062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:53.492 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.489+0000 7f4b7359e700 1 --2- 172.21.15.110:0/3789081228 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f4b54060550 0x7f4b54062a00 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f4b68000c00 tx=0x7f4b68009040 comp rx=0 tx=0).stop 2023-10-06T03:29:53.493 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.489+0000 7f4b7359e700 1 -- 172.21.15.110:0/3789081228 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4b7415d040 msgr2=0x7f4b740c6f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:53.493 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.489+0000 7f4b7359e700 1 --2- 172.21.15.110:0/3789081228 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4b7415d040 0x7f4b740c6f70 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f4b6c00b6c0 tx=0x7f4b6c019000 comp rx=0 tx=0).stop 2023-10-06T03:29:53.493 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.489+0000 7f4b7359e700 1 -- 172.21.15.110:0/3789081228 shutdown_connections 2023-10-06T03:29:53.493 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.489+0000 7f4b7359e700 1 --2- 172.21.15.110:0/3789081228 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f4b54060550 0x7f4b54062a00 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:53.493 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.489+0000 7f4b7359e700 1 --2- 172.21.15.110:0/3789081228 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4b7415d040 0x7f4b740c6f70 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:53.493 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.489+0000 7f4b7359e700 1 --2- 172.21.15.110:0/3789081228 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4b740cb1b0 0x7f4b740c9080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:53.493 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.489+0000 7f4b7359e700 1 --2- 172.21.15.110:0/3789081228 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4b740aef40 0x7f4b74143dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:53.493 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.489+0000 7f4b7359e700 1 -- 172.21.15.110:0/3789081228 >> 172.21.15.110:0/3789081228 conn(0x7f4b740cfec0 msgr2=0x7f4b74158710 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:29:53.493 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.489+0000 7f4b7359e700 1 -- 172.21.15.110:0/3789081228 shutdown_connections 2023-10-06T03:29:53.493 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.489+0000 7f4b7359e700 1 -- 172.21.15.110:0/3789081228 wait complete. 2023-10-06T03:29:53.503 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-10-06T03:29:53.613 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.609+0000 7fba87fff700 1 Processor -- start 2023-10-06T03:29:53.613 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.609+0000 7fba87fff700 1 -- start start 2023-10-06T03:29:53.617 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.609+0000 7fba87fff700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fba8806b7c0 0x7fba880ce170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:53.617 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.609+0000 7fba87fff700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba880cfa00 con 0x7fba8806c2a0 2023-10-06T03:29:53.617 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.609+0000 7fba87fff700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba880cedf0 con 0x7fba880ce6b0 2023-10-06T03:29:53.617 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.609+0000 7fba87fff700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba880cb1c0 con 0x7fba8806b7c0 2023-10-06T03:29:53.617 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.609+0000 7fba877fe700 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7fba8806c2a0 0x7fba88162ac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:40148/0 (socket says 172.21.15.110:40148) 2023-10-06T03:29:53.617 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.609+0000 7fba877fe700 1 -- 172.21.15.110:0/196053964 learned_addr learned my addr 172.21.15.110:0/196053964 (peer_addr_for_me v1:172.21.15.110:0/0) 2023-10-06T03:29:53.618 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.609+0000 7fba86ffd700 1 --2- 172.21.15.110:0/196053964 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fba8806b7c0 0x7fba880ce170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:53.618 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.609+0000 7fba85ffb700 1 -- 172.21.15.110:0/196053964 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 20798193 0 0) 0x7fba880cfa00 con 0x7fba8806c2a0 2023-10-06T03:29:53.618 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.609+0000 7fba85ffb700 1 -- 172.21.15.110:0/196053964 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fba70003570 con 0x7fba8806c2a0 2023-10-06T03:29:53.618 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.613+0000 7fba85ffb700 1 -- 172.21.15.110:0/196053964 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4022768202 0 0) 0x7fba880cedf0 con 0x7fba880ce6b0 2023-10-06T03:29:53.618 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.613+0000 7fba85ffb700 1 -- 172.21.15.110:0/196053964 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fba880cfa00 con 0x7fba880ce6b0 2023-10-06T03:29:53.618 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.613+0000 7fba85ffb700 1 -- 172.21.15.110:0/196053964 <== mon.0 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3265866118 0 0) 0x7fba70003570 con 0x7fba8806c2a0 2023-10-06T03:29:53.618 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.613+0000 7fba85ffb700 1 -- 172.21.15.110:0/196053964 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fba8806b7c0 msgr2=0x7fba880ce170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:29:53.618 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.613+0000 7fba85ffb700 1 --2- 172.21.15.110:0/196053964 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fba8806b7c0 0x7fba880ce170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:53.618 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.613+0000 7fba85ffb700 1 -- 172.21.15.110:0/196053964 >> v1:172.21.15.192:6789/0 conn(0x7fba880ce6b0 legacy=0x7fba8806bb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:53.618 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.613+0000 7fba85ffb700 1 -- 172.21.15.110:0/196053964 --> v1:172.21.15.110:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba880cf0d0 con 0x7fba8806c2a0 2023-10-06T03:29:53.619 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.613+0000 7fba85ffb700 1 -- 172.21.15.110:0/196053964 <== mon.0 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 118603703 0 0) 0x7fba80001470 con 0x7fba8806c2a0 2023-10-06T03:29:53.619 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.613+0000 7fba85ffb700 1 -- 172.21.15.110:0/196053964 >> v1:172.21.15.110:6789/0 conn(0x7fba8806c2a0 legacy=0x7fba88162ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:53.619 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.613+0000 7fba85ffb700 1 --2- 172.21.15.110:0/196053964 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fba70003c20 0x7fba70003ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:53.619 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.613+0000 7fba85ffb700 1 --2- 172.21.15.110:0/196053964 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fba8806b7c0 0x7fba70004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:53.619 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.613+0000 7fba85ffb700 1 --2- 172.21.15.110:0/196053964 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fba70004d50 0x7fba700051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:53.619 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.613+0000 7fba85ffb700 1 -- 172.21.15.110:0/196053964 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba880cb1c0 con 0x7fba70003c20 2023-10-06T03:29:53.619 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.613+0000 7fba85ffb700 1 -- 172.21.15.110:0/196053964 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba7000bb10 con 0x7fba70004d50 2023-10-06T03:29:53.619 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.613+0000 7fba85ffb700 1 -- 172.21.15.110:0/196053964 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba7000bcd0 con 0x7fba8806b7c0 2023-10-06T03:29:53.619 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.613+0000 7fba86ffd700 1 --2- 172.21.15.110:0/196053964 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fba70003c20 0x7fba70003ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:53.619 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.613+0000 7fba867fc700 1 --2- 172.21.15.110:0/196053964 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fba8806b7c0 0x7fba70004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:53.619 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.613+0000 7fba877fe700 1 --2- 172.21.15.110:0/196053964 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fba70004d50 0x7fba700051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:53.620 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.613+0000 7fba86ffd700 1 -- 172.21.15.110:0/196053964 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fba8806b7c0 msgr2=0x7fba70004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:29:53.620 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.613+0000 7fba86ffd700 1 --2- 172.21.15.110:0/196053964 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fba8806b7c0 0x7fba70004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:53.620 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.613+0000 7fba86ffd700 1 -- 172.21.15.110:0/196053964 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fba70004d50 msgr2=0x7fba700051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:29:53.620 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.613+0000 7fba86ffd700 1 --2- 172.21.15.110:0/196053964 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fba70004d50 0x7fba700051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:53.620 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.613+0000 7fba86ffd700 1 -- 172.21.15.110:0/196053964 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba70003570 con 0x7fba70003c20 2023-10-06T03:29:53.620 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.613+0000 7fba867fc700 1 --2- 172.21.15.110:0/196053964 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fba8806b7c0 0x7fba70004810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-06T03:29:53.620 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.613+0000 7fba86ffd700 1 --2- 172.21.15.110:0/196053964 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fba70003c20 0x7fba70003ff0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fba78009c00 tx=0x7fba780091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fec0f522613b5aec server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:29:53.620 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.613+0000 7fba85ffb700 1 -- 172.21.15.110:0/196053964 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba7800e710 con 0x7fba70003c20 2023-10-06T03:29:53.620 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.613+0000 7fba85ffb700 1 -- 172.21.15.110:0/196053964 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba7800e870 con 0x7fba70003c20 2023-10-06T03:29:53.621 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.613+0000 7fba85ffb700 1 -- 172.21.15.110:0/196053964 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba7800ec30 con 0x7fba70003c20 2023-10-06T03:29:53.622 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.613+0000 7fba87fff700 1 -- 172.21.15.110:0/196053964 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fba70003c20 msgr2=0x7fba70003ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:53.622 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.613+0000 7fba87fff700 1 --2- 172.21.15.110:0/196053964 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fba70003c20 0x7fba70003ff0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fba78009c00 tx=0x7fba780091a0 comp rx=0 tx=0).stop 2023-10-06T03:29:53.622 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.613+0000 7fba877fe700 1 -- 172.21.15.110:0/196053964 reap_dead start 2023-10-06T03:29:53.622 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.613+0000 7fba87fff700 1 -- 172.21.15.110:0/196053964 shutdown_connections 2023-10-06T03:29:53.622 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.613+0000 7fba87fff700 1 -- 172.21.15.110:0/196053964 >> 172.21.15.110:0/196053964 conn(0x7fba880cfea0 msgr2=0x7fba880d02a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:29:53.622 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.613+0000 7fba87fff700 1 -- 172.21.15.110:0/196053964 shutdown_connections 2023-10-06T03:29:53.622 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.613+0000 7fba87fff700 1 -- 172.21.15.110:0/196053964 wait complete. 2023-10-06T03:29:53.622 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.617+0000 7fba87fff700 1 Processor -- start 2023-10-06T03:29:53.622 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.617+0000 7fba87fff700 1 -- start start 2023-10-06T03:29:53.624 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.617+0000 7fba87fff700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fba88154c20 0x7fba881591c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:53.624 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.617+0000 7fba87fff700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fba8815df90 0x7fba8815b9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:53.624 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.617+0000 7fba87fff700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fba8815bf30 0x7fba881690c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:53.624 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.617+0000 7fba87fff700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba8815b560 con 0x7fba8815bf30 2023-10-06T03:29:53.624 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.617+0000 7fba87fff700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba88159aa0 con 0x7fba88154c20 2023-10-06T03:29:53.625 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.617+0000 7fba87fff700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba88159d80 con 0x7fba8815df90 2023-10-06T03:29:53.625 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.617+0000 7fba877fe700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fba8815bf30 0x7fba881690c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:53.625 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.617+0000 7fba877fe700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fba8815bf30 0x7fba881690c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:47718/0 (socket says 172.21.15.110:47718) 2023-10-06T03:29:53.625 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.617+0000 7fba877fe700 1 -- 172.21.15.110:0/539485783 learned_addr learned my addr 172.21.15.110:0/539485783 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:29:53.625 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.617+0000 7fba867fc700 1 --2- 172.21.15.110:0/539485783 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fba8815df90 0x7fba8815b9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:53.625 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.617+0000 7fba86ffd700 1 --2- 172.21.15.110:0/539485783 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fba88154c20 0x7fba881591c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:53.625 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.621+0000 7fba877fe700 1 -- 172.21.15.110:0/539485783 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fba8815df90 msgr2=0x7fba8815b9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:53.626 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.621+0000 7fba877fe700 1 --2- 172.21.15.110:0/539485783 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fba8815df90 0x7fba8815b9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:53.626 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.621+0000 7fba877fe700 1 -- 172.21.15.110:0/539485783 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fba88154c20 msgr2=0x7fba881591c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:53.626 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.621+0000 7fba877fe700 1 --2- 172.21.15.110:0/539485783 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fba88154c20 0x7fba881591c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:53.626 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.621+0000 7fba877fe700 1 -- 172.21.15.110:0/539485783 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba8815a060 con 0x7fba8815bf30 2023-10-06T03:29:53.626 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.621+0000 7fba877fe700 1 --2- 172.21.15.110:0/539485783 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fba8815bf30 0x7fba881690c0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fba88088c70 tx=0x7fba80008c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:29:53.626 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.621+0000 7fba6effd700 1 -- 172.21.15.110:0/539485783 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba8000f830 con 0x7fba8815bf30 2023-10-06T03:29:53.626 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.621+0000 7fba6effd700 1 -- 172.21.15.110:0/539485783 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba8000f990 con 0x7fba8815bf30 2023-10-06T03:29:53.626 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.621+0000 7fba6effd700 1 -- 172.21.15.110:0/539485783 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba8000fd50 con 0x7fba8815bf30 2023-10-06T03:29:53.626 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.621+0000 7fba87fff700 1 -- 172.21.15.110:0/539485783 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba8815b1f0 con 0x7fba8815bf30 2023-10-06T03:29:53.627 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.621+0000 7fba87fff700 1 -- 172.21.15.110:0/539485783 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba88159700 con 0x7fba8815bf30 2023-10-06T03:29:53.627 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.621+0000 7fba87fff700 1 -- 172.21.15.110:0/539485783 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba8815b040 con 0x7fba8815bf30 2023-10-06T03:29:53.627 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.621+0000 7fba6effd700 1 -- 172.21.15.110:0/539485783 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7fba80015640 con 0x7fba8815bf30 2023-10-06T03:29:53.628 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.625+0000 7fba6effd700 1 --2- 172.21.15.110:0/539485783 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fba74060550 0x7fba74062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:53.628 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.625+0000 7fba6effd700 1 -- 172.21.15.110:0/539485783 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6809+0+0 (secure 0 0 0) 0x7fba8007c720 con 0x7fba8815bf30 2023-10-06T03:29:53.628 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.625+0000 7fba86ffd700 1 --2- 172.21.15.110:0/539485783 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fba74060550 0x7fba74062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:53.629 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.625+0000 7fba86ffd700 1 --2- 172.21.15.110:0/539485783 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fba74060550 0x7fba74062a00 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fba8806f8a0 tx=0x7fba7801a000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:29:53.633 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.629+0000 7fba6effd700 1 -- 172.21.15.110:0/539485783 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fba8004bbe0 con 0x7fba8815bf30 2023-10-06T03:29:53.789 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.785+0000 7fba87fff700 1 -- 172.21.15.110:0/539485783 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7fba88070ff0 con 0x7fba8815bf30 2023-10-06T03:29:53.790 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.785+0000 7fba6effd700 1 -- 172.21.15.110:0/539485783 <== mon.0 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v28) v1 ==== 92+0+10 (secure 0 0 0) 0x7fba8004c8c0 con 0x7fba8815bf30 2023-10-06T03:29:53.790 INFO:teuthology.orchestra.run.smithi110.stdout:pg_num: 8 2023-10-06T03:29:53.792 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.789+0000 7fba87fff700 1 -- 172.21.15.110:0/539485783 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fba74060550 msgr2=0x7fba74062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:53.792 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.789+0000 7fba87fff700 1 --2- 172.21.15.110:0/539485783 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fba74060550 0x7fba74062a00 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fba8806f8a0 tx=0x7fba7801a000 comp rx=0 tx=0).stop 2023-10-06T03:29:53.792 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.789+0000 7fba87fff700 1 -- 172.21.15.110:0/539485783 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fba8815bf30 msgr2=0x7fba881690c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:53.793 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.789+0000 7fba87fff700 1 --2- 172.21.15.110:0/539485783 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fba8815bf30 0x7fba881690c0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fba88088c70 tx=0x7fba80008c10 comp rx=0 tx=0).stop 2023-10-06T03:29:53.793 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.789+0000 7fba87fff700 1 -- 172.21.15.110:0/539485783 shutdown_connections 2023-10-06T03:29:53.793 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.789+0000 7fba87fff700 1 --2- 172.21.15.110:0/539485783 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fba74060550 0x7fba74062a00 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:53.793 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.789+0000 7fba87fff700 1 --2- 172.21.15.110:0/539485783 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fba8815bf30 0x7fba881690c0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:53.793 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.789+0000 7fba87fff700 1 --2- 172.21.15.110:0/539485783 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fba8815df90 0x7fba8815b9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:53.793 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.789+0000 7fba87fff700 1 --2- 172.21.15.110:0/539485783 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fba88154c20 0x7fba881591c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:53.793 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.789+0000 7fba87fff700 1 -- 172.21.15.110:0/539485783 >> 172.21.15.110:0/539485783 conn(0x7fba880cfea0 msgr2=0x7fba880af950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:29:53.793 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.789+0000 7fba87fff700 1 -- 172.21.15.110:0/539485783 shutdown_connections 2023-10-06T03:29:53.793 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.789+0000 7fba87fff700 1 -- 172.21.15.110:0/539485783 wait complete. 2023-10-06T03:29:53.803 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-06T03:29:53.915 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.913+0000 7f241dd30700 1 Processor -- start 2023-10-06T03:29:53.916 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.913+0000 7f241dd30700 1 -- start start 2023-10-06T03:29:53.918 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.913+0000 7f241dd30700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f2418156cc0 0x7f24180c8a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:53.919 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.913+0000 7f241dd30700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f241815ab40 0x7f241806c7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:53.919 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.913+0000 7f241dd30700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24180672b0 con 0x7f2418157090 2023-10-06T03:29:53.919 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.913+0000 7f241dd30700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f24181574e0 con 0x7f2418156cc0 2023-10-06T03:29:53.919 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.913+0000 7f241dd30700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f241815b2b0 con 0x7f241815ab40 2023-10-06T03:29:53.919 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.913+0000 7f2417fff700 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f2418157090 0x7f24180caee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:40158/0 (socket says 172.21.15.110:40158) 2023-10-06T03:29:53.919 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.913+0000 7f2417fff700 1 -- 172.21.15.110:0/1568516001 learned_addr learned my addr 172.21.15.110:0/1568516001 (peer_addr_for_me v1:172.21.15.110:0/0) 2023-10-06T03:29:53.919 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.913+0000 7f241cd2e700 1 --2- 172.21.15.110:0/1568516001 >> v2:172.21.15.110:3300/0 conn(0x7f2418156cc0 0x7f24180c8a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:53.920 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.913+0000 7f241d52f700 1 --2- 172.21.15.110:0/1568516001 >> v2:172.21.15.192:3300/0 conn(0x7f241815ab40 0x7f241806c7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:53.920 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.913+0000 7f24177fe700 1 -- 172.21.15.110:0/1568516001 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 528358006 0 0) 0x7f24180672b0 con 0x7f2418157090 2023-10-06T03:29:53.920 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.913+0000 7f24177fe700 1 -- 172.21.15.110:0/1568516001 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2408003570 con 0x7f2418157090 2023-10-06T03:29:53.920 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.913+0000 7f241cd2e700 1 -- 172.21.15.110:0/1568516001 >> v2:172.21.15.192:3300/0 conn(0x7f241815ab40 msgr2=0x7f241806c7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:29:53.920 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.913+0000 7f241cd2e700 1 --2- 172.21.15.110:0/1568516001 >> v2:172.21.15.192:3300/0 conn(0x7f241815ab40 0x7f241806c7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:53.920 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.913+0000 7f241cd2e700 1 -- 172.21.15.110:0/1568516001 >> v1:172.21.15.110:6789/0 conn(0x7f2418157090 legacy=0x7f24180caee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:53.920 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.913+0000 7f241cd2e700 1 -- 172.21.15.110:0/1568516001 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f241815b590 con 0x7f2418156cc0 2023-10-06T03:29:53.920 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.913+0000 7f241cd2e700 1 --2- 172.21.15.110:0/1568516001 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2418156cc0 0x7f24180c8a10 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f24180abf80 tx=0x7f240c0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=84045e20656fe835 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:29:53.921 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.913+0000 7f24177fe700 1 -- 172.21.15.110:0/1568516001 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f240c011630 con 0x7f2418156cc0 2023-10-06T03:29:53.921 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.913+0000 7f24177fe700 1 -- 172.21.15.110:0/1568516001 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f240c011790 con 0x7f2418156cc0 2023-10-06T03:29:53.921 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.913+0000 7f24177fe700 1 -- 172.21.15.110:0/1568516001 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f240c011b30 con 0x7f2418156cc0 2023-10-06T03:29:53.922 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.917+0000 7f241dd30700 1 -- 172.21.15.110:0/1568516001 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2418156cc0 msgr2=0x7f24180c8a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:53.922 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.917+0000 7f241dd30700 1 --2- 172.21.15.110:0/1568516001 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2418156cc0 0x7f24180c8a10 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f24180abf80 tx=0x7f240c0091a0 comp rx=0 tx=0).stop 2023-10-06T03:29:53.922 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.917+0000 7f241dd30700 1 -- 172.21.15.110:0/1568516001 shutdown_connections 2023-10-06T03:29:53.923 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.917+0000 7f241dd30700 1 --2- 172.21.15.110:0/1568516001 >> v2:172.21.15.192:3300/0 conn(0x7f241815ab40 0x7f241806c7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:53.923 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.917+0000 7f241dd30700 1 --2- 172.21.15.110:0/1568516001 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2418156cc0 0x7f24180c8a10 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:53.923 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.917+0000 7f241dd30700 1 -- 172.21.15.110:0/1568516001 >> 172.21.15.110:0/1568516001 conn(0x7f241806ab30 msgr2=0x7f24180adff0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:29:53.923 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.917+0000 7f241dd30700 1 -- 172.21.15.110:0/1568516001 shutdown_connections 2023-10-06T03:29:53.924 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.917+0000 7f241dd30700 1 -- 172.21.15.110:0/1568516001 wait complete. 2023-10-06T03:29:53.924 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.917+0000 7f241dd30700 1 Processor -- start 2023-10-06T03:29:53.924 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.917+0000 7f241dd30700 1 -- start start 2023-10-06T03:29:53.929 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.921+0000 7f241dd30700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2418156cc0 0x7f241814df80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:53.929 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.921+0000 7f241dd30700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2418157090 0x7f24180b0b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:53.929 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.921+0000 7f241dd30700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f241815ab40 0x7f2418169170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:53.929 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.921+0000 7f241dd30700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24180b0690 con 0x7f2418157090 2023-10-06T03:29:53.930 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.921+0000 7f241dd30700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24181552a0 con 0x7f2418156cc0 2023-10-06T03:29:53.930 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.921+0000 7f241dd30700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24180afa80 con 0x7f241815ab40 2023-10-06T03:29:53.930 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.921+0000 7f2417fff700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2418157090 0x7f24180b0b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:53.930 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.921+0000 7f2417fff700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2418157090 0x7f24180b0b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:47732/0 (socket says 172.21.15.110:47732) 2023-10-06T03:29:53.930 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.921+0000 7f2417fff700 1 -- 172.21.15.110:0/3704829236 learned_addr learned my addr 172.21.15.110:0/3704829236 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:29:53.930 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.921+0000 7f241cd2e700 1 --2- 172.21.15.110:0/3704829236 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2418156cc0 0x7f241814df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:53.930 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.921+0000 7f241d52f700 1 --2- 172.21.15.110:0/3704829236 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f241815ab40 0x7f2418169170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:53.930 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.921+0000 7f2417fff700 1 -- 172.21.15.110:0/3704829236 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f241815ab40 msgr2=0x7f2418169170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:53.931 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.921+0000 7f2417fff700 1 --2- 172.21.15.110:0/3704829236 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f241815ab40 0x7f2418169170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:53.931 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.921+0000 7f2417fff700 1 -- 172.21.15.110:0/3704829236 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2418156cc0 msgr2=0x7f241814df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:53.931 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.921+0000 7f2417fff700 1 --2- 172.21.15.110:0/3704829236 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2418156cc0 0x7f241814df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:53.931 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.921+0000 7f2417fff700 1 -- 172.21.15.110:0/3704829236 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f240c009420 con 0x7f2418157090 2023-10-06T03:29:53.931 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.921+0000 7f2417fff700 1 --2- 172.21.15.110:0/3704829236 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2418157090 0x7f24180b0b20 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f24040011e0 tx=0x7f2404007df0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:29:53.931 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.921+0000 7f2414ff9700 1 -- 172.21.15.110:0/3704829236 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2404005b30 con 0x7f2418157090 2023-10-06T03:29:53.931 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.921+0000 7f2414ff9700 1 -- 172.21.15.110:0/3704829236 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2404005c90 con 0x7f2418157090 2023-10-06T03:29:53.931 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.921+0000 7f241dd30700 1 -- 172.21.15.110:0/3704829236 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24180afd60 con 0x7f2418157090 2023-10-06T03:29:53.932 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.921+0000 7f2414ff9700 1 -- 172.21.15.110:0/3704829236 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24040072e0 con 0x7f2418157090 2023-10-06T03:29:53.932 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.921+0000 7f241dd30700 1 -- 172.21.15.110:0/3704829236 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24180b0320 con 0x7f2418157090 2023-10-06T03:29:53.932 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.921+0000 7f241dd30700 1 -- 172.21.15.110:0/3704829236 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2418070ff0 con 0x7f2418157090 2023-10-06T03:29:53.932 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.925+0000 7f2414ff9700 1 -- 172.21.15.110:0/3704829236 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7f2404007650 con 0x7f2418157090 2023-10-06T03:29:53.932 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.925+0000 7f2414ff9700 1 --2- 172.21.15.110:0/3704829236 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f23fc060550 0x7f23fc062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:53.932 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.925+0000 7f2414ff9700 1 -- 172.21.15.110:0/3704829236 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6809+0+0 (secure 0 0 0) 0x7f240407c030 con 0x7f2418157090 2023-10-06T03:29:53.932 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.925+0000 7f241cd2e700 1 --2- 172.21.15.110:0/3704829236 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f23fc060550 0x7f23fc062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:53.932 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.925+0000 7f241cd2e700 1 --2- 172.21.15.110:0/3704829236 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f23fc060550 0x7f23fc062a00 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f24180abf80 tx=0x7f240c017000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:29:53.936 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:53.933+0000 7f2414ff9700 1 -- 172.21.15.110:0/3704829236 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f2404049be0 con 0x7f2418157090 2023-10-06T03:29:54.096 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:54.093+0000 7f241dd30700 1 -- 172.21.15.110:0/3704829236 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f241815b870 con 0x7f2418157090 2023-10-06T03:29:54.097 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:54.093+0000 7f2414ff9700 1 -- 172.21.15.110:0/3704829236 <== mon.0 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v28) v1 ==== 104+0+11 (secure 0 0 0) 0x7f2404053e60 con 0x7f2418157090 2023-10-06T03:29:54.097 INFO:teuthology.orchestra.run.smithi110.stdout:pg_num: 32 2023-10-06T03:29:54.100 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:54.097+0000 7f241dd30700 1 -- 172.21.15.110:0/3704829236 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f23fc060550 msgr2=0x7f23fc062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:54.100 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:54.097+0000 7f241dd30700 1 --2- 172.21.15.110:0/3704829236 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f23fc060550 0x7f23fc062a00 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f24180abf80 tx=0x7f240c017000 comp rx=0 tx=0).stop 2023-10-06T03:29:54.100 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:54.097+0000 7f241dd30700 1 -- 172.21.15.110:0/3704829236 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2418157090 msgr2=0x7f24180b0b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:54.101 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:54.097+0000 7f241dd30700 1 --2- 172.21.15.110:0/3704829236 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2418157090 0x7f24180b0b20 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f24040011e0 tx=0x7f2404007df0 comp rx=0 tx=0).stop 2023-10-06T03:29:54.101 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:54.097+0000 7f241dd30700 1 -- 172.21.15.110:0/3704829236 shutdown_connections 2023-10-06T03:29:54.101 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:54.097+0000 7f241dd30700 1 --2- 172.21.15.110:0/3704829236 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f23fc060550 0x7f23fc062a00 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:54.101 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:54.097+0000 7f241dd30700 1 --2- 172.21.15.110:0/3704829236 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f241815ab40 0x7f2418169170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:54.102 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:54.097+0000 7f241dd30700 1 --2- 172.21.15.110:0/3704829236 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2418157090 0x7f24180b0b20 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:54.102 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:54.097+0000 7f241dd30700 1 --2- 172.21.15.110:0/3704829236 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2418156cc0 0x7f241814df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:54.102 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:54.097+0000 7f241dd30700 1 -- 172.21.15.110:0/3704829236 >> 172.21.15.110:0/3704829236 conn(0x7f241806ab30 msgr2=0x7f2418067740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:29:54.102 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:54.097+0000 7f241dd30700 1 -- 172.21.15.110:0/3704829236 shutdown_connections 2023-10-06T03:29:54.103 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:54.097+0000 7f241dd30700 1 -- 172.21.15.110:0/3704829236 wait complete. 2023-10-06T03:29:54.111 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-06T03:29:54.227 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:54.221+0000 7f0daffff700 1 Processor -- start 2023-10-06T03:29:54.227 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:54.221+0000 7f0daffff700 1 -- start start 2023-10-06T03:29:54.231 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:54.225+0000 7f0daffff700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f0db006cee0 0x7f0db0157090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:54.231 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:54.225+0000 7f0daffff700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0db00672b0 con 0x7f0db00c8a10 2023-10-06T03:29:54.231 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:54.225+0000 7f0daffff700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0db015b2b0 con 0x7f0db0156cc0 2023-10-06T03:29:54.231 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:54.225+0000 7f0daffff700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0db015b590 con 0x7f0db006cee0 2023-10-06T03:29:54.232 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:54.225+0000 7f0dae7fc700 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f0db00c8a10 0x7f0db006c7d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:40166/0 (socket says 172.21.15.110:40166) 2023-10-06T03:29:54.232 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:54.225+0000 7f0dae7fc700 1 -- 172.21.15.110:0/1489853270 learned_addr learned my addr 172.21.15.110:0/1489853270 (peer_addr_for_me v1:172.21.15.110:0/0) 2023-10-06T03:29:54.232 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:54.225+0000 7f0daf7fe700 1 --2- 172.21.15.110:0/1489853270 >> v2:172.21.15.192:3300/0 conn(0x7f0db006cee0 0x7f0db0157090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:54.232 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:54.225+0000 7f0dadffb700 1 -- 172.21.15.110:0/1489853270 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1216809740 0 0) 0x7f0db00672b0 con 0x7f0db00c8a10 2023-10-06T03:29:54.232 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:54.225+0000 7f0dadffb700 1 -- 172.21.15.110:0/1489853270 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0d9c003570 con 0x7f0db00c8a10 2023-10-06T03:29:54.232 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:54.225+0000 7f0dadffb700 1 -- 172.21.15.110:0/1489853270 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3679302579 0 0) 0x7f0db015b2b0 con 0x7f0db0156cc0 2023-10-06T03:29:54.232 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:54.225+0000 7f0dadffb700 1 -- 172.21.15.110:0/1489853270 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0db00672b0 con 0x7f0db0156cc0 2023-10-06T03:29:54.232 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:54.225+0000 7f0dadffb700 1 -- 172.21.15.110:0/1489853270 <== mon.0 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2506807631 0 0) 0x7f0d9c003570 con 0x7f0db00c8a10 2023-10-06T03:29:54.232 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:54.225+0000 7f0dadffb700 1 -- 172.21.15.110:0/1489853270 >> v2:172.21.15.192:3300/0 conn(0x7f0db006cee0 msgr2=0x7f0db0157090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:29:54.232 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:54.225+0000 7f0dadffb700 1 --2- 172.21.15.110:0/1489853270 >> v2:172.21.15.192:3300/0 conn(0x7f0db006cee0 0x7f0db0157090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:54.233 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:54.225+0000 7f0dadffb700 1 -- 172.21.15.110:0/1489853270 >> v1:172.21.15.192:6789/0 conn(0x7f0db0156cc0 legacy=0x7f0db00caee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:54.233 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:54.225+0000 7f0dadffb700 1 -- 172.21.15.110:0/1489853270 --> v1:172.21.15.110:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0db015b870 con 0x7f0db00c8a10 2023-10-06T03:29:54.233 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:54.225+0000 7f0dadffb700 1 -- 172.21.15.110:0/1489853270 <== mon.0 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 118603703 0 0) 0x7f0d98001470 con 0x7f0db00c8a10 2023-10-06T03:29:54.233 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:54.225+0000 7f0dadffb700 1 -- 172.21.15.110:0/1489853270 >> v1:172.21.15.110:6789/0 conn(0x7f0db00c8a10 legacy=0x7f0db006c7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:54.233 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:54.225+0000 7f0dadffb700 1 --2- 172.21.15.110:0/1489853270 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0d9c003bf0 0x7f0d9c003fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:54.233 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:54.225+0000 7f0dadffb700 1 --2- 172.21.15.110:0/1489853270 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0d9c004810 0x7f0d9c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:54.233 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:54.225+0000 7f0dadffb700 1 --2- 172.21.15.110:0/1489853270 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0d9c0051c0 0x7f0d9c00ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:54.233 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:54.225+0000 7f0dadffb700 1 -- 172.21.15.110:0/1489853270 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0db015b590 con 0x7f0d9c0051c0 2023-10-06T03:29:54.233 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:54.225+0000 7f0dadffb700 1 -- 172.21.15.110:0/1489853270 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d9c00c160 con 0x7f0d9c004810 2023-10-06T03:29:54.233 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:54.225+0000 7f0dadffb700 1 -- 172.21.15.110:0/1489853270 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d9c00c320 con 0x7f0d9c003bf0 2023-10-06T03:29:54.233 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:54.225+0000 7f0dae7fc700 1 --2- 172.21.15.110:0/1489853270 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0d9c004810 0x7f0d9c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:54.234 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:54.225+0000 7f0daeffd700 1 --2- 172.21.15.110:0/1489853270 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0d9c003bf0 0x7f0d9c003fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:54.234 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:54.225+0000 7f0daf7fe700 1 --2- 172.21.15.110:0/1489853270 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0d9c0051c0 0x7f0d9c00ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:54.234 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:54.229+0000 7f0dae7fc700 1 -- 172.21.15.110:0/1489853270 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0d9c003bf0 msgr2=0x7f0d9c003fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:29:54.234 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:54.229+0000 7f0dae7fc700 1 --2- 172.21.15.110:0/1489853270 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0d9c003bf0 0x7f0d9c003fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:54.234 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:54.229+0000 7f0dae7fc700 1 -- 172.21.15.110:0/1489853270 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0d9c0051c0 msgr2=0x7f0d9c00ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:29:54.234 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:54.229+0000 7f0dae7fc700 1 --2- 172.21.15.110:0/1489853270 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0d9c0051c0 0x7f0d9c00ba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:54.234 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:54.229+0000 7f0daf7fe700 1 -- 172.21.15.110:0/1489853270 reap_dead start 2023-10-06T03:29:54.234 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:54.229+0000 7f0dae7fc700 1 -- 172.21.15.110:0/1489853270 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d9c003570 con 0x7f0d9c004810 2023-10-06T03:29:54.237 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:54.229+0000 7f0dae7fc700 1 --2- 172.21.15.110:0/1489853270 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0d9c004810 0x7f0d9c004c80 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f0d9800c2a0 tx=0x7f0d98001e10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4b54d1722db5421 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:29:54.237 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:54.229+0000 7f0dadffb700 1 -- 172.21.15.110:0/1489853270 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d9800cb50 con 0x7f0d9c004810 2023-10-06T03:29:54.238 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:54.229+0000 7f0dadffb700 1 -- 172.21.15.110:0/1489853270 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d9800ccb0 con 0x7f0d9c004810 2023-10-06T03:29:54.238 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:54.229+0000 7f0dadffb700 1 -- 172.21.15.110:0/1489853270 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d98005b70 con 0x7f0d9c004810 2023-10-06T03:29:54.238 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:54.229+0000 7f0daffff700 1 -- 172.21.15.110:0/1489853270 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0d9c004810 msgr2=0x7f0d9c004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:54.238 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:54.229+0000 7f0daffff700 1 --2- 172.21.15.110:0/1489853270 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0d9c004810 0x7f0d9c004c80 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f0d9800c2a0 tx=0x7f0d98001e10 comp rx=0 tx=0).stop 2023-10-06T03:29:54.238 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:54.229+0000 7f0daffff700 1 -- 172.21.15.110:0/1489853270 shutdown_connections 2023-10-06T03:29:54.238 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:54.229+0000 7f0daffff700 1 --2- 172.21.15.110:0/1489853270 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0d9c004810 0x7f0d9c004c80 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:54.238 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:54.229+0000 7f0daffff700 1 -- 172.21.15.110:0/1489853270 >> 172.21.15.110:0/1489853270 conn(0x7f0db006ab30 msgr2=0x7f0db00adff0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:29:54.238 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:54.229+0000 7f0daffff700 1 -- 172.21.15.110:0/1489853270 shutdown_connections 2023-10-06T03:29:54.239 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:54.229+0000 7f0daffff700 1 -- 172.21.15.110:0/1489853270 wait complete. 2023-10-06T03:29:54.239 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:54.229+0000 7f0daffff700 1 Processor -- start 2023-10-06T03:29:54.239 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:54.229+0000 7f0daffff700 1 -- start start 2023-10-06T03:29:54.239 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:54.229+0000 7f0daffff700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0d9c004810 0x7f0db0155310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:54.239 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:54.229+0000 7f0daffff700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0db015dbc0 0x7f0db015c4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:54.239 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:54.229+0000 7f0daffff700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0db00c8fe0 0x7f0db00c9460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:54.239 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:54.229+0000 7f0daffff700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0db01695f0 con 0x7f0d9c004810 2023-10-06T03:29:54.239 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:54.229+0000 7f0daffff700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0db00c99a0 con 0x7f0db015dbc0 2023-10-06T03:29:54.239 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:54.229+0000 7f0daffff700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0db016a410 con 0x7f0db00c8fe0 2023-10-06T03:29:54.239 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:54.229+0000 7f0daeffd700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0d9c004810 0x7f0db0155310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:54.239 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:54.229+0000 7f0daeffd700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0d9c004810 0x7f0db0155310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:47744/0 (socket says 172.21.15.110:47744) 2023-10-06T03:29:54.240 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:54.229+0000 7f0daeffd700 1 -- 172.21.15.110:0/1171711338 learned_addr learned my addr 172.21.15.110:0/1171711338 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:29:54.240 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:54.229+0000 7f0daf7fe700 1 --2- 172.21.15.110:0/1171711338 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0db00c8fe0 0x7f0db00c9460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:54.240 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:54.229+0000 7f0dae7fc700 1 --2- 172.21.15.110:0/1171711338 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0db015dbc0 0x7f0db015c4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:54.240 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:54.233+0000 7f0daeffd700 1 -- 172.21.15.110:0/1171711338 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0db00c8fe0 msgr2=0x7f0db00c9460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:54.240 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:54.233+0000 7f0daeffd700 1 --2- 172.21.15.110:0/1171711338 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0db00c8fe0 0x7f0db00c9460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:54.240 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:54.233+0000 7f0daeffd700 1 -- 172.21.15.110:0/1171711338 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0db015dbc0 msgr2=0x7f0db015c4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:54.240 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:54.233+0000 7f0daeffd700 1 --2- 172.21.15.110:0/1171711338 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0db015dbc0 0x7f0db015c4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:54.240 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:54.233+0000 7f0daeffd700 1 -- 172.21.15.110:0/1171711338 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0db0169800 con 0x7f0d9c004810 2023-10-06T03:29:54.240 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:54.233+0000 7f0daf7fe700 1 --2- 172.21.15.110:0/1171711338 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0db00c8fe0 0x7f0db00c9460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-06T03:29:54.240 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:54.233+0000 7f0daeffd700 1 --2- 172.21.15.110:0/1171711338 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0d9c004810 0x7f0db0155310 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f0da0002b00 tx=0x7f0da0007d30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:29:54.240 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:54.233+0000 7f0da6ffd700 1 -- 172.21.15.110:0/1171711338 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0da000f870 con 0x7f0d9c004810 2023-10-06T03:29:54.240 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:54.233+0000 7f0da6ffd700 1 -- 172.21.15.110:0/1171711338 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0da000f9f0 con 0x7f0d9c004810 2023-10-06T03:29:54.241 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:54.233+0000 7f0daffff700 1 -- 172.21.15.110:0/1171711338 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0db0169ae0 con 0x7f0d9c004810 2023-10-06T03:29:54.241 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:54.233+0000 7f0da6ffd700 1 -- 172.21.15.110:0/1171711338 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0da000fdb0 con 0x7f0d9c004810 2023-10-06T03:29:54.241 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:54.233+0000 7f0daffff700 1 -- 172.21.15.110:0/1171711338 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0db016a0a0 con 0x7f0d9c004810 2023-10-06T03:29:54.242 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:54.233+0000 7f0daffff700 1 -- 172.21.15.110:0/1171711338 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0db0070ff0 con 0x7f0d9c004810 2023-10-06T03:29:54.242 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:54.233+0000 7f0da6ffd700 1 -- 172.21.15.110:0/1171711338 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7f0da0013600 con 0x7f0d9c004810 2023-10-06T03:29:54.242 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:54.233+0000 7f0da6ffd700 1 --2- 172.21.15.110:0/1171711338 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f0d90060580 0x7f0d90062a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:54.242 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:54.237+0000 7f0da6ffd700 1 -- 172.21.15.110:0/1171711338 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6809+0+0 (secure 0 0 0) 0x7f0da007cd80 con 0x7f0d9c004810 2023-10-06T03:29:54.242 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:54.237+0000 7f0dae7fc700 1 --2- 172.21.15.110:0/1171711338 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f0d90060580 0x7f0d90062a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:54.242 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:54.237+0000 7f0dae7fc700 1 --2- 172.21.15.110:0/1171711338 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f0d90060580 0x7f0d90062a30 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f0d98008fd0 tx=0x7f0d98015000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:29:54.248 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:54.245+0000 7f0da6ffd700 1 -- 172.21.15.110:0/1171711338 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f0da004cf20 con 0x7f0d9c004810 2023-10-06T03:29:54.407 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:54.405+0000 7f0daffff700 1 -- 172.21.15.110:0/1171711338 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f0db0157090 con 0x7f0d9c004810 2023-10-06T03:29:54.408 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:54.405+0000 7f0da6ffd700 1 -- 172.21.15.110:0/1171711338 <== mon.0 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v28) v1 ==== 100+0+11 (secure 0 0 0) 0x7f0da0050540 con 0x7f0d9c004810 2023-10-06T03:29:54.408 INFO:teuthology.orchestra.run.smithi110.stdout:pg_num: 64 2023-10-06T03:29:54.411 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:54.405+0000 7f0daffff700 1 -- 172.21.15.110:0/1171711338 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f0d90060580 msgr2=0x7f0d90062a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:54.411 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:54.405+0000 7f0daffff700 1 --2- 172.21.15.110:0/1171711338 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f0d90060580 0x7f0d90062a30 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f0d98008fd0 tx=0x7f0d98015000 comp rx=0 tx=0).stop 2023-10-06T03:29:54.411 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:54.405+0000 7f0daffff700 1 -- 172.21.15.110:0/1171711338 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0d9c004810 msgr2=0x7f0db0155310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:54.411 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:54.405+0000 7f0daffff700 1 --2- 172.21.15.110:0/1171711338 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0d9c004810 0x7f0db0155310 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f0da0002b00 tx=0x7f0da0007d30 comp rx=0 tx=0).stop 2023-10-06T03:29:54.411 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:54.405+0000 7f0daffff700 1 -- 172.21.15.110:0/1171711338 shutdown_connections 2023-10-06T03:29:54.411 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:54.405+0000 7f0daffff700 1 --2- 172.21.15.110:0/1171711338 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f0d90060580 0x7f0d90062a30 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:54.411 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:54.405+0000 7f0daffff700 1 --2- 172.21.15.110:0/1171711338 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0db00c8fe0 0x7f0db00c9460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:54.411 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:54.405+0000 7f0daffff700 1 --2- 172.21.15.110:0/1171711338 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0db015dbc0 0x7f0db015c4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:54.412 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:54.405+0000 7f0daffff700 1 --2- 172.21.15.110:0/1171711338 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0d9c004810 0x7f0db0155310 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:54.412 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:54.405+0000 7f0daffff700 1 -- 172.21.15.110:0/1171711338 >> 172.21.15.110:0/1171711338 conn(0x7f0db006ab30 msgr2=0x7f0db00afc90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:29:54.412 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:54.405+0000 7f0daffff700 1 -- 172.21.15.110:0/1171711338 shutdown_connections 2023-10-06T03:29:54.412 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:54.405+0000 7f0daffff700 1 -- 172.21.15.110:0/1171711338 wait complete. 2023-10-06T03:29:54.422 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2023-10-06T03:29:54.532 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:54.529+0000 7f8173f9e700 1 Processor -- start 2023-10-06T03:29:54.532 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:54.529+0000 7f8173f9e700 1 -- start start 2023-10-06T03:29:54.534 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:54.529+0000 7f8173f9e700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f816c0af070 0x7f816c0af440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:54.534 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:54.529+0000 7f8173f9e700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f816c15d360 0x7f816c0cb220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:54.534 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:54.529+0000 7f8173f9e700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f816c0cf5b0 con 0x7f816c0cfa40 2023-10-06T03:29:54.534 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:54.529+0000 7f8173f9e700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f816c0ce9a0 con 0x7f816c15d360 2023-10-06T03:29:54.534 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:54.529+0000 7f8173f9e700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f816c0cec80 con 0x7f816c0af070 2023-10-06T03:29:54.534 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:54.529+0000 7f817253b700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f816c15d360 0x7f816c0cb220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:54.534 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:54.529+0000 7f817253b700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f816c15d360 0x7f816c0cb220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:47746/0 (socket says 172.21.15.110:47746) 2023-10-06T03:29:54.534 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:54.529+0000 7f817253b700 1 -- 172.21.15.110:0/2375575332 learned_addr learned my addr 172.21.15.110:0/2375575332 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:29:54.535 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:54.529+0000 7f8171d3a700 1 --2- 172.21.15.110:0/2375575332 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f816c0af070 0x7f816c0af440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:54.536 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:54.529+0000 7f817253b700 1 -- 172.21.15.110:0/2375575332 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f816c0af070 msgr2=0x7f816c0af440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:29:54.536 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:54.529+0000 7f817253b700 1 --2- 172.21.15.110:0/2375575332 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f816c0af070 0x7f816c0af440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:54.536 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:54.529+0000 7f817253b700 1 -- 172.21.15.110:0/2375575332 >> v1:172.21.15.192:6789/0 conn(0x7f816c0cfa40 legacy=0x7f816c15cc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:54.536 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:54.529+0000 7f817253b700 1 -- 172.21.15.110:0/2375575332 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f816c0cef60 con 0x7f816c15d360 2023-10-06T03:29:54.536 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:54.529+0000 7f817253b700 1 --2- 172.21.15.110:0/2375575332 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f816c15d360 0x7f816c0cb220 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f8164009f20 tx=0x7f8164009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f415bccfadfd890 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:29:54.536 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:54.529+0000 7f8170d38700 1 -- 172.21.15.110:0/2375575332 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f816400e070 con 0x7f816c15d360 2023-10-06T03:29:54.536 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:54.529+0000 7f8170d38700 1 -- 172.21.15.110:0/2375575332 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f816400b9d0 con 0x7f816c15d360 2023-10-06T03:29:54.536 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:54.529+0000 7f8170d38700 1 -- 172.21.15.110:0/2375575332 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81640092e0 con 0x7f816c15d360 2023-10-06T03:29:54.536 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:54.533+0000 7f8173f9e700 1 -- 172.21.15.110:0/2375575332 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f816c15d360 msgr2=0x7f816c0cb220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:54.537 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:54.533+0000 7f8173f9e700 1 --2- 172.21.15.110:0/2375575332 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f816c15d360 0x7f816c0cb220 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f8164009f20 tx=0x7f8164009840 comp rx=0 tx=0).stop 2023-10-06T03:29:54.537 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:54.533+0000 7f8173f9e700 1 -- 172.21.15.110:0/2375575332 shutdown_connections 2023-10-06T03:29:54.537 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:54.533+0000 7f8173f9e700 1 --2- 172.21.15.110:0/2375575332 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f816c15d360 0x7f816c0cb220 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:54.537 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:54.533+0000 7f8173f9e700 1 --2- 172.21.15.110:0/2375575332 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f816c0af070 0x7f816c0af440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:54.537 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:54.533+0000 7f8173f9e700 1 -- 172.21.15.110:0/2375575332 >> 172.21.15.110:0/2375575332 conn(0x7f816c06e840 msgr2=0x7f816c0d0010 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:29:54.537 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:54.533+0000 7f8173f9e700 1 -- 172.21.15.110:0/2375575332 shutdown_connections 2023-10-06T03:29:54.537 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:54.533+0000 7f8173f9e700 1 -- 172.21.15.110:0/2375575332 wait complete. 2023-10-06T03:29:54.538 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:54.533+0000 7f8173f9e700 1 Processor -- start 2023-10-06T03:29:54.538 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:54.533+0000 7f8173f9e700 1 -- start start 2023-10-06T03:29:54.539 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:54.533+0000 7f8173f9e700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f816c0af070 0x7f816c0b2db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:54.539 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:54.533+0000 7f8173f9e700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f816c0cfa40 0x7f816c06c790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:54.539 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:54.533+0000 7f8173f9e700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f816c15d360 0x7f816c06ccd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:54.539 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:54.533+0000 7f8173f9e700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f816c0b57c0 con 0x7f816c15d360 2023-10-06T03:29:54.539 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:54.533+0000 7f8173f9e700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f816c0b37a0 con 0x7f816c0af070 2023-10-06T03:29:54.539 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:54.533+0000 7f8173f9e700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f816c0b5c50 con 0x7f816c0cfa40 2023-10-06T03:29:54.540 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:54.533+0000 7f817253b700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f816c15d360 0x7f816c06ccd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:54.540 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:54.533+0000 7f817253b700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f816c15d360 0x7f816c06ccd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:47762/0 (socket says 172.21.15.110:47762) 2023-10-06T03:29:54.540 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:54.533+0000 7f817253b700 1 -- 172.21.15.110:0/3389544837 learned_addr learned my addr 172.21.15.110:0/3389544837 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:29:54.540 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:54.533+0000 7f8171539700 1 --2- 172.21.15.110:0/3389544837 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f816c0cfa40 0x7f816c06c790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:54.540 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:54.533+0000 7f8171d3a700 1 --2- 172.21.15.110:0/3389544837 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f816c0af070 0x7f816c0b2db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:54.540 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:54.537+0000 7f817253b700 1 -- 172.21.15.110:0/3389544837 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f816c0cfa40 msgr2=0x7f816c06c790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:54.540 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:54.537+0000 7f817253b700 1 --2- 172.21.15.110:0/3389544837 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f816c0cfa40 0x7f816c06c790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:54.540 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:54.537+0000 7f817253b700 1 -- 172.21.15.110:0/3389544837 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f816c0af070 msgr2=0x7f816c0b2db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:54.540 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:54.537+0000 7f817253b700 1 --2- 172.21.15.110:0/3389544837 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f816c0af070 0x7f816c0b2db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:54.540 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:54.537+0000 7f817253b700 1 -- 172.21.15.110:0/3389544837 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8164009ae0 con 0x7f816c15d360 2023-10-06T03:29:54.540 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:54.537+0000 7f8171539700 1 --2- 172.21.15.110:0/3389544837 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f816c0cfa40 0x7f816c06c790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-06T03:29:54.541 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:54.537+0000 7f817253b700 1 --2- 172.21.15.110:0/3389544837 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f816c15d360 0x7f816c06ccd0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f816c06f840 tx=0x7f8164018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:29:54.541 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:54.537+0000 7f8161ffb700 1 -- 172.21.15.110:0/3389544837 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8164007ac0 con 0x7f816c15d360 2023-10-06T03:29:54.541 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:54.537+0000 7f8161ffb700 1 -- 172.21.15.110:0/3389544837 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8164007c20 con 0x7f816c15d360 2023-10-06T03:29:54.542 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:54.537+0000 7f8173f9e700 1 -- 172.21.15.110:0/3389544837 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f816c0b4e90 con 0x7f816c15d360 2023-10-06T03:29:54.542 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:54.537+0000 7f8161ffb700 1 -- 172.21.15.110:0/3389544837 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f816401d420 con 0x7f816c15d360 2023-10-06T03:29:54.542 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:54.537+0000 7f8173f9e700 1 -- 172.21.15.110:0/3389544837 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f816c0b5450 con 0x7f816c15d360 2023-10-06T03:29:54.542 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:54.537+0000 7f8173f9e700 1 -- 172.21.15.110:0/3389544837 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f816c0b5970 con 0x7f816c15d360 2023-10-06T03:29:54.543 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:54.537+0000 7f8161ffb700 1 -- 172.21.15.110:0/3389544837 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7f8164003040 con 0x7f816c15d360 2023-10-06T03:29:54.543 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:54.541+0000 7f8161ffb700 1 --2- 172.21.15.110:0/3389544837 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f8150060550 0x7f8150062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:54.543 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:54.541+0000 7f8161ffb700 1 -- 172.21.15.110:0/3389544837 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6809+0+0 (secure 0 0 0) 0x7f816407e8b0 con 0x7f816c15d360 2023-10-06T03:29:54.543 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:54.541+0000 7f8171d3a700 1 --2- 172.21.15.110:0/3389544837 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f8150060550 0x7f8150062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:54.544 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:54.541+0000 7f8171d3a700 1 --2- 172.21.15.110:0/3389544837 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f8150060550 0x7f8150062a00 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f8168009db0 tx=0x7f81680091a0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:29:54.549 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:54.545+0000 7f8161ffb700 1 -- 172.21.15.110:0/3389544837 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f816c0b5970 con 0x7f816c15d360 2023-10-06T03:29:54.714 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:54.709+0000 7f8173f9e700 1 -- 172.21.15.110:0/3389544837 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f816c0b5970 con 0x7f816c15d360 2023-10-06T03:29:54.714 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:54.709+0000 7f8161ffb700 1 -- 172.21.15.110:0/3389544837 <== mon.0 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v28) v1 ==== 103+0+11 (secure 0 0 0) 0x7f816c0b5970 con 0x7f816c15d360 2023-10-06T03:29:54.715 INFO:teuthology.orchestra.run.smithi110.stdout:pg_num: 32 2023-10-06T03:29:54.717 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:54.713+0000 7f8173f9e700 1 -- 172.21.15.110:0/3389544837 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f8150060550 msgr2=0x7f8150062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:54.717 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:54.713+0000 7f8173f9e700 1 --2- 172.21.15.110:0/3389544837 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f8150060550 0x7f8150062a00 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f8168009db0 tx=0x7f81680091a0 comp rx=0 tx=0).stop 2023-10-06T03:29:54.717 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:54.713+0000 7f8173f9e700 1 -- 172.21.15.110:0/3389544837 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f816c15d360 msgr2=0x7f816c06ccd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:54.717 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:54.713+0000 7f8173f9e700 1 --2- 172.21.15.110:0/3389544837 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f816c15d360 0x7f816c06ccd0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f816c06f840 tx=0x7f8164018000 comp rx=0 tx=0).stop 2023-10-06T03:29:54.718 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:54.713+0000 7f8173f9e700 1 -- 172.21.15.110:0/3389544837 shutdown_connections 2023-10-06T03:29:54.718 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:54.713+0000 7f8173f9e700 1 --2- 172.21.15.110:0/3389544837 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f8150060550 0x7f8150062a00 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:54.718 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:54.713+0000 7f8173f9e700 1 --2- 172.21.15.110:0/3389544837 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f816c15d360 0x7f816c06ccd0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:54.718 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:54.713+0000 7f8173f9e700 1 --2- 172.21.15.110:0/3389544837 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f816c0cfa40 0x7f816c06c790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:54.718 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:54.713+0000 7f8173f9e700 1 --2- 172.21.15.110:0/3389544837 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f816c0af070 0x7f816c0b2db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:54.718 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:54.713+0000 7f8173f9e700 1 -- 172.21.15.110:0/3389544837 >> 172.21.15.110:0/3389544837 conn(0x7f816c06e840 msgr2=0x7f816c156ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:29:54.718 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:54.713+0000 7f8173f9e700 1 -- 172.21.15.110:0/3389544837 shutdown_connections 2023-10-06T03:29:54.718 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:54.713+0000 7f8173f9e700 1 -- 172.21.15.110:0/3389544837 wait complete. 2023-10-06T03:29:54.729 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-06T03:29:54.836 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:54.833+0000 7f74fd7bb700 1 Processor -- start 2023-10-06T03:29:54.836 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:54.833+0000 7f74fd7bb700 1 -- start start 2023-10-06T03:29:54.838 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:54.833+0000 7f74fd7bb700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f74f813f6b0 0x7f74f80b0490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:54.838 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:54.833+0000 7f74fd7bb700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f74f813fa80 0x7f74f80c9110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:54.838 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:54.833+0000 7f74fd7bb700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74f80b0bc0 con 0x7f74f80b1960 2023-10-06T03:29:54.839 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:54.833+0000 7f74fd7bb700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f74f80b09d0 con 0x7f74f813f6b0 2023-10-06T03:29:54.839 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:54.833+0000 7f74fd7bb700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74f80c9650 con 0x7f74f813fa80 2023-10-06T03:29:54.839 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:54.833+0000 7f74f77fe700 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f74f80b1960 0x7f74f81599e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:40170/0 (socket says 172.21.15.110:40170) 2023-10-06T03:29:54.839 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:54.833+0000 7f74f77fe700 1 -- 172.21.15.110:0/179951984 learned_addr learned my addr 172.21.15.110:0/179951984 (peer_addr_for_me v1:172.21.15.110:0/0) 2023-10-06T03:29:54.839 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:54.833+0000 7f74f6ffd700 1 --2- 172.21.15.110:0/179951984 >> v2:172.21.15.192:3300/0 conn(0x7f74f813f6b0 0x7f74f80b0490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:54.839 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:54.833+0000 7f74f67fc700 1 --2- 172.21.15.110:0/179951984 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f74f813fa80 0x7f74f80c9110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:54.839 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:54.833+0000 7f74f5ffb700 1 -- 172.21.15.110:0/179951984 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 184435951 0 0) 0x7f74f80b0bc0 con 0x7f74f80b1960 2023-10-06T03:29:54.839 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:54.833+0000 7f74f5ffb700 1 -- 172.21.15.110:0/179951984 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f74e0003570 con 0x7f74f80b1960 2023-10-06T03:29:54.839 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:54.833+0000 7f74f6ffd700 1 -- 172.21.15.110:0/179951984 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f74f813fa80 msgr2=0x7f74f80c9110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:29:54.839 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:54.833+0000 7f74f6ffd700 1 --2- 172.21.15.110:0/179951984 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f74f813fa80 0x7f74f80c9110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:54.840 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:54.833+0000 7f74f6ffd700 1 -- 172.21.15.110:0/179951984 >> v1:172.21.15.110:6789/0 conn(0x7f74f80b1960 legacy=0x7f74f81599e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:54.840 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:54.833+0000 7f74f5ffb700 1 -- 172.21.15.110:0/179951984 <== mon.0 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1357324116 0 0) 0x7f74e0003570 con 0x7f74f80b1960 2023-10-06T03:29:54.840 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:54.833+0000 7f74f6ffd700 1 -- 172.21.15.110:0/179951984 --> v2:172.21.15.192:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74f80ca8c0 con 0x7f74f813f6b0 2023-10-06T03:29:54.840 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:54.833+0000 7f74f67fc700 1 --2- 172.21.15.110:0/179951984 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f74f813fa80 0x7f74f80c9110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-06T03:29:54.840 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:54.833+0000 7f74f6ffd700 1 --2- 172.21.15.110:0/179951984 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f74f813f6b0 0x7f74f80b0490 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f74ec0098e0 tx=0x7f74ec0091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=40304355c0697fa7 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:29:54.840 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:54.833+0000 7f74f5ffb700 1 -- 172.21.15.110:0/179951984 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74f0001470 con 0x7f74f813f6b0 2023-10-06T03:29:54.840 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:54.833+0000 7f74f5ffb700 1 -- 172.21.15.110:0/179951984 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74ec011420 con 0x7f74f813f6b0 2023-10-06T03:29:54.840 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:54.833+0000 7f74f5ffb700 1 -- 172.21.15.110:0/179951984 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74ec0117a0 con 0x7f74f813f6b0 2023-10-06T03:29:54.841 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:54.837+0000 7f74fd7bb700 1 -- 172.21.15.110:0/179951984 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f74f813f6b0 msgr2=0x7f74f80b0490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:54.841 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:54.837+0000 7f74fd7bb700 1 --2- 172.21.15.110:0/179951984 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f74f813f6b0 0x7f74f80b0490 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f74ec0098e0 tx=0x7f74ec0091a0 comp rx=0 tx=0).stop 2023-10-06T03:29:54.841 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:54.837+0000 7f74fd7bb700 1 -- 172.21.15.110:0/179951984 shutdown_connections 2023-10-06T03:29:54.841 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:54.837+0000 7f74fd7bb700 1 --2- 172.21.15.110:0/179951984 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f74f813fa80 0x7f74f80c9110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:54.842 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:54.837+0000 7f74fd7bb700 1 --2- 172.21.15.110:0/179951984 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f74f813f6b0 0x7f74f80b0490 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:54.842 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:54.837+0000 7f74fd7bb700 1 -- 172.21.15.110:0/179951984 >> 172.21.15.110:0/179951984 conn(0x7f74f806c8d0 msgr2=0x7f74f806ccd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:29:54.842 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:54.837+0000 7f74fd7bb700 1 -- 172.21.15.110:0/179951984 shutdown_connections 2023-10-06T03:29:54.842 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:54.837+0000 7f74fd7bb700 1 -- 172.21.15.110:0/179951984 wait complete. 2023-10-06T03:29:54.842 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:54.837+0000 7f74fd7bb700 1 Processor -- start 2023-10-06T03:29:54.842 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:54.837+0000 7f74fd7bb700 1 -- start start 2023-10-06T03:29:54.844 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:54.837+0000 7f74fd7bb700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f74f80b1960 0x7f74f815d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:54.844 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:54.837+0000 7f74fd7bb700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f74f813f6b0 0x7f74f8067420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:54.845 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:54.837+0000 7f74fd7bb700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f74f813fa80 0x7f74f813d520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:54.845 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:54.837+0000 7f74fd7bb700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74f81697a0 con 0x7f74f813f6b0 2023-10-06T03:29:54.845 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:54.837+0000 7f74fd7bb700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74f8169950 con 0x7f74f80b1960 2023-10-06T03:29:54.845 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:54.837+0000 7f74fd7bb700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74f816d3a0 con 0x7f74f813fa80 2023-10-06T03:29:54.845 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:54.837+0000 7f74f67fc700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f74f813f6b0 0x7f74f8067420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:54.845 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:54.837+0000 7f74f67fc700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f74f813f6b0 0x7f74f8067420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:47764/0 (socket says 172.21.15.110:47764) 2023-10-06T03:29:54.845 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:54.837+0000 7f74f67fc700 1 -- 172.21.15.110:0/583000914 learned_addr learned my addr 172.21.15.110:0/583000914 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:29:54.845 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:54.837+0000 7f74f6ffd700 1 --2- 172.21.15.110:0/583000914 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f74f80b1960 0x7f74f815d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:54.845 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:54.837+0000 7f74f77fe700 1 --2- 172.21.15.110:0/583000914 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f74f813fa80 0x7f74f813d520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:54.845 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:54.841+0000 7f74f67fc700 1 -- 172.21.15.110:0/583000914 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f74f813fa80 msgr2=0x7f74f813d520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:54.846 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:54.841+0000 7f74f67fc700 1 --2- 172.21.15.110:0/583000914 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f74f813fa80 0x7f74f813d520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:54.846 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:54.841+0000 7f74f67fc700 1 -- 172.21.15.110:0/583000914 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f74f80b1960 msgr2=0x7f74f815d5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:54.846 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:54.841+0000 7f74f67fc700 1 --2- 172.21.15.110:0/583000914 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f74f80b1960 0x7f74f815d5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:54.846 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:54.841+0000 7f74f67fc700 1 -- 172.21.15.110:0/583000914 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74e8009a20 con 0x7f74f813f6b0 2023-10-06T03:29:54.846 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:54.841+0000 7f74f6ffd700 1 --2- 172.21.15.110:0/583000914 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f74f80b1960 0x7f74f815d5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-06T03:29:54.846 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:54.841+0000 7f74f67fc700 1 --2- 172.21.15.110:0/583000914 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f74f813f6b0 0x7f74f8067420 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f74e800c0c0 tx=0x7f74e800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:29:54.846 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:54.841+0000 7f74deffd700 1 -- 172.21.15.110:0/583000914 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74e800cc00 con 0x7f74f813f6b0 2023-10-06T03:29:54.846 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:54.841+0000 7f74deffd700 1 -- 172.21.15.110:0/583000914 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74e800f040 con 0x7f74f813f6b0 2023-10-06T03:29:54.846 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:54.841+0000 7f74deffd700 1 -- 172.21.15.110:0/583000914 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74e8014480 con 0x7f74f813f6b0 2023-10-06T03:29:54.846 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:54.841+0000 7f74fd7bb700 1 -- 172.21.15.110:0/583000914 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74ec009420 con 0x7f74f813f6b0 2023-10-06T03:29:54.846 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:54.841+0000 7f74fd7bb700 1 -- 172.21.15.110:0/583000914 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74f816d960 con 0x7f74f813f6b0 2023-10-06T03:29:54.848 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:54.841+0000 7f74fd7bb700 1 -- 172.21.15.110:0/583000914 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f74f80cb0c0 con 0x7f74f813f6b0 2023-10-06T03:29:54.848 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:54.841+0000 7f74deffd700 1 -- 172.21.15.110:0/583000914 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7f74e80147f0 con 0x7f74f813f6b0 2023-10-06T03:29:54.848 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:54.841+0000 7f74deffd700 1 --2- 172.21.15.110:0/583000914 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f74e4060550 0x7f74e4062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:54.848 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:54.841+0000 7f74deffd700 1 -- 172.21.15.110:0/583000914 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6809+0+0 (secure 0 0 0) 0x7f74e807d900 con 0x7f74f813f6b0 2023-10-06T03:29:54.848 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:54.841+0000 7f74f6ffd700 1 --2- 172.21.15.110:0/583000914 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f74e4060550 0x7f74e4062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:54.848 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:54.845+0000 7f74f6ffd700 1 --2- 172.21.15.110:0/583000914 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f74e4060550 0x7f74e4062a00 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f74ec0091e0 tx=0x7f74ec016000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:29:54.854 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:54.849+0000 7f74deffd700 1 -- 172.21.15.110:0/583000914 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f74f80cb0c0 con 0x7f74f813f6b0 2023-10-06T03:29:55.049 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:55.045+0000 7f74fd7bb700 1 -- 172.21.15.110:0/583000914 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f74f8169a90 con 0x7f74f813f6b0 2023-10-06T03:29:55.049 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:55.045+0000 7f74deffd700 1 -- 172.21.15.110:0/583000914 <== mon.0 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4602 (secure 0 0 0) 0x7f74f8169a90 con 0x7f74f813f6b0 2023-10-06T03:29:55.049 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-10-06T03:29:55.049 INFO:teuthology.orchestra.run.smithi110.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4279,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6833/1434505700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6832","nonce":1434505700},{"type":"v1","addr":"172.21.15.192:6833","nonce":1434505700}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4281,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/1168370789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6834","nonce":1168370789},{"type":"v1","addr":"172.21.15.110:6835","nonce":1168370789}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4288,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6835/1271810086","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":1271810086},{"type":"v1","addr":"172.21.15.192:6835","nonce":1271810086}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-06T03:29:33.894998+0000","modified":"2023-10-06T03:29:47.307075+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4293},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4293":{"gid":4293,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.110:6837/540320464","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6836","nonce":540320464},{"type":"v1","addr":"172.21.15.110:6837","nonce":540320464}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-06T03:29:55.051 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:55.045+0000 7f74fd7bb700 1 -- 172.21.15.110:0/583000914 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f74e4060550 msgr2=0x7f74e4062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:55.051 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:55.045+0000 7f74fd7bb700 1 --2- 172.21.15.110:0/583000914 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f74e4060550 0x7f74e4062a00 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f74ec0091e0 tx=0x7f74ec016000 comp rx=0 tx=0).stop 2023-10-06T03:29:55.052 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:55.045+0000 7f74fd7bb700 1 -- 172.21.15.110:0/583000914 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f74f813f6b0 msgr2=0x7f74f8067420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:55.052 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:55.045+0000 7f74fd7bb700 1 --2- 172.21.15.110:0/583000914 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f74f813f6b0 0x7f74f8067420 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f74e800c0c0 tx=0x7f74e800b040 comp rx=0 tx=0).stop 2023-10-06T03:29:55.052 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:55.045+0000 7f74fd7bb700 1 -- 172.21.15.110:0/583000914 shutdown_connections 2023-10-06T03:29:55.052 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:55.045+0000 7f74fd7bb700 1 --2- 172.21.15.110:0/583000914 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f74e4060550 0x7f74e4062a00 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:55.052 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:55.045+0000 7f74fd7bb700 1 --2- 172.21.15.110:0/583000914 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f74f813fa80 0x7f74f813d520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:55.052 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:55.045+0000 7f74fd7bb700 1 --2- 172.21.15.110:0/583000914 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f74f813f6b0 0x7f74f8067420 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:55.052 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:55.045+0000 7f74fd7bb700 1 --2- 172.21.15.110:0/583000914 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f74f80b1960 0x7f74f815d5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:55.052 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:55.045+0000 7f74fd7bb700 1 -- 172.21.15.110:0/583000914 >> 172.21.15.110:0/583000914 conn(0x7f74f806c8d0 msgr2=0x7f74f806b7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:29:55.052 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:55.049+0000 7f74fd7bb700 1 -- 172.21.15.110:0/583000914 shutdown_connections 2023-10-06T03:29:55.053 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:55.049+0000 7f74fd7bb700 1 -- 172.21.15.110:0/583000914 wait complete. 2023-10-06T03:29:55.053 INFO:teuthology.orchestra.run.smithi110.stderr:dumped fsmap epoch 8 2023-10-06T03:29:55.063 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-06T03:29:55.171 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:55.169+0000 7f6d089ae700 1 Processor -- start 2023-10-06T03:29:55.171 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:55.169+0000 7f6d089ae700 1 -- start start 2023-10-06T03:29:55.173 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:55.169+0000 7f6d089ae700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6d040afb80 0x7f6d040aff50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:55.174 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:55.169+0000 7f6d089ae700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f6d040b1cd0 0x7f6d040b2100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:55.174 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:55.169+0000 7f6d089ae700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f6d04158a50 0x7f6d04158eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:55.174 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:55.169+0000 7f6d089ae700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6d040b2640 con 0x7f6d04158a50 2023-10-06T03:29:55.174 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:55.169+0000 7f6d089ae700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6d040cac30 con 0x7f6d040b1cd0 2023-10-06T03:29:55.174 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:55.169+0000 7f6d089ae700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d040caf10 con 0x7f6d040afb80 2023-10-06T03:29:55.174 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:55.169+0000 7f6d0359e700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f6d04158a50 0x7f6d04158eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:55.174 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:55.169+0000 7f6d0359e700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f6d04158a50 0x7f6d04158eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:47776/0 (socket says 172.21.15.110:47776) 2023-10-06T03:29:55.174 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:55.169+0000 7f6d0359e700 1 -- 172.21.15.110:0/2319346476 learned_addr learned my addr 172.21.15.110:0/2319346476 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:29:55.177 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:55.169+0000 7f6cfa59c700 1 --2- 172.21.15.110:0/2319346476 >> v2:172.21.15.192:3300/0 conn(0x7f6d040b1cd0 0x7f6d040b2100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:55.177 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:55.169+0000 7f6d02d9d700 1 --2- 172.21.15.110:0/2319346476 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6d040afb80 0x7f6d040aff50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:55.177 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:55.169+0000 7f6cfa59c700 1 -- 172.21.15.110:0/2319346476 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6d040afb80 msgr2=0x7f6d040aff50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:29:55.178 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:55.169+0000 7f6cfa59c700 1 --2- 172.21.15.110:0/2319346476 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6d040afb80 0x7f6d040aff50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:55.178 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:55.169+0000 7f6cfa59c700 1 -- 172.21.15.110:0/2319346476 >> v2:172.21.15.110:3300/0 conn(0x7f6d04158a50 msgr2=0x7f6d04158eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:29:55.178 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:55.169+0000 7f6cfa59c700 1 --2- 172.21.15.110:0/2319346476 >> v2:172.21.15.110:3300/0 conn(0x7f6d04158a50 0x7f6d04158eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:55.178 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:55.169+0000 7f6cfa59c700 1 -- 172.21.15.110:0/2319346476 --> v2:172.21.15.192:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d040cb1f0 con 0x7f6d040b1cd0 2023-10-06T03:29:55.178 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:55.169+0000 7f6d0359e700 1 --2- 172.21.15.110:0/2319346476 >> v2:172.21.15.110:3300/0 conn(0x7f6d04158a50 0x7f6d04158eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-06T03:29:55.178 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:55.169+0000 7f6cfa59c700 1 --2- 172.21.15.110:0/2319346476 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6d040b1cd0 0x7f6d040b2100 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f6d040d0e70 tx=0x7f6cec0091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f344f4a19b2126da server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:29:55.178 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:55.169+0000 7f6d0259c700 1 -- 172.21.15.110:0/2319346476 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cec011630 con 0x7f6d040b1cd0 2023-10-06T03:29:55.179 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:55.169+0000 7f6d0259c700 1 -- 172.21.15.110:0/2319346476 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6cec0117b0 con 0x7f6d040b1cd0 2023-10-06T03:29:55.179 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:55.169+0000 7f6d0259c700 1 -- 172.21.15.110:0/2319346476 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cec011b70 con 0x7f6d040b1cd0 2023-10-06T03:29:55.182 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:55.169+0000 7f6d089ae700 1 -- 172.21.15.110:0/2319346476 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6d040b1cd0 msgr2=0x7f6d040b2100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:55.182 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:55.169+0000 7f6d089ae700 1 --2- 172.21.15.110:0/2319346476 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6d040b1cd0 0x7f6d040b2100 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f6d040d0e70 tx=0x7f6cec0091a0 comp rx=0 tx=0).stop 2023-10-06T03:29:55.182 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:55.169+0000 7f6d089ae700 1 -- 172.21.15.110:0/2319346476 shutdown_connections 2023-10-06T03:29:55.183 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:55.169+0000 7f6d089ae700 1 --2- 172.21.15.110:0/2319346476 >> v2:172.21.15.110:3300/0 conn(0x7f6d04158a50 0x7f6d04158eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:55.183 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:55.169+0000 7f6d089ae700 1 --2- 172.21.15.110:0/2319346476 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6d040b1cd0 0x7f6d040b2100 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:55.183 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:55.169+0000 7f6d089ae700 1 --2- 172.21.15.110:0/2319346476 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6d040afb80 0x7f6d040aff50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:55.183 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:55.169+0000 7f6d089ae700 1 -- 172.21.15.110:0/2319346476 >> 172.21.15.110:0/2319346476 conn(0x7f6d0406dc70 msgr2=0x7f6d0406e070 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:29:55.183 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:55.169+0000 7f6d089ae700 1 -- 172.21.15.110:0/2319346476 shutdown_connections 2023-10-06T03:29:55.183 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:55.169+0000 7f6d089ae700 1 -- 172.21.15.110:0/2319346476 wait complete. 2023-10-06T03:29:55.183 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:55.173+0000 7f6d089ae700 1 Processor -- start 2023-10-06T03:29:55.183 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:55.173+0000 7f6d089ae700 1 -- start start 2023-10-06T03:29:55.183 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:55.173+0000 7f6d089ae700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6d040afb80 0x7f6d04144d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:55.183 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:55.173+0000 7f6d089ae700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6d040b1cd0 0x7f6d04158eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:55.183 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:55.173+0000 7f6d089ae700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6d04158a50 0x7f6d041593f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:55.184 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:55.173+0000 7f6d089ae700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d0415c570 con 0x7f6d040afb80 2023-10-06T03:29:55.184 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:55.173+0000 7f6d089ae700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d040c8470 con 0x7f6d040b1cd0 2023-10-06T03:29:55.184 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:55.173+0000 7f6d089ae700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d0415bfb0 con 0x7f6d04158a50 2023-10-06T03:29:55.184 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:55.173+0000 7f6d02d9d700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6d040afb80 0x7f6d04144d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:55.184 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:55.173+0000 7f6d02d9d700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6d040afb80 0x7f6d04144d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:47778/0 (socket says 172.21.15.110:47778) 2023-10-06T03:29:55.184 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:55.173+0000 7f6d02d9d700 1 -- 172.21.15.110:0/4274904755 learned_addr learned my addr 172.21.15.110:0/4274904755 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:29:55.184 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:55.173+0000 7f6d0359e700 1 --2- 172.21.15.110:0/4274904755 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6d04158a50 0x7f6d041593f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:55.184 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:55.173+0000 7f6cfa59c700 1 --2- 172.21.15.110:0/4274904755 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6d040b1cd0 0x7f6d04158eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:55.184 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:55.173+0000 7f6d02d9d700 1 -- 172.21.15.110:0/4274904755 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6d04158a50 msgr2=0x7f6d041593f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:55.184 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:55.173+0000 7f6d02d9d700 1 --2- 172.21.15.110:0/4274904755 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6d04158a50 0x7f6d041593f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:55.184 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:55.173+0000 7f6d02d9d700 1 -- 172.21.15.110:0/4274904755 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6d040b1cd0 msgr2=0x7f6d04158eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:55.185 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:55.173+0000 7f6d02d9d700 1 --2- 172.21.15.110:0/4274904755 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6d040b1cd0 0x7f6d04158eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:55.185 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:55.173+0000 7f6d02d9d700 1 -- 172.21.15.110:0/4274904755 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6cfc009ae0 con 0x7f6d040afb80 2023-10-06T03:29:55.185 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:55.173+0000 7f6d02d9d700 1 --2- 172.21.15.110:0/4274904755 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6d040afb80 0x7f6d04144d70 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f6cf400e930 tx=0x7f6cf40091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:29:55.185 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:55.173+0000 7f6cfb7fe700 1 -- 172.21.15.110:0/4274904755 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cf4013680 con 0x7f6d040afb80 2023-10-06T03:29:55.185 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:55.177+0000 7f6cfb7fe700 1 -- 172.21.15.110:0/4274904755 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6cf40137e0 con 0x7f6d040afb80 2023-10-06T03:29:55.185 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:55.177+0000 7f6d089ae700 1 -- 172.21.15.110:0/4274904755 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6cec009420 con 0x7f6d040afb80 2023-10-06T03:29:55.185 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:55.177+0000 7f6cfb7fe700 1 -- 172.21.15.110:0/4274904755 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cf4013b80 con 0x7f6d040afb80 2023-10-06T03:29:55.185 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:55.177+0000 7f6d089ae700 1 -- 172.21.15.110:0/4274904755 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d040c8730 con 0x7f6d040afb80 2023-10-06T03:29:55.185 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:55.177+0000 7f6d089ae700 1 -- 172.21.15.110:0/4274904755 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d04070ff0 con 0x7f6d040afb80 2023-10-06T03:29:55.185 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:55.177+0000 7f6cfb7fe700 1 -- 172.21.15.110:0/4274904755 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7f6cf400c3e0 con 0x7f6d040afb80 2023-10-06T03:29:55.185 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:55.177+0000 7f6cfb7fe700 1 --2- 172.21.15.110:0/4274904755 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f6ce40605d0 0x7f6ce4062a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:55.186 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:55.177+0000 7f6cfb7fe700 1 -- 172.21.15.110:0/4274904755 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6809+0+0 (secure 0 0 0) 0x7f6cf407cdc0 con 0x7f6d040afb80 2023-10-06T03:29:55.186 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:55.177+0000 7f6cfa59c700 1 --2- 172.21.15.110:0/4274904755 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f6ce40605d0 0x7f6ce4062a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:55.186 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:55.177+0000 7f6cfa59c700 1 --2- 172.21.15.110:0/4274904755 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f6ce40605d0 0x7f6ce4062a80 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f6d040d0e70 tx=0x7f6cec017000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:29:55.189 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:55.185+0000 7f6cfb7fe700 1 -- 172.21.15.110:0/4274904755 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f6cf404cec0 con 0x7f6d040afb80 2023-10-06T03:29:55.353 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:55.349+0000 7f6d089ae700 1 -- 172.21.15.110:0/4274904755 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6d0406c820 con 0x7f6d040afb80 2023-10-06T03:29:55.355 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:55.353+0000 7f6cfb7fe700 1 -- 172.21.15.110:0/4274904755 <== mon.0 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v28) v1 ==== 74+0+18755 (secure 0 0 0) 0x7f6cf40504e0 con 0x7f6d040afb80 2023-10-06T03:29:55.356 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-10-06T03:29:55.356 INFO:teuthology.orchestra.run.smithi110.stdout:{"epoch":28,"fsid":"93b8f6ac-4c92-4b45-b8e4-b59617024aa2","created":"2023-10-06T03:29:10.519583+0000","modified":"2023-10-06T03:29:38.951740+0000","last_up_change":"2023-10-06T03:29:21.960511+0000","last_in_change":"2023-10-06T03:29:12.560436+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-06T03:29:20.703743+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2023-10-06T03:29:28.307281+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2023-10-06T03:29:32.247251+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data","create_time":"2023-10-06T03:29:32.880086+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}}},{"pool":5,"pool_name":"cephfs_data_ec","create_time":"2023-10-06T03:29:36.613779+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"28","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"5866eb6c-2adc-46e4-be7a-e5bcda0f4065","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6824","nonce":33394},{"type":"v1","addr":"172.21.15.110:6825","nonce":33394}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6826","nonce":33394},{"type":"v1","addr":"172.21.15.110:6827","nonce":33394}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6830","nonce":33394},{"type":"v1","addr":"172.21.15.110:6831","nonce":33394}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6828","nonce":33394},{"type":"v1","addr":"172.21.15.110:6829","nonce":33394}]},"public_addr":"172.21.15.110:6825/33394","cluster_addr":"172.21.15.110:6827/33394","heartbeat_back_addr":"172.21.15.110:6831/33394","heartbeat_front_addr":"172.21.15.110:6829/33394","state":["exists","up"]},{"osd":1,"uuid":"a38de118-062b-4a05-9782-2ceddf6ff4df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6800","nonce":33396},{"type":"v1","addr":"172.21.15.110:6801","nonce":33396}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6802","nonce":33396},{"type":"v1","addr":"172.21.15.110:6803","nonce":33396}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6806","nonce":33396},{"type":"v1","addr":"172.21.15.110:6807","nonce":33396}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6804","nonce":33396},{"type":"v1","addr":"172.21.15.110:6805","nonce":33396}]},"public_addr":"172.21.15.110:6801/33396","cluster_addr":"172.21.15.110:6803/33396","heartbeat_back_addr":"172.21.15.110:6807/33396","heartbeat_front_addr":"172.21.15.110:6805/33396","state":["exists","up"]},{"osd":2,"uuid":"56e39be2-3407-4b69-877e-cf16617b06d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6816","nonce":33397},{"type":"v1","addr":"172.21.15.110:6817","nonce":33397}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6818","nonce":33397},{"type":"v1","addr":"172.21.15.110:6819","nonce":33397}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6822","nonce":33397},{"type":"v1","addr":"172.21.15.110:6823","nonce":33397}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6820","nonce":33397},{"type":"v1","addr":"172.21.15.110:6821","nonce":33397}]},"public_addr":"172.21.15.110:6817/33397","cluster_addr":"172.21.15.110:6819/33397","heartbeat_back_addr":"172.21.15.110:6823/33397","heartbeat_front_addr":"172.21.15.110:6821/33397","state":["exists","up"]},{"osd":3,"uuid":"01c82d13-c1d4-4bde-b55e-3e0089b40115","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6808","nonce":33395},{"type":"v1","addr":"172.21.15.110:6809","nonce":33395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6810","nonce":33395},{"type":"v1","addr":"172.21.15.110:6811","nonce":33395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6814","nonce":33395},{"type":"v1","addr":"172.21.15.110:6815","nonce":33395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6812","nonce":33395},{"type":"v1","addr":"172.21.15.110:6813","nonce":33395}]},"public_addr":"172.21.15.110:6809/33395","cluster_addr":"172.21.15.110:6811/33395","heartbeat_back_addr":"172.21.15.110:6815/33395","heartbeat_front_addr":"172.21.15.110:6813/33395","state":["exists","up"]},{"osd":4,"uuid":"86983430-5fa3-4d17-998f-03e2cd843237","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":33683},{"type":"v1","addr":"172.21.15.192:6817","nonce":33683}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":33683},{"type":"v1","addr":"172.21.15.192:6819","nonce":33683}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":33683},{"type":"v1","addr":"172.21.15.192:6823","nonce":33683}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6820","nonce":33683},{"type":"v1","addr":"172.21.15.192:6821","nonce":33683}]},"public_addr":"172.21.15.192:6817/33683","cluster_addr":"172.21.15.192:6819/33683","heartbeat_back_addr":"172.21.15.192:6823/33683","heartbeat_front_addr":"172.21.15.192:6821/33683","state":["exists","up"]},{"osd":5,"uuid":"6945257f-821d-4619-a63f-ebe857eba6f4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":33684},{"type":"v1","addr":"172.21.15.192:6801","nonce":33684}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":33684},{"type":"v1","addr":"172.21.15.192:6803","nonce":33684}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":33684},{"type":"v1","addr":"172.21.15.192:6807","nonce":33684}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6804","nonce":33684},{"type":"v1","addr":"172.21.15.192:6805","nonce":33684}]},"public_addr":"172.21.15.192:6801/33684","cluster_addr":"172.21.15.192:6803/33684","heartbeat_back_addr":"172.21.15.192:6807/33684","heartbeat_front_addr":"172.21.15.192:6805/33684","state":["exists","up"]},{"osd":6,"uuid":"7854d1ae-7060-4780-8f97-73cea5e7a12f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6824","nonce":33686},{"type":"v1","addr":"172.21.15.192:6825","nonce":33686}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":33686},{"type":"v1","addr":"172.21.15.192:6827","nonce":33686}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":33686},{"type":"v1","addr":"172.21.15.192:6831","nonce":33686}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6828","nonce":33686},{"type":"v1","addr":"172.21.15.192:6829","nonce":33686}]},"public_addr":"172.21.15.192:6825/33686","cluster_addr":"172.21.15.192:6827/33686","heartbeat_back_addr":"172.21.15.192:6831/33686","heartbeat_front_addr":"172.21.15.192:6829/33686","state":["exists","up"]},{"osd":7,"uuid":"f1ebdfd5-b859-482b-9f97-c18c4a26f445","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6808","nonce":33685},{"type":"v1","addr":"172.21.15.192:6809","nonce":33685}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":33685},{"type":"v1","addr":"172.21.15.192:6811","nonce":33685}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":33685},{"type":"v1","addr":"172.21.15.192:6815","nonce":33685}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6812","nonce":33685},{"type":"v1","addr":"172.21.15.192:6813","nonce":33685}]},"public_addr":"172.21.15.192:6809/33685","cluster_addr":"172.21.15.192:6811/33685","heartbeat_back_addr":"172.21.15.192:6815/33685","heartbeat_front_addr":"172.21.15.192:6813/33685","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T03:29:18.002048+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T03:29:17.478576+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T03:29:18.034944+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T03:29:17.628513+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T03:29:18.047338+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T03:29:17.772349+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T03:29:17.883958+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T03:29:18.151838+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-06T03:29:55.358 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:55.353+0000 7f6d089ae700 1 -- 172.21.15.110:0/4274904755 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f6ce40605d0 msgr2=0x7f6ce4062a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:55.358 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:55.353+0000 7f6d089ae700 1 --2- 172.21.15.110:0/4274904755 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f6ce40605d0 0x7f6ce4062a80 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f6d040d0e70 tx=0x7f6cec017000 comp rx=0 tx=0).stop 2023-10-06T03:29:55.358 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:55.353+0000 7f6d089ae700 1 -- 172.21.15.110:0/4274904755 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6d040afb80 msgr2=0x7f6d04144d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:55.359 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:55.353+0000 7f6d089ae700 1 --2- 172.21.15.110:0/4274904755 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6d040afb80 0x7f6d04144d70 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f6cf400e930 tx=0x7f6cf40091a0 comp rx=0 tx=0).stop 2023-10-06T03:29:55.359 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:55.353+0000 7f6d089ae700 1 -- 172.21.15.110:0/4274904755 shutdown_connections 2023-10-06T03:29:55.359 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:55.353+0000 7f6d089ae700 1 --2- 172.21.15.110:0/4274904755 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f6ce40605d0 0x7f6ce4062a80 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:55.359 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:55.353+0000 7f6d089ae700 1 --2- 172.21.15.110:0/4274904755 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6d04158a50 0x7f6d041593f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:55.359 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:55.353+0000 7f6d089ae700 1 --2- 172.21.15.110:0/4274904755 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6d040b1cd0 0x7f6d04158eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:55.359 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:55.353+0000 7f6d089ae700 1 --2- 172.21.15.110:0/4274904755 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6d040afb80 0x7f6d04144d70 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:55.359 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:55.353+0000 7f6d089ae700 1 -- 172.21.15.110:0/4274904755 >> 172.21.15.110:0/4274904755 conn(0x7f6d0406dc70 msgr2=0x7f6d041571c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:29:55.359 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:55.353+0000 7f6d089ae700 1 -- 172.21.15.110:0/4274904755 shutdown_connections 2023-10-06T03:29:55.360 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:55.353+0000 7f6d089ae700 1 -- 172.21.15.110:0/4274904755 wait complete. 2023-10-06T03:29:55.372 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-06T03:29:20.703743+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-10-06T03:29:28.307281+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 3, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-06T03:29:32.247251+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 4, 'pool_name': 'cephfs_data', 'create_time': '2023-10-06T03:29:32.880086+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}, {'pool': 5, 'pool_name': 'cephfs_data_ec', 'create_time': '2023-10-06T03:29:36.613779+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '28', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-10-06T03:29:55.372 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-06T03:29:55.485 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:55.481+0000 7fd774efd700 1 Processor -- start 2023-10-06T03:29:55.485 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:55.481+0000 7fd774efd700 1 -- start start 2023-10-06T03:29:55.486 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:55.481+0000 7fd774efd700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd7700af070 0x7fd7700af440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:55.487 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:55.481+0000 7fd774efd700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fd77015d360 0x7fd7700cb220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:55.487 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:55.481+0000 7fd774efd700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7700cf5b0 con 0x7fd7700cfa40 2023-10-06T03:29:55.487 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:55.481+0000 7fd774efd700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd7700ce9a0 con 0x7fd77015d360 2023-10-06T03:29:55.487 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:55.481+0000 7fd774efd700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7700cec80 con 0x7fd7700af070 2023-10-06T03:29:55.487 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:55.481+0000 7fd76dd9b700 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7fd7700cfa40 0x7fd77015cc50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:40172/0 (socket says 172.21.15.110:40172) 2023-10-06T03:29:55.487 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:55.481+0000 7fd76dd9b700 1 -- 172.21.15.110:0/157276300 learned_addr learned my addr 172.21.15.110:0/157276300 (peer_addr_for_me v1:172.21.15.110:0/0) 2023-10-06T03:29:55.487 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:55.481+0000 7fd76ed9d700 1 --2- 172.21.15.110:0/157276300 >> v2:172.21.15.192:3300/0 conn(0x7fd77015d360 0x7fd7700cb220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:55.488 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:55.481+0000 7fd76e59c700 1 --2- 172.21.15.110:0/157276300 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd7700af070 0x7fd7700af440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:55.489 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:55.485+0000 7fd76d59a700 1 -- 172.21.15.110:0/157276300 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3415063088 0 0) 0x7fd7700cf5b0 con 0x7fd7700cfa40 2023-10-06T03:29:55.489 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:55.485+0000 7fd76d59a700 1 -- 172.21.15.110:0/157276300 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd75c003570 con 0x7fd7700cfa40 2023-10-06T03:29:55.489 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:55.485+0000 7fd76ed9d700 1 -- 172.21.15.110:0/157276300 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd7700af070 msgr2=0x7fd7700af440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:29:55.489 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:55.485+0000 7fd76ed9d700 1 --2- 172.21.15.110:0/157276300 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd7700af070 0x7fd7700af440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:55.489 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:55.485+0000 7fd76ed9d700 1 -- 172.21.15.110:0/157276300 >> v1:172.21.15.110:6789/0 conn(0x7fd7700cfa40 legacy=0x7fd77015cc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:55.489 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:55.485+0000 7fd76ed9d700 1 -- 172.21.15.110:0/157276300 --> v2:172.21.15.192:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7700cef60 con 0x7fd77015d360 2023-10-06T03:29:55.489 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:55.485+0000 7fd76ed9d700 1 --2- 172.21.15.110:0/157276300 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd77015d360 0x7fd7700cb220 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fd768009e60 tx=0x7fd768009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a8103dda645b6a3e server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:29:55.489 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:55.485+0000 7fd76d59a700 1 -- 172.21.15.110:0/157276300 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd76800e070 con 0x7fd77015d360 2023-10-06T03:29:55.490 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:55.485+0000 7fd76d59a700 1 -- 172.21.15.110:0/157276300 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd76800bab0 con 0x7fd77015d360 2023-10-06T03:29:55.490 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:55.485+0000 7fd76d59a700 1 -- 172.21.15.110:0/157276300 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd76800be70 con 0x7fd77015d360 2023-10-06T03:29:55.490 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:55.485+0000 7fd774efd700 1 -- 172.21.15.110:0/157276300 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd77015d360 msgr2=0x7fd7700cb220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:55.491 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:55.485+0000 7fd774efd700 1 --2- 172.21.15.110:0/157276300 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd77015d360 0x7fd7700cb220 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fd768009e60 tx=0x7fd768009780 comp rx=0 tx=0).stop 2023-10-06T03:29:55.491 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:55.485+0000 7fd774efd700 1 -- 172.21.15.110:0/157276300 shutdown_connections 2023-10-06T03:29:55.491 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:55.485+0000 7fd774efd700 1 --2- 172.21.15.110:0/157276300 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd77015d360 0x7fd7700cb220 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:55.491 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:55.485+0000 7fd774efd700 1 --2- 172.21.15.110:0/157276300 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd7700af070 0x7fd7700af440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:55.491 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:55.485+0000 7fd774efd700 1 -- 172.21.15.110:0/157276300 >> 172.21.15.110:0/157276300 conn(0x7fd77006e840 msgr2=0x7fd7700d0010 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:29:55.491 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:55.485+0000 7fd774efd700 1 -- 172.21.15.110:0/157276300 shutdown_connections 2023-10-06T03:29:55.491 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:55.485+0000 7fd774efd700 1 -- 172.21.15.110:0/157276300 wait complete. 2023-10-06T03:29:55.493 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:55.489+0000 7fd774efd700 1 Processor -- start 2023-10-06T03:29:55.493 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:55.489+0000 7fd774efd700 1 -- start start 2023-10-06T03:29:55.497 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:55.489+0000 7fd774efd700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd7700af070 0x7fd77016c070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:55.497 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:55.489+0000 7fd774efd700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd7700cfa40 0x7fd77016c5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:55.497 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:55.489+0000 7fd774efd700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd77015d360 0x7fd7701542a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:55.497 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:55.489+0000 7fd774efd700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd770169d70 con 0x7fd7700af070 2023-10-06T03:29:55.497 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:55.489+0000 7fd774efd700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd77015b100 con 0x7fd7700cfa40 2023-10-06T03:29:55.498 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:55.489+0000 7fd774efd700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd77015b3e0 con 0x7fd77015d360 2023-10-06T03:29:55.498 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:55.489+0000 7fd76e59c700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd7700af070 0x7fd77016c070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:55.498 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:55.489+0000 7fd76e59c700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd7700af070 0x7fd77016c070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:47788/0 (socket says 172.21.15.110:47788) 2023-10-06T03:29:55.498 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:55.489+0000 7fd76e59c700 1 -- 172.21.15.110:0/1673081930 learned_addr learned my addr 172.21.15.110:0/1673081930 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:29:55.498 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:55.489+0000 7fd76dd9b700 1 --2- 172.21.15.110:0/1673081930 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd7700cfa40 0x7fd77016c5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:55.498 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:55.489+0000 7fd76ed9d700 1 --2- 172.21.15.110:0/1673081930 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd77015d360 0x7fd7701542a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:55.498 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:55.493+0000 7fd76e59c700 1 -- 172.21.15.110:0/1673081930 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd77015d360 msgr2=0x7fd7701542a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:55.498 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:55.493+0000 7fd76e59c700 1 --2- 172.21.15.110:0/1673081930 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd77015d360 0x7fd7701542a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:55.498 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:55.493+0000 7fd76e59c700 1 -- 172.21.15.110:0/1673081930 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd7700cfa40 msgr2=0x7fd77016c5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:55.499 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:55.493+0000 7fd76e59c700 1 --2- 172.21.15.110:0/1673081930 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd7700cfa40 0x7fd77016c5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:55.499 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:55.493+0000 7fd76e59c700 1 -- 172.21.15.110:0/1673081930 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd768009a20 con 0x7fd7700af070 2023-10-06T03:29:55.499 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:55.493+0000 7fd76e59c700 1 --2- 172.21.15.110:0/1673081930 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd7700af070 0x7fd77016c070 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fd76000ec80 tx=0x7fd7600091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:29:55.499 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:55.493+0000 7fd7667fc700 1 -- 172.21.15.110:0/1673081930 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd76000c850 con 0x7fd7700af070 2023-10-06T03:29:55.499 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:55.493+0000 7fd7667fc700 1 -- 172.21.15.110:0/1673081930 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd76000c9b0 con 0x7fd7700af070 2023-10-06T03:29:55.499 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:55.493+0000 7fd774efd700 1 -- 172.21.15.110:0/1673081930 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd77016a010 con 0x7fd7700af070 2023-10-06T03:29:55.499 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:55.493+0000 7fd7667fc700 1 -- 172.21.15.110:0/1673081930 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd76000cd50 con 0x7fd7700af070 2023-10-06T03:29:55.499 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:55.493+0000 7fd774efd700 1 -- 172.21.15.110:0/1673081930 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7701547e0 con 0x7fd7700af070 2023-10-06T03:29:55.500 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:55.493+0000 7fd774efd700 1 -- 172.21.15.110:0/1673081930 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd77006ff60 con 0x7fd7700af070 2023-10-06T03:29:55.500 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:55.497+0000 7fd7667fc700 1 -- 172.21.15.110:0/1673081930 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7fd760013680 con 0x7fd7700af070 2023-10-06T03:29:55.500 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:55.497+0000 7fd7667fc700 1 --2- 172.21.15.110:0/1673081930 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fd750060550 0x7fd750062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:55.500 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:55.497+0000 7fd7667fc700 1 -- 172.21.15.110:0/1673081930 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6809+0+0 (secure 0 0 0) 0x7fd76007ce20 con 0x7fd7700af070 2023-10-06T03:29:55.500 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:55.497+0000 7fd76dd9b700 1 --2- 172.21.15.110:0/1673081930 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fd750060550 0x7fd750062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:55.501 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:55.497+0000 7fd76dd9b700 1 --2- 172.21.15.110:0/1673081930 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fd750060550 0x7fd750062a00 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fd77015cdc0 tx=0x7fd758009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:29:55.509 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:55.505+0000 7fd7667fc700 1 -- 172.21.15.110:0/1673081930 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fd76004cfc0 con 0x7fd7700af070 2023-10-06T03:29:55.676 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:55.673+0000 7fd774efd700 1 -- 172.21.15.110:0/1673081930 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fd7700cf240 con 0x7fd7700af070 2023-10-06T03:29:55.676 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:55.673+0000 7fd7667fc700 1 -- 172.21.15.110:0/1673081930 <== mon.0 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v28) v1 ==== 93+0+10 (secure 0 0 0) 0x7fd7600505e0 con 0x7fd7700af070 2023-10-06T03:29:55.676 INFO:teuthology.orchestra.run.smithi110.stdout:pg_num: 1 2023-10-06T03:29:55.679 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:55.673+0000 7fd774efd700 1 -- 172.21.15.110:0/1673081930 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fd750060550 msgr2=0x7fd750062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:55.679 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:55.673+0000 7fd774efd700 1 --2- 172.21.15.110:0/1673081930 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fd750060550 0x7fd750062a00 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fd77015cdc0 tx=0x7fd758009040 comp rx=0 tx=0).stop 2023-10-06T03:29:55.679 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:55.673+0000 7fd774efd700 1 -- 172.21.15.110:0/1673081930 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd7700af070 msgr2=0x7fd77016c070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:55.679 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:55.673+0000 7fd774efd700 1 --2- 172.21.15.110:0/1673081930 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd7700af070 0x7fd77016c070 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fd76000ec80 tx=0x7fd7600091a0 comp rx=0 tx=0).stop 2023-10-06T03:29:55.679 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:55.673+0000 7fd774efd700 1 -- 172.21.15.110:0/1673081930 shutdown_connections 2023-10-06T03:29:55.679 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:55.673+0000 7fd774efd700 1 --2- 172.21.15.110:0/1673081930 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fd750060550 0x7fd750062a00 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:55.679 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:55.673+0000 7fd774efd700 1 --2- 172.21.15.110:0/1673081930 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd77015d360 0x7fd7701542a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:55.680 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:55.673+0000 7fd774efd700 1 --2- 172.21.15.110:0/1673081930 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd7700cfa40 0x7fd77016c5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:55.680 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:55.673+0000 7fd774efd700 1 --2- 172.21.15.110:0/1673081930 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd7700af070 0x7fd77016c070 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:55.680 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:55.673+0000 7fd774efd700 1 -- 172.21.15.110:0/1673081930 >> 172.21.15.110:0/1673081930 conn(0x7fd77006e840 msgr2=0x7fd7700c4db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:29:55.680 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:55.673+0000 7fd774efd700 1 -- 172.21.15.110:0/1673081930 shutdown_connections 2023-10-06T03:29:55.680 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:55.673+0000 7fd774efd700 1 -- 172.21.15.110:0/1673081930 wait complete. 2023-10-06T03:29:55.691 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-10-06T03:29:55.804 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:55.801+0000 7fd8ccf46700 1 Processor -- start 2023-10-06T03:29:55.804 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:55.801+0000 7fd8ccf46700 1 -- start start 2023-10-06T03:29:55.806 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:55.801+0000 7fd8ccf46700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fd8c8155890 0x7fd8c80af3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:55.806 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:55.801+0000 7fd8ccf46700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd8c80b0e60 0x7fd8c806b7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:55.806 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:55.801+0000 7fd8ccf46700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8c80aef30 con 0x7fd8c80af900 2023-10-06T03:29:55.806 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:55.801+0000 7fd8ccf46700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8c8069790 con 0x7fd8c8155890 2023-10-06T03:29:55.806 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:55.801+0000 7fd8ccf46700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8c8156d70 con 0x7fd8c80b0e60 2023-10-06T03:29:55.807 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:55.801+0000 7fd8c7fff700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd8c80b0e60 0x7fd8c806b7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:55.807 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:55.801+0000 7fd8c77fe700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fd8c8155890 0x7fd8c80af3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:55.807 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:55.801+0000 7fd8c77fe700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fd8c8155890 0x7fd8c80af3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.110:36586/0 (socket says 172.21.15.110:36586) 2023-10-06T03:29:55.807 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:55.801+0000 7fd8c77fe700 1 -- 172.21.15.110:0/75359970 learned_addr learned my addr 172.21.15.110:0/75359970 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:29:55.807 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:55.801+0000 7fd8c6ffd700 1 --1- 172.21.15.110:0/75359970 >> v1:172.21.15.192:6789/0 conn(0x7fd8c80af900 0x7fd8c80b0750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.110:55750/0 (socket says 172.21.15.110:55750) 2023-10-06T03:29:55.809 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:55.801+0000 7fd8c77fe700 1 -- 172.21.15.110:0/75359970 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd8c80b0e60 msgr2=0x7fd8c806b7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:29:55.809 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:55.801+0000 7fd8c77fe700 1 --2- 172.21.15.110:0/75359970 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd8c80b0e60 0x7fd8c806b7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:55.809 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:55.801+0000 7fd8c77fe700 1 -- 172.21.15.110:0/75359970 >> v1:172.21.15.192:6789/0 conn(0x7fd8c80af900 legacy=0x7fd8c80b0750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:55.809 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:55.801+0000 7fd8c77fe700 1 -- 172.21.15.110:0/75359970 --> v2:172.21.15.192:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8c8157050 con 0x7fd8c8155890 2023-10-06T03:29:55.809 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:55.801+0000 7fd8c67fc700 1 -- 172.21.15.110:0/75359970 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2019592364 0 0) 0x7fd8c80aef30 con 0x7fd8c80af900 2023-10-06T03:29:55.809 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:55.805+0000 7fd8c77fe700 1 --2- 172.21.15.110:0/75359970 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd8c8155890 0x7fd8c80af3c0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fd8b80099b0 tx=0x7fd8b8009250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d26e12e527800d88 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:29:55.809 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:55.805+0000 7fd8c67fc700 1 -- 172.21.15.110:0/75359970 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8b8009ea0 con 0x7fd8c8155890 2023-10-06T03:29:55.809 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:55.805+0000 7fd8c67fc700 1 -- 172.21.15.110:0/75359970 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8b800b860 con 0x7fd8c8155890 2023-10-06T03:29:55.809 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:55.805+0000 7fd8c67fc700 1 -- 172.21.15.110:0/75359970 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8b800bc00 con 0x7fd8c8155890 2023-10-06T03:29:55.809 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:55.805+0000 7fd8ccf46700 1 -- 172.21.15.110:0/75359970 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd8c8155890 msgr2=0x7fd8c80af3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:55.810 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:55.805+0000 7fd8ccf46700 1 --2- 172.21.15.110:0/75359970 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd8c8155890 0x7fd8c80af3c0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fd8b80099b0 tx=0x7fd8b8009250 comp rx=0 tx=0).stop 2023-10-06T03:29:55.810 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:55.805+0000 7fd8ccf46700 1 -- 172.21.15.110:0/75359970 shutdown_connections 2023-10-06T03:29:55.810 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:55.805+0000 7fd8ccf46700 1 --2- 172.21.15.110:0/75359970 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd8c80b0e60 0x7fd8c806b7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:55.810 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:55.805+0000 7fd8ccf46700 1 --2- 172.21.15.110:0/75359970 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd8c8155890 0x7fd8c80af3c0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:55.810 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:55.805+0000 7fd8ccf46700 1 -- 172.21.15.110:0/75359970 >> 172.21.15.110:0/75359970 conn(0x7fd8c80cbac0 msgr2=0x7fd8c80cbec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:29:55.810 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:55.805+0000 7fd8ccf46700 1 -- 172.21.15.110:0/75359970 shutdown_connections 2023-10-06T03:29:55.810 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:55.805+0000 7fd8ccf46700 1 -- 172.21.15.110:0/75359970 wait complete. 2023-10-06T03:29:55.811 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:55.809+0000 7fd8ccf46700 1 Processor -- start 2023-10-06T03:29:55.811 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:55.809+0000 7fd8ccf46700 1 -- start start 2023-10-06T03:29:55.814 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:55.809+0000 7fd8ccf46700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd8c80af900 0x7fd8c8139040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:55.814 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:55.809+0000 7fd8ccf46700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd8c80b0e60 0x7fd8c813bef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:55.815 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:55.809+0000 7fd8ccf46700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd8c8155890 0x7fd8c813cd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:55.815 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:55.809+0000 7fd8ccf46700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8c806cb20 con 0x7fd8c80b0e60 2023-10-06T03:29:55.815 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:55.809+0000 7fd8ccf46700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8c813ba60 con 0x7fd8c80af900 2023-10-06T03:29:55.815 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:55.809+0000 7fd8ccf46700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8c8139580 con 0x7fd8c8155890 2023-10-06T03:29:55.815 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:55.809+0000 7fd8c6ffd700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd8c80b0e60 0x7fd8c813bef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:55.815 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:55.809+0000 7fd8c6ffd700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd8c80b0e60 0x7fd8c813bef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:47804/0 (socket says 172.21.15.110:47804) 2023-10-06T03:29:55.815 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:55.809+0000 7fd8c6ffd700 1 -- 172.21.15.110:0/1314281767 learned_addr learned my addr 172.21.15.110:0/1314281767 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:29:55.815 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:55.809+0000 7fd8c77fe700 1 --2- 172.21.15.110:0/1314281767 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd8c80af900 0x7fd8c8139040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:55.815 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:55.809+0000 7fd8c7fff700 1 --2- 172.21.15.110:0/1314281767 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd8c8155890 0x7fd8c813cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:55.817 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:55.809+0000 7fd8c6ffd700 1 -- 172.21.15.110:0/1314281767 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd8c8155890 msgr2=0x7fd8c813cd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:55.817 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:55.809+0000 7fd8c6ffd700 1 --2- 172.21.15.110:0/1314281767 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd8c8155890 0x7fd8c813cd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:55.817 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:55.809+0000 7fd8c6ffd700 1 -- 172.21.15.110:0/1314281767 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd8c80af900 msgr2=0x7fd8c8139040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:55.817 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:55.809+0000 7fd8c6ffd700 1 --2- 172.21.15.110:0/1314281767 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd8c80af900 0x7fd8c8139040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:55.818 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:55.809+0000 7fd8c6ffd700 1 -- 172.21.15.110:0/1314281767 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8b80094f0 con 0x7fd8c80b0e60 2023-10-06T03:29:55.818 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:55.809+0000 7fd8c6ffd700 1 --2- 172.21.15.110:0/1314281767 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd8c80b0e60 0x7fd8c813bef0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fd8b000c810 tx=0x7fd8b0007de0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:29:55.818 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:55.809+0000 7fd8bf7fe700 1 -- 172.21.15.110:0/1314281767 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8b0005b40 con 0x7fd8c80b0e60 2023-10-06T03:29:55.818 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:55.809+0000 7fd8bf7fe700 1 -- 172.21.15.110:0/1314281767 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8b0005cc0 con 0x7fd8c80b0e60 2023-10-06T03:29:55.818 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:55.809+0000 7fd8ccf46700 1 -- 172.21.15.110:0/1314281767 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8c813ae50 con 0x7fd8c80b0e60 2023-10-06T03:29:55.818 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:55.813+0000 7fd8bf7fe700 1 -- 172.21.15.110:0/1314281767 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8b0007320 con 0x7fd8c80b0e60 2023-10-06T03:29:55.818 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:55.813+0000 7fd8ccf46700 1 -- 172.21.15.110:0/1314281767 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8c813b410 con 0x7fd8c80b0e60 2023-10-06T03:29:55.818 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:55.813+0000 7fd8bf7fe700 1 -- 172.21.15.110:0/1314281767 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7fd8b001b3b0 con 0x7fd8c80b0e60 2023-10-06T03:29:55.818 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:55.813+0000 7fd8ccf46700 1 -- 172.21.15.110:0/1314281767 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8c813bc10 con 0x7fd8c80b0e60 2023-10-06T03:29:55.819 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:55.813+0000 7fd8bf7fe700 1 --2- 172.21.15.110:0/1314281767 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fd8a8060550 0x7fd8a8062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:55.819 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:55.813+0000 7fd8bf7fe700 1 -- 172.21.15.110:0/1314281767 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6809+0+0 (secure 0 0 0) 0x7fd8b007c0f0 con 0x7fd8c80b0e60 2023-10-06T03:29:55.819 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:55.813+0000 7fd8c77fe700 1 --2- 172.21.15.110:0/1314281767 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fd8a8060550 0x7fd8a8062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:55.819 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:55.813+0000 7fd8c77fe700 1 --2- 172.21.15.110:0/1314281767 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fd8a8060550 0x7fd8a8062a00 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fd8b80099b0 tx=0x7fd8b8017000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:29:55.823 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:55.821+0000 7fd8bf7fe700 1 -- 172.21.15.110:0/1314281767 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fd8c813bc10 con 0x7fd8c80b0e60 2023-10-06T03:29:55.985 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:55.981+0000 7fd8ccf46700 1 -- 172.21.15.110:0/1314281767 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7fd8c813bc10 con 0x7fd8c80b0e60 2023-10-06T03:29:55.986 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:55.981+0000 7fd8bf7fe700 1 -- 172.21.15.110:0/1314281767 <== mon.0 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v28) v1 ==== 92+0+10 (secure 0 0 0) 0x7fd8c813bc10 con 0x7fd8c80b0e60 2023-10-06T03:29:55.986 INFO:teuthology.orchestra.run.smithi110.stdout:pg_num: 8 2023-10-06T03:29:55.988 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:55.985+0000 7fd8ccf46700 1 -- 172.21.15.110:0/1314281767 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fd8a8060550 msgr2=0x7fd8a8062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:55.988 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:55.985+0000 7fd8ccf46700 1 --2- 172.21.15.110:0/1314281767 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fd8a8060550 0x7fd8a8062a00 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fd8b80099b0 tx=0x7fd8b8017000 comp rx=0 tx=0).stop 2023-10-06T03:29:55.988 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:55.985+0000 7fd8ccf46700 1 -- 172.21.15.110:0/1314281767 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd8c80b0e60 msgr2=0x7fd8c813bef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:55.988 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:55.985+0000 7fd8ccf46700 1 --2- 172.21.15.110:0/1314281767 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd8c80b0e60 0x7fd8c813bef0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fd8b000c810 tx=0x7fd8b0007de0 comp rx=0 tx=0).stop 2023-10-06T03:29:55.989 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:55.985+0000 7fd8ccf46700 1 -- 172.21.15.110:0/1314281767 shutdown_connections 2023-10-06T03:29:55.989 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:55.985+0000 7fd8ccf46700 1 --2- 172.21.15.110:0/1314281767 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fd8a8060550 0x7fd8a8062a00 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:55.989 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:55.985+0000 7fd8ccf46700 1 --2- 172.21.15.110:0/1314281767 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd8c8155890 0x7fd8c813cd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:55.989 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:55.985+0000 7fd8ccf46700 1 --2- 172.21.15.110:0/1314281767 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd8c80b0e60 0x7fd8c813bef0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:55.989 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:55.985+0000 7fd8ccf46700 1 --2- 172.21.15.110:0/1314281767 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd8c80af900 0x7fd8c8139040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:55.989 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:55.985+0000 7fd8ccf46700 1 -- 172.21.15.110:0/1314281767 >> 172.21.15.110:0/1314281767 conn(0x7fd8c80cbac0 msgr2=0x7fd8c8154a70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:29:55.989 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:55.985+0000 7fd8ccf46700 1 -- 172.21.15.110:0/1314281767 shutdown_connections 2023-10-06T03:29:55.989 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:55.985+0000 7fd8ccf46700 1 -- 172.21.15.110:0/1314281767 wait complete. 2023-10-06T03:29:56.000 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-06T03:29:56.115 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:56.109+0000 7f9a65445700 1 Processor -- start 2023-10-06T03:29:56.115 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:56.109+0000 7f9a65445700 1 -- start start 2023-10-06T03:29:56.116 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:56.113+0000 7f9a65445700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f9a600af120 0x7f9a600cb3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:56.116 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:56.113+0000 7f9a65445700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9a600af4f0 0x7f9a6006b7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:56.117 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:56.113+0000 7f9a65445700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f9a6015cc20 0x7f9a6015d080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:56.117 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:56.113+0000 7f9a65445700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9a6015d5c0 con 0x7f9a600af120 2023-10-06T03:29:56.117 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:56.113+0000 7f9a65445700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9a6006bd00 con 0x7f9a6015cc20 2023-10-06T03:29:56.117 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:56.113+0000 7f9a65445700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a6006bfd0 con 0x7f9a600af4f0 2023-10-06T03:29:56.117 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:56.113+0000 7f9a5effd700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f9a600af120 0x7f9a600cb3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:56.117 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:56.113+0000 7f9a5effd700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f9a600af120 0x7f9a600cb3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:47814/0 (socket says 172.21.15.110:47814) 2023-10-06T03:29:56.117 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:56.113+0000 7f9a5e7fc700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9a600af4f0 0x7f9a6006b7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:56.117 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:56.113+0000 7f9a5effd700 1 -- 172.21.15.110:0/3892227789 learned_addr learned my addr 172.21.15.110:0/3892227789 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:29:56.117 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:56.113+0000 7f9a5f7fe700 1 --2- 172.21.15.110:0/3892227789 >> v2:172.21.15.192:3300/0 conn(0x7f9a6015cc20 0x7f9a6015d080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:56.119 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:56.113+0000 7f9a5effd700 1 -- 172.21.15.110:0/3892227789 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9a600af4f0 msgr2=0x7f9a6006b7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:29:56.119 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:56.113+0000 7f9a5effd700 1 --2- 172.21.15.110:0/3892227789 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9a600af4f0 0x7f9a6006b7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:56.119 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:56.113+0000 7f9a5effd700 1 -- 172.21.15.110:0/3892227789 >> v2:172.21.15.192:3300/0 conn(0x7f9a6015cc20 msgr2=0x7f9a6015d080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:29:56.119 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:56.113+0000 7f9a5effd700 1 --2- 172.21.15.110:0/3892227789 >> v2:172.21.15.192:3300/0 conn(0x7f9a6015cc20 0x7f9a6015d080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:56.120 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:56.113+0000 7f9a5effd700 1 -- 172.21.15.110:0/3892227789 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a6006c2b0 con 0x7f9a600af120 2023-10-06T03:29:56.120 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:56.113+0000 7f9a5f7fe700 1 --2- 172.21.15.110:0/3892227789 >> v2:172.21.15.192:3300/0 conn(0x7f9a6015cc20 0x7f9a6015d080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-06T03:29:56.120 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:56.113+0000 7f9a5e7fc700 1 --2- 172.21.15.110:0/3892227789 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9a600af4f0 0x7f9a6006b7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-06T03:29:56.120 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:56.113+0000 7f9a5effd700 1 --2- 172.21.15.110:0/3892227789 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9a600af120 0x7f9a600cb3a0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f9a60089440 tx=0x7f9a54009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a091ed0d967f204c server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:29:56.120 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:56.113+0000 7f9a5dffb700 1 -- 172.21.15.110:0/3892227789 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a5400b640 con 0x7f9a600af120 2023-10-06T03:29:56.120 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:56.113+0000 7f9a5dffb700 1 -- 172.21.15.110:0/3892227789 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a5400b7a0 con 0x7f9a600af120 2023-10-06T03:29:56.120 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:56.113+0000 7f9a5dffb700 1 -- 172.21.15.110:0/3892227789 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a5400bb60 con 0x7f9a600af120 2023-10-06T03:29:56.120 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:56.113+0000 7f9a65445700 1 -- 172.21.15.110:0/3892227789 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9a600af120 msgr2=0x7f9a600cb3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:56.120 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:56.113+0000 7f9a65445700 1 --2- 172.21.15.110:0/3892227789 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9a600af120 0x7f9a600cb3a0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f9a60089440 tx=0x7f9a54009250 comp rx=0 tx=0).stop 2023-10-06T03:29:56.120 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:56.113+0000 7f9a65445700 1 -- 172.21.15.110:0/3892227789 shutdown_connections 2023-10-06T03:29:56.120 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:56.113+0000 7f9a65445700 1 --2- 172.21.15.110:0/3892227789 >> v2:172.21.15.192:3300/0 conn(0x7f9a6015cc20 0x7f9a6015d080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:56.121 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:56.113+0000 7f9a65445700 1 --2- 172.21.15.110:0/3892227789 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9a600af4f0 0x7f9a6006b7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:56.121 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:56.113+0000 7f9a65445700 1 --2- 172.21.15.110:0/3892227789 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9a600af120 0x7f9a600cb3a0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:56.121 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:56.113+0000 7f9a65445700 1 -- 172.21.15.110:0/3892227789 >> 172.21.15.110:0/3892227789 conn(0x7f9a60087ac0 msgr2=0x7f9a600d0030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:29:56.121 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:56.113+0000 7f9a65445700 1 -- 172.21.15.110:0/3892227789 shutdown_connections 2023-10-06T03:29:56.121 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:56.113+0000 7f9a65445700 1 -- 172.21.15.110:0/3892227789 wait complete. 2023-10-06T03:29:56.121 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:56.117+0000 7f9a65445700 1 Processor -- start 2023-10-06T03:29:56.122 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:56.117+0000 7f9a65445700 1 -- start start 2023-10-06T03:29:56.123 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:56.117+0000 7f9a65445700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9a600af120 0x7f9a600b3cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:56.123 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:56.117+0000 7f9a65445700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9a600af4f0 0x7f9a60150f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:56.123 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:56.117+0000 7f9a65445700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9a6015cc20 0x7f9a600c70e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:56.123 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:56.117+0000 7f9a65445700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a6015b6f0 con 0x7f9a600af4f0 2023-10-06T03:29:56.123 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:56.117+0000 7f9a65445700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a600b4520 con 0x7f9a6015cc20 2023-10-06T03:29:56.123 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:56.117+0000 7f9a65445700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a6015adc0 con 0x7f9a600af120 2023-10-06T03:29:56.123 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:56.117+0000 7f9a5e7fc700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9a600af4f0 0x7f9a60150f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:56.123 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:56.117+0000 7f9a5e7fc700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9a600af4f0 0x7f9a60150f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:47820/0 (socket says 172.21.15.110:47820) 2023-10-06T03:29:56.123 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:56.117+0000 7f9a5e7fc700 1 -- 172.21.15.110:0/2785266617 learned_addr learned my addr 172.21.15.110:0/2785266617 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:29:56.124 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:56.117+0000 7f9a5f7fe700 1 --2- 172.21.15.110:0/2785266617 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9a6015cc20 0x7f9a600c70e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:56.124 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:56.117+0000 7f9a5effd700 1 --2- 172.21.15.110:0/2785266617 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9a600af120 0x7f9a600b3cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:56.124 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:56.117+0000 7f9a5e7fc700 1 -- 172.21.15.110:0/2785266617 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9a600af120 msgr2=0x7f9a600b3cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:56.124 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:56.117+0000 7f9a5e7fc700 1 --2- 172.21.15.110:0/2785266617 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9a600af120 0x7f9a600b3cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:56.124 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:56.117+0000 7f9a5e7fc700 1 -- 172.21.15.110:0/2785266617 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9a6015cc20 msgr2=0x7f9a600c70e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:56.124 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:56.117+0000 7f9a5e7fc700 1 --2- 172.21.15.110:0/2785266617 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9a6015cc20 0x7f9a600c70e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:56.124 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:56.117+0000 7f9a5e7fc700 1 -- 172.21.15.110:0/2785266617 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a50009420 con 0x7f9a600af4f0 2023-10-06T03:29:56.124 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:56.117+0000 7f9a5e7fc700 1 --2- 172.21.15.110:0/2785266617 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9a600af4f0 0x7f9a60150f70 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f9a5000ec80 tx=0x7f9a500091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:29:56.125 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:56.121+0000 7f9a46ffd700 1 -- 172.21.15.110:0/2785266617 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a5000c9d0 con 0x7f9a600af4f0 2023-10-06T03:29:56.125 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:56.121+0000 7f9a46ffd700 1 -- 172.21.15.110:0/2785266617 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a5000cb30 con 0x7f9a600af4f0 2023-10-06T03:29:56.125 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:56.121+0000 7f9a65445700 1 -- 172.21.15.110:0/2785266617 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a58009a20 con 0x7f9a600af4f0 2023-10-06T03:29:56.126 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:56.121+0000 7f9a46ffd700 1 -- 172.21.15.110:0/2785266617 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a5000ceb0 con 0x7f9a600af4f0 2023-10-06T03:29:56.126 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:56.121+0000 7f9a65445700 1 -- 172.21.15.110:0/2785266617 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a540094f0 con 0x7f9a600af4f0 2023-10-06T03:29:56.126 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:56.121+0000 7f9a65445700 1 -- 172.21.15.110:0/2785266617 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a600caed0 con 0x7f9a600af4f0 2023-10-06T03:29:56.126 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:56.121+0000 7f9a46ffd700 1 -- 172.21.15.110:0/2785266617 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7f9a50013680 con 0x7f9a600af4f0 2023-10-06T03:29:56.127 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:56.125+0000 7f9a46ffd700 1 --2- 172.21.15.110:0/2785266617 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f9a4c060550 0x7f9a4c062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:56.127 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:56.125+0000 7f9a46ffd700 1 -- 172.21.15.110:0/2785266617 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6809+0+0 (secure 0 0 0) 0x7f9a5007cf50 con 0x7f9a600af4f0 2023-10-06T03:29:56.127 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:56.125+0000 7f9a5effd700 1 --2- 172.21.15.110:0/2785266617 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f9a4c060550 0x7f9a4c062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:56.128 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:56.125+0000 7f9a5effd700 1 --2- 172.21.15.110:0/2785266617 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f9a4c060550 0x7f9a4c062a00 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f9a60089440 tx=0x7f9a54017000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:29:56.136 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:56.133+0000 7f9a46ffd700 1 -- 172.21.15.110:0/2785266617 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f9a600caed0 con 0x7f9a600af4f0 2023-10-06T03:29:56.300 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:56.297+0000 7f9a65445700 1 -- 172.21.15.110:0/2785266617 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f9a600caed0 con 0x7f9a600af4f0 2023-10-06T03:29:56.300 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:56.297+0000 7f9a46ffd700 1 -- 172.21.15.110:0/2785266617 <== mon.0 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v28) v1 ==== 104+0+11 (secure 0 0 0) 0x7f9a600caed0 con 0x7f9a600af4f0 2023-10-06T03:29:56.301 INFO:teuthology.orchestra.run.smithi110.stdout:pg_num: 32 2023-10-06T03:29:56.303 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:56.297+0000 7f9a65445700 1 -- 172.21.15.110:0/2785266617 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f9a4c060550 msgr2=0x7f9a4c062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:56.303 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:56.297+0000 7f9a65445700 1 --2- 172.21.15.110:0/2785266617 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f9a4c060550 0x7f9a4c062a00 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f9a60089440 tx=0x7f9a54017000 comp rx=0 tx=0).stop 2023-10-06T03:29:56.304 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:56.297+0000 7f9a65445700 1 -- 172.21.15.110:0/2785266617 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9a600af4f0 msgr2=0x7f9a60150f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:56.304 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:56.297+0000 7f9a65445700 1 --2- 172.21.15.110:0/2785266617 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9a600af4f0 0x7f9a60150f70 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f9a5000ec80 tx=0x7f9a500091a0 comp rx=0 tx=0).stop 2023-10-06T03:29:56.304 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:56.297+0000 7f9a65445700 1 -- 172.21.15.110:0/2785266617 shutdown_connections 2023-10-06T03:29:56.304 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:56.297+0000 7f9a65445700 1 --2- 172.21.15.110:0/2785266617 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f9a4c060550 0x7f9a4c062a00 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:56.304 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:56.297+0000 7f9a65445700 1 --2- 172.21.15.110:0/2785266617 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9a6015cc20 0x7f9a600c70e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:56.304 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:56.297+0000 7f9a65445700 1 --2- 172.21.15.110:0/2785266617 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9a600af4f0 0x7f9a60150f70 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:56.304 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:56.297+0000 7f9a65445700 1 --2- 172.21.15.110:0/2785266617 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9a600af120 0x7f9a600b3cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:56.304 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:56.297+0000 7f9a65445700 1 -- 172.21.15.110:0/2785266617 >> 172.21.15.110:0/2785266617 conn(0x7f9a60087ac0 msgr2=0x7f9a600cf720 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:29:56.304 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:56.301+0000 7f9a65445700 1 -- 172.21.15.110:0/2785266617 shutdown_connections 2023-10-06T03:29:56.304 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:56.301+0000 7f9a65445700 1 -- 172.21.15.110:0/2785266617 wait complete. 2023-10-06T03:29:56.315 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-06T03:29:56.428 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:56.421+0000 7f2ac51a2700 1 Processor -- start 2023-10-06T03:29:56.429 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:56.421+0000 7f2ac51a2700 1 -- start start 2023-10-06T03:29:56.431 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:56.425+0000 7f2ac51a2700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2ac006c970 0x7f2ac006cd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:56.431 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:56.425+0000 7f2ac51a2700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f2ac0159c20 0x7f2ac0155b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:56.431 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:56.425+0000 7f2ac51a2700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f2ac01560d0 0x7f2ac0152ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:56.432 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:56.425+0000 7f2ac51a2700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2ac01591d0 con 0x7f2ac01560d0 2023-10-06T03:29:56.432 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:56.425+0000 7f2ac51a2700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2ac0156700 con 0x7f2ac0159c20 2023-10-06T03:29:56.432 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:56.425+0000 7f2ac51a2700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ac01569e0 con 0x7f2ac006c970 2023-10-06T03:29:56.432 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:56.425+0000 7f2ac49a1700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f2ac01560d0 0x7f2ac0152ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:56.432 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:56.425+0000 7f2ac49a1700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f2ac01560d0 0x7f2ac0152ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:47824/0 (socket says 172.21.15.110:47824) 2023-10-06T03:29:56.433 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:56.425+0000 7f2ac49a1700 1 -- 172.21.15.110:0/2688520638 learned_addr learned my addr 172.21.15.110:0/2688520638 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:29:56.433 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:56.425+0000 7f2abffff700 1 --2- 172.21.15.110:0/2688520638 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2ac006c970 0x7f2ac006cd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:56.433 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:56.425+0000 7f2ab77fe700 1 --2- 172.21.15.110:0/2688520638 >> v2:172.21.15.192:3300/0 conn(0x7f2ac0159c20 0x7f2ac0155b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:56.433 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:56.425+0000 7f2ac49a1700 1 -- 172.21.15.110:0/2688520638 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2ac006c970 msgr2=0x7f2ac006cd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:29:56.433 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:56.425+0000 7f2ac49a1700 1 --2- 172.21.15.110:0/2688520638 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2ac006c970 0x7f2ac006cd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:56.434 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:56.425+0000 7f2ac49a1700 1 -- 172.21.15.110:0/2688520638 >> v2:172.21.15.192:3300/0 conn(0x7f2ac0159c20 msgr2=0x7f2ac0155b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:29:56.434 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:56.425+0000 7f2ac49a1700 1 --2- 172.21.15.110:0/2688520638 >> v2:172.21.15.192:3300/0 conn(0x7f2ac0159c20 0x7f2ac0155b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:56.434 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:56.425+0000 7f2ac49a1700 1 -- 172.21.15.110:0/2688520638 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ac0159420 con 0x7f2ac01560d0 2023-10-06T03:29:56.434 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:56.425+0000 7f2ab77fe700 1 --2- 172.21.15.110:0/2688520638 >> v2:172.21.15.192:3300/0 conn(0x7f2ac0159c20 0x7f2ac0155b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-06T03:29:56.438 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:56.425+0000 7f2ac49a1700 1 --2- 172.21.15.110:0/2688520638 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2ac01560d0 0x7f2ac0152ca0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f2ab8009f20 tx=0x7f2ab8009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e73ff53c12fe56af server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:29:56.438 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:56.425+0000 7f2abf7fe700 1 -- 172.21.15.110:0/2688520638 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ab800e070 con 0x7f2ac01560d0 2023-10-06T03:29:56.438 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:56.425+0000 7f2abf7fe700 1 -- 172.21.15.110:0/2688520638 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ab800b930 con 0x7f2ac01560d0 2023-10-06T03:29:56.438 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:56.425+0000 7f2abf7fe700 1 -- 172.21.15.110:0/2688520638 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ab80092e0 con 0x7f2ac01560d0 2023-10-06T03:29:56.438 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:56.429+0000 7f2ac51a2700 1 -- 172.21.15.110:0/2688520638 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2ac01560d0 msgr2=0x7f2ac0152ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:56.439 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:56.429+0000 7f2ac51a2700 1 --2- 172.21.15.110:0/2688520638 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2ac01560d0 0x7f2ac0152ca0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f2ab8009f20 tx=0x7f2ab8009840 comp rx=0 tx=0).stop 2023-10-06T03:29:56.439 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:56.429+0000 7f2ac51a2700 1 -- 172.21.15.110:0/2688520638 shutdown_connections 2023-10-06T03:29:56.439 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:56.429+0000 7f2ac51a2700 1 --2- 172.21.15.110:0/2688520638 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2ac01560d0 0x7f2ac0152ca0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:56.439 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:56.429+0000 7f2ac51a2700 1 --2- 172.21.15.110:0/2688520638 >> v2:172.21.15.192:3300/0 conn(0x7f2ac0159c20 0x7f2ac0155b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:56.439 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:56.429+0000 7f2ac51a2700 1 --2- 172.21.15.110:0/2688520638 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2ac006c970 0x7f2ac006cd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:56.440 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:56.429+0000 7f2ac51a2700 1 -- 172.21.15.110:0/2688520638 >> 172.21.15.110:0/2688520638 conn(0x7f2ac006ab30 msgr2=0x7f2ac00adff0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:29:56.440 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:56.429+0000 7f2ac51a2700 1 -- 172.21.15.110:0/2688520638 shutdown_connections 2023-10-06T03:29:56.440 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:56.429+0000 7f2ac51a2700 1 -- 172.21.15.110:0/2688520638 wait complete. 2023-10-06T03:29:56.440 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:56.429+0000 7f2ac51a2700 1 Processor -- start 2023-10-06T03:29:56.441 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:56.429+0000 7f2ac51a2700 1 -- start start 2023-10-06T03:29:56.441 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:56.433+0000 7f2ac51a2700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2ac006c970 0x7f2ac0152a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:56.441 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:56.433+0000 7f2ac51a2700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2ac01560d0 0x7f2ac00cb7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:56.441 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:56.433+0000 7f2ac51a2700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2ac0159c20 0x7f2ac0062dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:56.441 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:56.433+0000 7f2ac51a2700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ac0153c60 con 0x7f2ac006c970 2023-10-06T03:29:56.441 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:56.433+0000 7f2ac51a2700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ac00cbd00 con 0x7f2ac01560d0 2023-10-06T03:29:56.441 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:56.433+0000 7f2ac51a2700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ac01647d0 con 0x7f2ac0159c20 2023-10-06T03:29:56.441 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:56.433+0000 7f2abffff700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2ac006c970 0x7f2ac0152a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:56.441 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:56.433+0000 7f2abffff700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2ac006c970 0x7f2ac0152a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:47838/0 (socket says 172.21.15.110:47838) 2023-10-06T03:29:56.442 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:56.433+0000 7f2abffff700 1 -- 172.21.15.110:0/2846514883 learned_addr learned my addr 172.21.15.110:0/2846514883 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:29:56.442 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:56.433+0000 7f2ab77fe700 1 --2- 172.21.15.110:0/2846514883 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2ac01560d0 0x7f2ac00cb7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:56.442 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:56.433+0000 7f2ac49a1700 1 --2- 172.21.15.110:0/2846514883 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2ac0159c20 0x7f2ac0062dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:56.442 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:56.433+0000 7f2abffff700 1 -- 172.21.15.110:0/2846514883 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2ac0159c20 msgr2=0x7f2ac0062dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:56.443 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:56.433+0000 7f2abffff700 1 --2- 172.21.15.110:0/2846514883 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2ac0159c20 0x7f2ac0062dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:56.443 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:56.433+0000 7f2abffff700 1 -- 172.21.15.110:0/2846514883 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2ac01560d0 msgr2=0x7f2ac00cb7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:56.443 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:56.433+0000 7f2abffff700 1 --2- 172.21.15.110:0/2846514883 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2ac01560d0 0x7f2ac00cb7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:56.443 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:56.433+0000 7f2abffff700 1 -- 172.21.15.110:0/2846514883 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2aa8009420 con 0x7f2ac006c970 2023-10-06T03:29:56.443 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:56.433+0000 7f2abffff700 1 --2- 172.21.15.110:0/2846514883 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2ac006c970 0x7f2ac0152a70 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f2ab0010b60 tx=0x7f2ab000a290 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:29:56.443 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:56.433+0000 7f2abcff9700 1 -- 172.21.15.110:0/2846514883 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ab0005c10 con 0x7f2ac006c970 2023-10-06T03:29:56.443 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:56.433+0000 7f2abcff9700 1 -- 172.21.15.110:0/2846514883 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ab0005d70 con 0x7f2ac006c970 2023-10-06T03:29:56.443 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:56.433+0000 7f2ac51a2700 1 -- 172.21.15.110:0/2846514883 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ab8009ae0 con 0x7f2ac006c970 2023-10-06T03:29:56.443 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:56.433+0000 7f2abcff9700 1 -- 172.21.15.110:0/2846514883 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ab000faa0 con 0x7f2ac006c970 2023-10-06T03:29:56.443 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:56.433+0000 7f2ac51a2700 1 -- 172.21.15.110:0/2846514883 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ac01536a0 con 0x7f2ac006c970 2023-10-06T03:29:56.444 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:56.433+0000 7f2ac51a2700 1 -- 172.21.15.110:0/2846514883 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ac0164940 con 0x7f2ac006c970 2023-10-06T03:29:56.445 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:56.437+0000 7f2abcff9700 1 -- 172.21.15.110:0/2846514883 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7f2ab0017070 con 0x7f2ac006c970 2023-10-06T03:29:56.445 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:56.437+0000 7f2abcff9700 1 --2- 172.21.15.110:0/2846514883 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f2aa0060550 0x7f2aa0062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:56.445 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:56.437+0000 7f2abcff9700 1 -- 172.21.15.110:0/2846514883 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6809+0+0 (secure 0 0 0) 0x7f2ab007ebf0 con 0x7f2ac006c970 2023-10-06T03:29:56.445 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:56.437+0000 7f2ab77fe700 1 --2- 172.21.15.110:0/2846514883 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f2aa0060550 0x7f2aa0062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:56.445 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:56.437+0000 7f2ab77fe700 1 --2- 172.21.15.110:0/2846514883 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f2aa0060550 0x7f2aa0062a00 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f2ac00c8930 tx=0x7f2aa80091a0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:29:56.447 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:56.441+0000 7f2abcff9700 1 -- 172.21.15.110:0/2846514883 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f2ac0164940 con 0x7f2ac006c970 2023-10-06T03:29:56.615 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:56.605+0000 7f2ac51a2700 1 -- 172.21.15.110:0/2846514883 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f2ac0164940 con 0x7f2ac006c970 2023-10-06T03:29:56.616 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:56.605+0000 7f2abcff9700 1 -- 172.21.15.110:0/2846514883 <== mon.0 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v28) v1 ==== 100+0+11 (secure 0 0 0) 0x7f2ac0164940 con 0x7f2ac006c970 2023-10-06T03:29:56.616 INFO:teuthology.orchestra.run.smithi110.stdout:pg_num: 64 2023-10-06T03:29:56.617 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:56.605+0000 7f2ac51a2700 1 -- 172.21.15.110:0/2846514883 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f2aa0060550 msgr2=0x7f2aa0062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:56.617 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:56.609+0000 7f2ac51a2700 1 --2- 172.21.15.110:0/2846514883 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f2aa0060550 0x7f2aa0062a00 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f2ac00c8930 tx=0x7f2aa80091a0 comp rx=0 tx=0).stop 2023-10-06T03:29:56.617 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:56.609+0000 7f2ac51a2700 1 -- 172.21.15.110:0/2846514883 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2ac006c970 msgr2=0x7f2ac0152a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:56.617 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:56.609+0000 7f2ac51a2700 1 --2- 172.21.15.110:0/2846514883 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2ac006c970 0x7f2ac0152a70 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f2ab0010b60 tx=0x7f2ab000a290 comp rx=0 tx=0).stop 2023-10-06T03:29:56.617 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:56.609+0000 7f2ac51a2700 1 -- 172.21.15.110:0/2846514883 shutdown_connections 2023-10-06T03:29:56.618 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:56.609+0000 7f2ac51a2700 1 --2- 172.21.15.110:0/2846514883 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f2aa0060550 0x7f2aa0062a00 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:56.618 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:56.609+0000 7f2ac51a2700 1 --2- 172.21.15.110:0/2846514883 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2ac0159c20 0x7f2ac0062dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:56.618 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:56.609+0000 7f2ac51a2700 1 --2- 172.21.15.110:0/2846514883 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2ac01560d0 0x7f2ac00cb7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:56.618 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:56.609+0000 7f2ac51a2700 1 --2- 172.21.15.110:0/2846514883 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2ac006c970 0x7f2ac0152a70 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:56.618 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:56.609+0000 7f2ac51a2700 1 -- 172.21.15.110:0/2846514883 >> 172.21.15.110:0/2846514883 conn(0x7f2ac006ab30 msgr2=0x7f2ac0158980 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:29:56.618 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:56.609+0000 7f2ac51a2700 1 -- 172.21.15.110:0/2846514883 shutdown_connections 2023-10-06T03:29:56.618 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:56.609+0000 7f2ac51a2700 1 -- 172.21.15.110:0/2846514883 wait complete. 2023-10-06T03:29:56.623 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2023-10-06T03:29:56.737 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:56.733+0000 7f3d1cc0b700 1 Processor -- start 2023-10-06T03:29:56.737 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:56.733+0000 7f3d1cc0b700 1 -- start start 2023-10-06T03:29:56.739 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:56.733+0000 7f3d1cc0b700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f3d180acf90 0x7f3d180cd630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:56.739 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:56.733+0000 7f3d1cc0b700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f3d18069760 0x7f3d180c8ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:56.739 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:56.733+0000 7f3d1cc0b700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d180cd1a0 con 0x7f3d180ad360 2023-10-06T03:29:56.739 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:56.733+0000 7f3d1cc0b700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3d180cdb70 con 0x7f3d180acf90 2023-10-06T03:29:56.739 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:56.733+0000 7f3d1cc0b700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3d180cc590 con 0x7f3d18069760 2023-10-06T03:29:56.740 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:56.733+0000 7f3d1659c700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f3d180acf90 0x7f3d180cd630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:56.740 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:56.733+0000 7f3d15d9b700 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f3d180ad360 0x7f3d18069050 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:40184/0 (socket says 172.21.15.110:40184) 2023-10-06T03:29:56.741 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:56.733+0000 7f3d1659c700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f3d180acf90 0x7f3d180cd630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:47850/0 (socket says 172.21.15.110:47850) 2023-10-06T03:29:56.741 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:56.733+0000 7f3d15d9b700 1 -- 172.21.15.110:0/1649327468 learned_addr learned my addr 172.21.15.110:0/1649327468 (peer_addr_for_me v1:172.21.15.110:0/0) 2023-10-06T03:29:56.741 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:56.733+0000 7f3d16d9d700 1 --2- 172.21.15.110:0/1649327468 >> v2:172.21.15.192:3300/0 conn(0x7f3d18069760 0x7f3d180c8ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:56.742 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:56.737+0000 7f3d1659c700 1 -- 172.21.15.110:0/1649327468 >> v2:172.21.15.192:3300/0 conn(0x7f3d18069760 msgr2=0x7f3d180c8ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:29:56.742 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:56.737+0000 7f3d1659c700 1 --2- 172.21.15.110:0/1649327468 >> v2:172.21.15.192:3300/0 conn(0x7f3d18069760 0x7f3d180c8ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:56.742 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:56.737+0000 7f3d1559a700 1 -- 172.21.15.110:0/1649327468 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2488164915 0 0) 0x7f3d180cd1a0 con 0x7f3d180ad360 2023-10-06T03:29:56.742 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:56.737+0000 7f3d1659c700 1 -- 172.21.15.110:0/1649327468 >> v1:172.21.15.110:6789/0 conn(0x7f3d180ad360 legacy=0x7f3d18069050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:56.742 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:56.737+0000 7f3d1659c700 1 -- 172.21.15.110:0/1649327468 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d180cc870 con 0x7f3d180acf90 2023-10-06T03:29:56.742 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:56.737+0000 7f3d1659c700 1 --2- 172.21.15.110:0/1649327468 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3d180acf90 0x7f3d180cd630 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f3d180ce640 tx=0x7f3d10009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=131810a0d57786c7 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:29:56.745 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:56.737+0000 7f3d1559a700 1 -- 172.21.15.110:0/1649327468 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d1000e070 con 0x7f3d180acf90 2023-10-06T03:29:56.745 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:56.737+0000 7f3d1559a700 1 -- 172.21.15.110:0/1649327468 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d1000bb60 con 0x7f3d180acf90 2023-10-06T03:29:56.745 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:56.737+0000 7f3d1559a700 1 -- 172.21.15.110:0/1649327468 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d10012430 con 0x7f3d180acf90 2023-10-06T03:29:56.745 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:56.737+0000 7f3d1cc0b700 1 -- 172.21.15.110:0/1649327468 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3d180acf90 msgr2=0x7f3d180cd630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:56.745 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:56.737+0000 7f3d1cc0b700 1 --2- 172.21.15.110:0/1649327468 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3d180acf90 0x7f3d180cd630 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f3d180ce640 tx=0x7f3d10009780 comp rx=0 tx=0).stop 2023-10-06T03:29:56.745 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:56.737+0000 7f3d1cc0b700 1 -- 172.21.15.110:0/1649327468 shutdown_connections 2023-10-06T03:29:56.745 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:56.737+0000 7f3d1cc0b700 1 --2- 172.21.15.110:0/1649327468 >> v2:172.21.15.192:3300/0 conn(0x7f3d18069760 0x7f3d180c8ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:56.745 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:56.737+0000 7f3d1cc0b700 1 --2- 172.21.15.110:0/1649327468 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3d180acf90 0x7f3d180cd630 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:56.746 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:56.737+0000 7f3d1cc0b700 1 -- 172.21.15.110:0/1649327468 >> 172.21.15.110:0/1649327468 conn(0x7f3d180684c0 msgr2=0x7f3d180cde80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:29:56.746 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:56.737+0000 7f3d1cc0b700 1 -- 172.21.15.110:0/1649327468 shutdown_connections 2023-10-06T03:29:56.746 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:56.737+0000 7f3d1cc0b700 1 -- 172.21.15.110:0/1649327468 wait complete. 2023-10-06T03:29:56.746 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:56.741+0000 7f3d1cc0b700 1 Processor -- start 2023-10-06T03:29:56.746 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:56.741+0000 7f3d1cc0b700 1 -- start start 2023-10-06T03:29:56.748 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:56.741+0000 7f3d1cc0b700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3d18069760 0x7f3d18157640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:56.748 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:56.741+0000 7f3d1cc0b700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3d180acf90 0x7f3d180b0e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:56.748 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:56.741+0000 7f3d1cc0b700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3d180ad360 0x7f3d18159ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:56.748 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:56.741+0000 7f3d1cc0b700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d180c7ec0 con 0x7f3d180acf90 2023-10-06T03:29:56.748 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:56.741+0000 7f3d1cc0b700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d180c7900 con 0x7f3d180ad360 2023-10-06T03:29:56.749 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:56.741+0000 7f3d1cc0b700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d180c7be0 con 0x7f3d18069760 2023-10-06T03:29:56.749 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:56.741+0000 7f3d15d9b700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3d180acf90 0x7f3d180b0e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:56.749 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:56.741+0000 7f3d15d9b700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3d180acf90 0x7f3d180b0e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:47862/0 (socket says 172.21.15.110:47862) 2023-10-06T03:29:56.749 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:56.741+0000 7f3d15d9b700 1 -- 172.21.15.110:0/659718009 learned_addr learned my addr 172.21.15.110:0/659718009 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:29:56.749 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:56.741+0000 7f3d1659c700 1 --2- 172.21.15.110:0/659718009 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3d18069760 0x7f3d18157640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:56.749 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:56.741+0000 7f3d16d9d700 1 --2- 172.21.15.110:0/659718009 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3d180ad360 0x7f3d18159ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:56.749 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:56.741+0000 7f3d15d9b700 1 -- 172.21.15.110:0/659718009 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3d18069760 msgr2=0x7f3d18157640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:56.749 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:56.741+0000 7f3d15d9b700 1 --2- 172.21.15.110:0/659718009 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3d18069760 0x7f3d18157640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:56.750 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:56.741+0000 7f3d15d9b700 1 -- 172.21.15.110:0/659718009 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3d180ad360 msgr2=0x7f3d18159ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:56.750 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:56.741+0000 7f3d15d9b700 1 --2- 172.21.15.110:0/659718009 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3d180ad360 0x7f3d18159ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:56.750 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:56.741+0000 7f3d15d9b700 1 -- 172.21.15.110:0/659718009 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d10009a20 con 0x7f3d180acf90 2023-10-06T03:29:56.750 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:56.741+0000 7f3d15d9b700 1 --2- 172.21.15.110:0/659718009 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3d180acf90 0x7f3d180b0e40 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f3d0000c5b0 tx=0x7f3d00001070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:29:56.750 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:56.741+0000 7f3d0e7fc700 1 -- 172.21.15.110:0/659718009 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d00005b30 con 0x7f3d180acf90 2023-10-06T03:29:56.750 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:56.741+0000 7f3d0e7fc700 1 -- 172.21.15.110:0/659718009 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d00005c90 con 0x7f3d180acf90 2023-10-06T03:29:56.750 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:56.741+0000 7f3d1cc0b700 1 -- 172.21.15.110:0/659718009 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d181599b0 con 0x7f3d180acf90 2023-10-06T03:29:56.750 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:56.741+0000 7f3d0e7fc700 1 -- 172.21.15.110:0/659718009 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d00007300 con 0x7f3d180acf90 2023-10-06T03:29:56.751 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:56.741+0000 7f3d1cc0b700 1 -- 172.21.15.110:0/659718009 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d180b1520 con 0x7f3d180acf90 2023-10-06T03:29:56.752 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:56.741+0000 7f3d1cc0b700 1 -- 172.21.15.110:0/659718009 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d1806ff60 con 0x7f3d180acf90 2023-10-06T03:29:56.752 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:56.745+0000 7f3d0e7fc700 1 -- 172.21.15.110:0/659718009 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7f3d0001b3b0 con 0x7f3d180acf90 2023-10-06T03:29:56.752 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:56.745+0000 7f3d0e7fc700 1 --2- 172.21.15.110:0/659718009 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f3cf8060550 0x7f3cf8062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:56.752 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:56.745+0000 7f3d1659c700 1 --2- 172.21.15.110:0/659718009 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f3cf8060550 0x7f3cf8062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:56.752 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:56.745+0000 7f3d0e7fc700 1 -- 172.21.15.110:0/659718009 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6809+0+0 (secure 0 0 0) 0x7f3d0007c1c0 con 0x7f3d180acf90 2023-10-06T03:29:56.752 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:56.745+0000 7f3d1659c700 1 --2- 172.21.15.110:0/659718009 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f3cf8060550 0x7f3cf8062a00 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f3d10003440 tx=0x7f3d10019000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:29:56.759 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:56.753+0000 7f3d0e7fc700 1 -- 172.21.15.110:0/659718009 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f3d0004c2c0 con 0x7f3d180acf90 2023-10-06T03:29:56.917 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:56.913+0000 7f3d1cc0b700 1 -- 172.21.15.110:0/659718009 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f3d180b3c70 con 0x7f3d180acf90 2023-10-06T03:29:56.918 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:56.913+0000 7f3d0e7fc700 1 -- 172.21.15.110:0/659718009 <== mon.0 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v28) v1 ==== 103+0+11 (secure 0 0 0) 0x7f3d0004f8e0 con 0x7f3d180acf90 2023-10-06T03:29:56.918 INFO:teuthology.orchestra.run.smithi110.stdout:pg_num: 32 2023-10-06T03:29:56.920 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:56.917+0000 7f3d1cc0b700 1 -- 172.21.15.110:0/659718009 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f3cf8060550 msgr2=0x7f3cf8062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:56.920 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:56.917+0000 7f3d1cc0b700 1 --2- 172.21.15.110:0/659718009 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f3cf8060550 0x7f3cf8062a00 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f3d10003440 tx=0x7f3d10019000 comp rx=0 tx=0).stop 2023-10-06T03:29:56.921 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:56.917+0000 7f3d1cc0b700 1 -- 172.21.15.110:0/659718009 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3d180acf90 msgr2=0x7f3d180b0e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:56.921 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:56.917+0000 7f3d1cc0b700 1 --2- 172.21.15.110:0/659718009 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3d180acf90 0x7f3d180b0e40 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f3d0000c5b0 tx=0x7f3d00001070 comp rx=0 tx=0).stop 2023-10-06T03:29:56.921 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:56.917+0000 7f3d1cc0b700 1 -- 172.21.15.110:0/659718009 shutdown_connections 2023-10-06T03:29:56.921 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:56.917+0000 7f3d1cc0b700 1 --2- 172.21.15.110:0/659718009 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f3cf8060550 0x7f3cf8062a00 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:56.921 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:56.917+0000 7f3d1cc0b700 1 --2- 172.21.15.110:0/659718009 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3d180ad360 0x7f3d18159ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:56.921 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:56.917+0000 7f3d1cc0b700 1 --2- 172.21.15.110:0/659718009 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3d180acf90 0x7f3d180b0e40 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:56.921 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:56.917+0000 7f3d1cc0b700 1 --2- 172.21.15.110:0/659718009 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3d18069760 0x7f3d18157640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:56.921 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:56.917+0000 7f3d1cc0b700 1 -- 172.21.15.110:0/659718009 >> 172.21.15.110:0/659718009 conn(0x7f3d180684c0 msgr2=0x7f3d180afc30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:29:56.922 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:56.917+0000 7f3d1cc0b700 1 -- 172.21.15.110:0/659718009 shutdown_connections 2023-10-06T03:29:56.922 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:56.917+0000 7f3d1cc0b700 1 -- 172.21.15.110:0/659718009 wait complete. 2023-10-06T03:29:56.935 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-06T03:29:57.048 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:57.045+0000 7f4406372700 1 Processor -- start 2023-10-06T03:29:57.049 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:57.045+0000 7f4406372700 1 -- start start 2023-10-06T03:29:57.050 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:57.045+0000 7f4406372700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f44000aefa0 0x7f44000af370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:57.050 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:57.045+0000 7f4406372700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f44000ce1b0 0x7f44000cb140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:57.050 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:57.045+0000 7f4406372700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44000cf4a0 con 0x7f44000cf930 2023-10-06T03:29:57.051 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:57.045+0000 7f4406372700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f44000ce5e0 con 0x7f44000ce1b0 2023-10-06T03:29:57.051 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:57.045+0000 7f4406372700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44000cfdb0 con 0x7f44000aefa0 2023-10-06T03:29:57.051 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:57.045+0000 7f440490f700 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f44000cf930 0x7f440015cb90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:40186/0 (socket says 172.21.15.110:40186) 2023-10-06T03:29:57.051 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:57.045+0000 7f440490f700 1 -- 172.21.15.110:0/2561369662 learned_addr learned my addr 172.21.15.110:0/2561369662 (peer_addr_for_me v1:172.21.15.110:0/0) 2023-10-06T03:29:57.051 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:57.045+0000 7f43fffff700 1 --2- 172.21.15.110:0/2561369662 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f44000aefa0 0x7f44000af370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:57.051 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:57.045+0000 7f43ff7fe700 1 --2- 172.21.15.110:0/2561369662 >> v2:172.21.15.192:3300/0 conn(0x7f44000ce1b0 0x7f44000cb140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:57.051 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:57.045+0000 7f43feffd700 1 -- 172.21.15.110:0/2561369662 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1321192258 0 0) 0x7f44000cf4a0 con 0x7f44000cf930 2023-10-06T03:29:57.051 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:57.045+0000 7f43feffd700 1 -- 172.21.15.110:0/2561369662 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f43e8003570 con 0x7f44000cf930 2023-10-06T03:29:57.051 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:57.045+0000 7f43fffff700 1 -- 172.21.15.110:0/2561369662 >> v2:172.21.15.192:3300/0 conn(0x7f44000ce1b0 msgr2=0x7f44000cb140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:29:57.051 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:57.045+0000 7f43fffff700 1 --2- 172.21.15.110:0/2561369662 >> v2:172.21.15.192:3300/0 conn(0x7f44000ce1b0 0x7f44000cb140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:57.052 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:57.045+0000 7f43fffff700 1 -- 172.21.15.110:0/2561369662 >> v1:172.21.15.110:6789/0 conn(0x7f44000cf930 legacy=0x7f440015cb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:57.052 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:57.045+0000 7f43fffff700 1 -- 172.21.15.110:0/2561369662 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44000ce890 con 0x7f44000aefa0 2023-10-06T03:29:57.053 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:57.045+0000 7f43fffff700 1 --2- 172.21.15.110:0/2561369662 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f44000aefa0 0x7f44000af370 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f43f8009e60 tx=0x7f43f8009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dd58b8c4623569d6 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:29:57.053 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:57.045+0000 7f43feffd700 1 -- 172.21.15.110:0/2561369662 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43f800e070 con 0x7f44000aefa0 2023-10-06T03:29:57.053 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:57.049+0000 7f43feffd700 1 -- 172.21.15.110:0/2561369662 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43f800bab0 con 0x7f44000aefa0 2023-10-06T03:29:57.053 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:57.049+0000 7f43feffd700 1 -- 172.21.15.110:0/2561369662 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43f800be70 con 0x7f44000aefa0 2023-10-06T03:29:57.053 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:57.049+0000 7f4406372700 1 -- 172.21.15.110:0/2561369662 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f44000aefa0 msgr2=0x7f44000af370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:57.053 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:57.049+0000 7f4406372700 1 --2- 172.21.15.110:0/2561369662 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f44000aefa0 0x7f44000af370 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f43f8009e60 tx=0x7f43f8009780 comp rx=0 tx=0).stop 2023-10-06T03:29:57.053 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:57.049+0000 7f4406372700 1 -- 172.21.15.110:0/2561369662 shutdown_connections 2023-10-06T03:29:57.054 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:57.049+0000 7f4406372700 1 --2- 172.21.15.110:0/2561369662 >> v2:172.21.15.192:3300/0 conn(0x7f44000ce1b0 0x7f44000cb140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:57.054 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:57.049+0000 7f4406372700 1 --2- 172.21.15.110:0/2561369662 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f44000aefa0 0x7f44000af370 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:57.054 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:57.049+0000 7f4406372700 1 -- 172.21.15.110:0/2561369662 >> 172.21.15.110:0/2561369662 conn(0x7f440006dda0 msgr2=0x7f440006e170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:29:57.054 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:57.049+0000 7f4406372700 1 -- 172.21.15.110:0/2561369662 shutdown_connections 2023-10-06T03:29:57.054 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:57.049+0000 7f4406372700 1 -- 172.21.15.110:0/2561369662 wait complete. 2023-10-06T03:29:57.055 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:57.049+0000 7f4406372700 1 Processor -- start 2023-10-06T03:29:57.055 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:57.049+0000 7f4406372700 1 -- start start 2023-10-06T03:29:57.056 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:57.053+0000 7f4406372700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f44000aefa0 0x7f440016a7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:57.056 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:57.053+0000 7f4406372700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f44000ce1b0 0x7f44001533e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:57.056 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:57.053+0000 7f4406372700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f44000cf930 0x7f44000b11f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:57.056 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:57.053+0000 7f4406372700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4400153920 con 0x7f44000cf930 2023-10-06T03:29:57.057 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:57.053+0000 7f4406372700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f440016ca50 con 0x7f44000aefa0 2023-10-06T03:29:57.057 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:57.053+0000 7f4406372700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4400159680 con 0x7f44000ce1b0 2023-10-06T03:29:57.057 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:57.053+0000 7f440490f700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f44000cf930 0x7f44000b11f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:57.057 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:57.053+0000 7f440490f700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f44000cf930 0x7f44000b11f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:47878/0 (socket says 172.21.15.110:47878) 2023-10-06T03:29:57.057 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:57.053+0000 7f440490f700 1 -- 172.21.15.110:0/3016000793 learned_addr learned my addr 172.21.15.110:0/3016000793 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:29:57.058 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:57.053+0000 7f43ff7fe700 1 --2- 172.21.15.110:0/3016000793 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f44000ce1b0 0x7f44001533e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:57.058 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:57.053+0000 7f43fffff700 1 --2- 172.21.15.110:0/3016000793 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f44000aefa0 0x7f440016a7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:57.058 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:57.053+0000 7f440490f700 1 -- 172.21.15.110:0/3016000793 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f44000ce1b0 msgr2=0x7f44001533e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:57.058 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:57.053+0000 7f440490f700 1 --2- 172.21.15.110:0/3016000793 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f44000ce1b0 0x7f44001533e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:57.058 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:57.053+0000 7f440490f700 1 -- 172.21.15.110:0/3016000793 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f44000aefa0 msgr2=0x7f440016a7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:57.058 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:57.053+0000 7f440490f700 1 --2- 172.21.15.110:0/3016000793 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f44000aefa0 0x7f440016a7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:57.058 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:57.053+0000 7f440490f700 1 -- 172.21.15.110:0/3016000793 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43f8009a20 con 0x7f44000cf930 2023-10-06T03:29:57.059 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:57.053+0000 7f440490f700 1 --2- 172.21.15.110:0/3016000793 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f44000cf930 0x7f44000b11f0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f43f000c560 tx=0x7f43f0007dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:29:57.059 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:57.053+0000 7f43e7fff700 1 -- 172.21.15.110:0/3016000793 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43f0005c00 con 0x7f44000cf930 2023-10-06T03:29:57.059 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:57.053+0000 7f43e7fff700 1 -- 172.21.15.110:0/3016000793 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43f0005d60 con 0x7f44000cf930 2023-10-06T03:29:57.060 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:57.053+0000 7f4406372700 1 -- 172.21.15.110:0/3016000793 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f440015a200 con 0x7f44000cf930 2023-10-06T03:29:57.060 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:57.053+0000 7f43e7fff700 1 -- 172.21.15.110:0/3016000793 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43f00072e0 con 0x7f44000cf930 2023-10-06T03:29:57.060 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:57.053+0000 7f4406372700 1 -- 172.21.15.110:0/3016000793 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4400159bb0 con 0x7f44000cf930 2023-10-06T03:29:57.063 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:57.053+0000 7f43e7fff700 1 -- 172.21.15.110:0/3016000793 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7f43f0012400 con 0x7f44000cf930 2023-10-06T03:29:57.063 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:57.057+0000 7f4406372700 1 -- 172.21.15.110:0/3016000793 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f440015a390 con 0x7f44000cf930 2023-10-06T03:29:57.063 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:57.057+0000 7f43e7fff700 1 --2- 172.21.15.110:0/3016000793 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f43ec060550 0x7f43ec062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:57.063 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:57.057+0000 7f43e7fff700 1 -- 172.21.15.110:0/3016000793 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6809+0+0 (secure 0 0 0) 0x7f43f007c0c0 con 0x7f44000cf930 2023-10-06T03:29:57.063 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:57.057+0000 7f43fffff700 1 --2- 172.21.15.110:0/3016000793 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f43ec060550 0x7f43ec062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:57.063 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:57.057+0000 7f43fffff700 1 --2- 172.21.15.110:0/3016000793 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f43ec060550 0x7f43ec062a00 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f43f80097c0 tx=0x7f43f8019000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:29:57.068 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:57.061+0000 7f43e7fff700 1 -- 172.21.15.110:0/3016000793 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f43f0049be0 con 0x7f44000cf930 2023-10-06T03:29:57.264 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:57.261+0000 7f4406372700 1 -- 172.21.15.110:0/3016000793 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f440006ff60 con 0x7f44000cf930 2023-10-06T03:29:57.268 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:57.261+0000 7f43e7fff700 1 -- 172.21.15.110:0/3016000793 <== mon.0 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4602 (secure 0 0 0) 0x7f43f0053e60 con 0x7f44000cf930 2023-10-06T03:29:57.268 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-10-06T03:29:57.269 INFO:teuthology.orchestra.run.smithi110.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4279,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6833/1434505700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6832","nonce":1434505700},{"type":"v1","addr":"172.21.15.192:6833","nonce":1434505700}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4281,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/1168370789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6834","nonce":1168370789},{"type":"v1","addr":"172.21.15.110:6835","nonce":1168370789}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4288,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6835/1271810086","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":1271810086},{"type":"v1","addr":"172.21.15.192:6835","nonce":1271810086}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-06T03:29:33.894998+0000","modified":"2023-10-06T03:29:47.307075+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4293},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4293":{"gid":4293,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.110:6837/540320464","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6836","nonce":540320464},{"type":"v1","addr":"172.21.15.110:6837","nonce":540320464}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-06T03:29:57.270 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:57.265+0000 7f4406372700 1 -- 172.21.15.110:0/3016000793 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f43ec060550 msgr2=0x7f43ec062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:57.270 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:57.265+0000 7f4406372700 1 --2- 172.21.15.110:0/3016000793 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f43ec060550 0x7f43ec062a00 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f43f80097c0 tx=0x7f43f8019000 comp rx=0 tx=0).stop 2023-10-06T03:29:57.271 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:57.265+0000 7f4406372700 1 -- 172.21.15.110:0/3016000793 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f44000cf930 msgr2=0x7f44000b11f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:57.271 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:57.265+0000 7f4406372700 1 --2- 172.21.15.110:0/3016000793 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f44000cf930 0x7f44000b11f0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f43f000c560 tx=0x7f43f0007dd0 comp rx=0 tx=0).stop 2023-10-06T03:29:57.271 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:57.265+0000 7f4406372700 1 -- 172.21.15.110:0/3016000793 shutdown_connections 2023-10-06T03:29:57.271 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:57.265+0000 7f4406372700 1 --2- 172.21.15.110:0/3016000793 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f43ec060550 0x7f43ec062a00 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:57.271 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:57.265+0000 7f4406372700 1 --2- 172.21.15.110:0/3016000793 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f44000cf930 0x7f44000b11f0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:57.271 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:57.265+0000 7f4406372700 1 --2- 172.21.15.110:0/3016000793 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f44000ce1b0 0x7f44001533e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:57.271 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:57.265+0000 7f4406372700 1 --2- 172.21.15.110:0/3016000793 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f44000aefa0 0x7f440016a7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:57.271 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:57.265+0000 7f4406372700 1 -- 172.21.15.110:0/3016000793 >> 172.21.15.110:0/3016000793 conn(0x7f440006dda0 msgr2=0x7f440015dc90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:29:57.271 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:57.265+0000 7f4406372700 1 -- 172.21.15.110:0/3016000793 shutdown_connections 2023-10-06T03:29:57.272 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:57.265+0000 7f4406372700 1 -- 172.21.15.110:0/3016000793 wait complete. 2023-10-06T03:29:57.272 INFO:teuthology.orchestra.run.smithi110.stderr:dumped fsmap epoch 8 2023-10-06T03:29:57.281 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-06T03:29:57.383 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:57.381+0000 7fdfea1b9700 1 Processor -- start 2023-10-06T03:29:57.383 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:57.381+0000 7fdfea1b9700 1 -- start start 2023-10-06T03:29:57.385 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:57.381+0000 7fdfea1b9700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fdfe40aefa0 0x7fdfe40af370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:57.385 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:57.381+0000 7fdfea1b9700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fdfe40ce1b0 0x7fdfe40cb140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:57.385 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:57.381+0000 7fdfea1b9700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdfe40cf4a0 con 0x7fdfe40cf930 2023-10-06T03:29:57.385 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:57.381+0000 7fdfea1b9700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdfe40ce5e0 con 0x7fdfe40aefa0 2023-10-06T03:29:57.385 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:57.381+0000 7fdfea1b9700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdfe40ce890 con 0x7fdfe40ce1b0 2023-10-06T03:29:57.385 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:57.381+0000 7fdfe3fff700 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7fdfe40cf930 0x7fdfe415cb90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:40192/0 (socket says 172.21.15.110:40192) 2023-10-06T03:29:57.386 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:57.381+0000 7fdfe3fff700 1 -- 172.21.15.110:0/1191348642 learned_addr learned my addr 172.21.15.110:0/1191348642 (peer_addr_for_me v1:172.21.15.110:0/0) 2023-10-06T03:29:57.386 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:57.381+0000 7fdfe37fe700 1 --2- 172.21.15.110:0/1191348642 >> v2:172.21.15.110:3300/0 conn(0x7fdfe40aefa0 0x7fdfe40af370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:57.386 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:57.381+0000 7fdfdbfff700 1 --2- 172.21.15.110:0/1191348642 >> v2:172.21.15.192:3300/0 conn(0x7fdfe40ce1b0 0x7fdfe40cb140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:57.386 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:57.381+0000 7fdfe2ffd700 1 -- 172.21.15.110:0/1191348642 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 431326765 0 0) 0x7fdfe40cf4a0 con 0x7fdfe40cf930 2023-10-06T03:29:57.386 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:57.381+0000 7fdfe2ffd700 1 -- 172.21.15.110:0/1191348642 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdfd0003570 con 0x7fdfe40cf930 2023-10-06T03:29:57.386 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:57.381+0000 7fdfe37fe700 1 -- 172.21.15.110:0/1191348642 >> v2:172.21.15.192:3300/0 conn(0x7fdfe40ce1b0 msgr2=0x7fdfe40cb140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:29:57.386 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:57.381+0000 7fdfe37fe700 1 --2- 172.21.15.110:0/1191348642 >> v2:172.21.15.192:3300/0 conn(0x7fdfe40ce1b0 0x7fdfe40cb140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:57.386 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:57.381+0000 7fdfe37fe700 1 -- 172.21.15.110:0/1191348642 >> v1:172.21.15.110:6789/0 conn(0x7fdfe40cf930 legacy=0x7fdfe415cb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:57.386 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:57.381+0000 7fdfe37fe700 1 -- 172.21.15.110:0/1191348642 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfe40ceb70 con 0x7fdfe40aefa0 2023-10-06T03:29:57.388 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:57.381+0000 7fdfe37fe700 1 --2- 172.21.15.110:0/1191348642 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdfe40aefa0 0x7fdfe40af370 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fdfe406efc0 tx=0x7fdfd40091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=458c22ec50622a27 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:29:57.388 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:57.381+0000 7fdfe2ffd700 1 -- 172.21.15.110:0/1191348642 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfd4011630 con 0x7fdfe40aefa0 2023-10-06T03:29:57.388 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:57.381+0000 7fdfe2ffd700 1 -- 172.21.15.110:0/1191348642 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdfd40117b0 con 0x7fdfe40aefa0 2023-10-06T03:29:57.388 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:57.381+0000 7fdfe2ffd700 1 -- 172.21.15.110:0/1191348642 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfd4011b70 con 0x7fdfe40aefa0 2023-10-06T03:29:57.388 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:57.381+0000 7fdfea1b9700 1 -- 172.21.15.110:0/1191348642 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdfe40aefa0 msgr2=0x7fdfe40af370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:57.389 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:57.381+0000 7fdfea1b9700 1 --2- 172.21.15.110:0/1191348642 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdfe40aefa0 0x7fdfe40af370 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fdfe406efc0 tx=0x7fdfd40091a0 comp rx=0 tx=0).stop 2023-10-06T03:29:57.389 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:57.381+0000 7fdfea1b9700 1 -- 172.21.15.110:0/1191348642 shutdown_connections 2023-10-06T03:29:57.389 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:57.381+0000 7fdfea1b9700 1 --2- 172.21.15.110:0/1191348642 >> v2:172.21.15.192:3300/0 conn(0x7fdfe40ce1b0 0x7fdfe40cb140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:57.389 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:57.381+0000 7fdfea1b9700 1 --2- 172.21.15.110:0/1191348642 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdfe40aefa0 0x7fdfe40af370 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:57.389 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:57.381+0000 7fdfea1b9700 1 -- 172.21.15.110:0/1191348642 >> 172.21.15.110:0/1191348642 conn(0x7fdfe406dda0 msgr2=0x7fdfe406e170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:29:57.389 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:57.381+0000 7fdfea1b9700 1 -- 172.21.15.110:0/1191348642 shutdown_connections 2023-10-06T03:29:57.389 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:57.381+0000 7fdfea1b9700 1 -- 172.21.15.110:0/1191348642 wait complete. 2023-10-06T03:29:57.390 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:57.385+0000 7fdfea1b9700 1 Processor -- start 2023-10-06T03:29:57.390 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:57.385+0000 7fdfea1b9700 1 -- start start 2023-10-06T03:29:57.392 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:57.385+0000 7fdfea1b9700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdfe40aefa0 0x7fdfe415c010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:57.393 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:57.385+0000 7fdfea1b9700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdfe40ce1b0 0x7fdfe40c7070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:57.393 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:57.385+0000 7fdfea1b9700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fdfe40cf930 0x7fdfe40b1560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:57.393 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:57.385+0000 7fdfea1b9700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfe4144fe0 con 0x7fdfe40ce1b0 2023-10-06T03:29:57.393 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:57.385+0000 7fdfea1b9700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfe40c75b0 con 0x7fdfe40aefa0 2023-10-06T03:29:57.394 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:57.385+0000 7fdfea1b9700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdfe40b10d0 con 0x7fdfe40cf930 2023-10-06T03:29:57.394 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:57.385+0000 7fdfdbfff700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdfe40ce1b0 0x7fdfe40c7070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:57.394 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:57.385+0000 7fdfdbfff700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdfe40ce1b0 0x7fdfe40c7070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:47892/0 (socket says 172.21.15.110:47892) 2023-10-06T03:29:57.395 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:57.385+0000 7fdfdbfff700 1 -- 172.21.15.110:0/4142435989 learned_addr learned my addr 172.21.15.110:0/4142435989 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:29:57.395 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:57.385+0000 7fdfe3fff700 1 --2- 172.21.15.110:0/4142435989 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fdfe40cf930 0x7fdfe40b1560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:57.395 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:57.385+0000 7fdfe37fe700 1 --2- 172.21.15.110:0/4142435989 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdfe40aefa0 0x7fdfe415c010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:57.395 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:57.385+0000 7fdfdbfff700 1 -- 172.21.15.110:0/4142435989 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fdfe40cf930 msgr2=0x7fdfe40b1560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:57.395 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:57.385+0000 7fdfdbfff700 1 --2- 172.21.15.110:0/4142435989 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fdfe40cf930 0x7fdfe40b1560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:57.395 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:57.385+0000 7fdfdbfff700 1 -- 172.21.15.110:0/4142435989 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdfe40aefa0 msgr2=0x7fdfe415c010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:57.395 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:57.385+0000 7fdfdbfff700 1 --2- 172.21.15.110:0/4142435989 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdfe40aefa0 0x7fdfe415c010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:57.395 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:57.385+0000 7fdfdbfff700 1 -- 172.21.15.110:0/4142435989 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfd4009420 con 0x7fdfe40ce1b0 2023-10-06T03:29:57.396 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:57.385+0000 7fdfdbfff700 1 --2- 172.21.15.110:0/4142435989 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdfe40ce1b0 0x7fdfe40c7070 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fdfcc00ec80 tx=0x7fdfcc0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:29:57.396 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:57.385+0000 7fdfdb7fe700 1 -- 172.21.15.110:0/4142435989 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfcc00c9d0 con 0x7fdfe40ce1b0 2023-10-06T03:29:57.396 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:57.385+0000 7fdfdb7fe700 1 -- 172.21.15.110:0/4142435989 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdfcc00cb30 con 0x7fdfe40ce1b0 2023-10-06T03:29:57.396 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:57.385+0000 7fdfea1b9700 1 -- 172.21.15.110:0/4142435989 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdfe40c9b80 con 0x7fdfe40ce1b0 2023-10-06T03:29:57.396 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:57.385+0000 7fdfdb7fe700 1 -- 172.21.15.110:0/4142435989 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfcc013470 con 0x7fdfe40ce1b0 2023-10-06T03:29:57.396 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:57.385+0000 7fdfea1b9700 1 -- 172.21.15.110:0/4142435989 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdfe40b0d60 con 0x7fdfe40ce1b0 2023-10-06T03:29:57.396 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:57.385+0000 7fdfea1b9700 1 -- 172.21.15.110:0/4142435989 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdfe406ff60 con 0x7fdfe40ce1b0 2023-10-06T03:29:57.396 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:57.389+0000 7fdfdb7fe700 1 -- 172.21.15.110:0/4142435989 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7fdfcc0137e0 con 0x7fdfe40ce1b0 2023-10-06T03:29:57.396 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:57.389+0000 7fdfdb7fe700 1 --2- 172.21.15.110:0/4142435989 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fdfc4060550 0x7fdfc4062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:57.396 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:57.389+0000 7fdfdb7fe700 1 -- 172.21.15.110:0/4142435989 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6809+0+0 (secure 0 0 0) 0x7fdfcc07cfd0 con 0x7fdfe40ce1b0 2023-10-06T03:29:57.396 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:57.389+0000 7fdfe37fe700 1 --2- 172.21.15.110:0/4142435989 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fdfc4060550 0x7fdfc4062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:57.397 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:57.389+0000 7fdfe37fe700 1 --2- 172.21.15.110:0/4142435989 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fdfc4060550 0x7fdfc4062a00 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fdfe406efc0 tx=0x7fdfd4017000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:29:57.397 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:57.393+0000 7fdfdb7fe700 1 -- 172.21.15.110:0/4142435989 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fdfcc04d0d0 con 0x7fdfe40ce1b0 2023-10-06T03:29:57.558 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:57.553+0000 7fdfea1b9700 1 -- 172.21.15.110:0/4142435989 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdfe40b1210 con 0x7fdfe40ce1b0 2023-10-06T03:29:57.559 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:57.553+0000 7fdfdb7fe700 1 -- 172.21.15.110:0/4142435989 <== mon.0 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v28) v1 ==== 74+0+18755 (secure 0 0 0) 0x7fdfcc0506f0 con 0x7fdfe40ce1b0 2023-10-06T03:29:57.559 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-10-06T03:29:57.559 INFO:teuthology.orchestra.run.smithi110.stdout:{"epoch":28,"fsid":"93b8f6ac-4c92-4b45-b8e4-b59617024aa2","created":"2023-10-06T03:29:10.519583+0000","modified":"2023-10-06T03:29:38.951740+0000","last_up_change":"2023-10-06T03:29:21.960511+0000","last_in_change":"2023-10-06T03:29:12.560436+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-06T03:29:20.703743+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2023-10-06T03:29:28.307281+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2023-10-06T03:29:32.247251+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data","create_time":"2023-10-06T03:29:32.880086+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}}},{"pool":5,"pool_name":"cephfs_data_ec","create_time":"2023-10-06T03:29:36.613779+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"28","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"5866eb6c-2adc-46e4-be7a-e5bcda0f4065","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6824","nonce":33394},{"type":"v1","addr":"172.21.15.110:6825","nonce":33394}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6826","nonce":33394},{"type":"v1","addr":"172.21.15.110:6827","nonce":33394}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6830","nonce":33394},{"type":"v1","addr":"172.21.15.110:6831","nonce":33394}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6828","nonce":33394},{"type":"v1","addr":"172.21.15.110:6829","nonce":33394}]},"public_addr":"172.21.15.110:6825/33394","cluster_addr":"172.21.15.110:6827/33394","heartbeat_back_addr":"172.21.15.110:6831/33394","heartbeat_front_addr":"172.21.15.110:6829/33394","state":["exists","up"]},{"osd":1,"uuid":"a38de118-062b-4a05-9782-2ceddf6ff4df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6800","nonce":33396},{"type":"v1","addr":"172.21.15.110:6801","nonce":33396}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6802","nonce":33396},{"type":"v1","addr":"172.21.15.110:6803","nonce":33396}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6806","nonce":33396},{"type":"v1","addr":"172.21.15.110:6807","nonce":33396}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6804","nonce":33396},{"type":"v1","addr":"172.21.15.110:6805","nonce":33396}]},"public_addr":"172.21.15.110:6801/33396","cluster_addr":"172.21.15.110:6803/33396","heartbeat_back_addr":"172.21.15.110:6807/33396","heartbeat_front_addr":"172.21.15.110:6805/33396","state":["exists","up"]},{"osd":2,"uuid":"56e39be2-3407-4b69-877e-cf16617b06d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6816","nonce":33397},{"type":"v1","addr":"172.21.15.110:6817","nonce":33397}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6818","nonce":33397},{"type":"v1","addr":"172.21.15.110:6819","nonce":33397}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6822","nonce":33397},{"type":"v1","addr":"172.21.15.110:6823","nonce":33397}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6820","nonce":33397},{"type":"v1","addr":"172.21.15.110:6821","nonce":33397}]},"public_addr":"172.21.15.110:6817/33397","cluster_addr":"172.21.15.110:6819/33397","heartbeat_back_addr":"172.21.15.110:6823/33397","heartbeat_front_addr":"172.21.15.110:6821/33397","state":["exists","up"]},{"osd":3,"uuid":"01c82d13-c1d4-4bde-b55e-3e0089b40115","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6808","nonce":33395},{"type":"v1","addr":"172.21.15.110:6809","nonce":33395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6810","nonce":33395},{"type":"v1","addr":"172.21.15.110:6811","nonce":33395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6814","nonce":33395},{"type":"v1","addr":"172.21.15.110:6815","nonce":33395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6812","nonce":33395},{"type":"v1","addr":"172.21.15.110:6813","nonce":33395}]},"public_addr":"172.21.15.110:6809/33395","cluster_addr":"172.21.15.110:6811/33395","heartbeat_back_addr":"172.21.15.110:6815/33395","heartbeat_front_addr":"172.21.15.110:6813/33395","state":["exists","up"]},{"osd":4,"uuid":"86983430-5fa3-4d17-998f-03e2cd843237","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":33683},{"type":"v1","addr":"172.21.15.192:6817","nonce":33683}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":33683},{"type":"v1","addr":"172.21.15.192:6819","nonce":33683}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":33683},{"type":"v1","addr":"172.21.15.192:6823","nonce":33683}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6820","nonce":33683},{"type":"v1","addr":"172.21.15.192:6821","nonce":33683}]},"public_addr":"172.21.15.192:6817/33683","cluster_addr":"172.21.15.192:6819/33683","heartbeat_back_addr":"172.21.15.192:6823/33683","heartbeat_front_addr":"172.21.15.192:6821/33683","state":["exists","up"]},{"osd":5,"uuid":"6945257f-821d-4619-a63f-ebe857eba6f4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":33684},{"type":"v1","addr":"172.21.15.192:6801","nonce":33684}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":33684},{"type":"v1","addr":"172.21.15.192:6803","nonce":33684}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":33684},{"type":"v1","addr":"172.21.15.192:6807","nonce":33684}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6804","nonce":33684},{"type":"v1","addr":"172.21.15.192:6805","nonce":33684}]},"public_addr":"172.21.15.192:6801/33684","cluster_addr":"172.21.15.192:6803/33684","heartbeat_back_addr":"172.21.15.192:6807/33684","heartbeat_front_addr":"172.21.15.192:6805/33684","state":["exists","up"]},{"osd":6,"uuid":"7854d1ae-7060-4780-8f97-73cea5e7a12f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6824","nonce":33686},{"type":"v1","addr":"172.21.15.192:6825","nonce":33686}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":33686},{"type":"v1","addr":"172.21.15.192:6827","nonce":33686}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":33686},{"type":"v1","addr":"172.21.15.192:6831","nonce":33686}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6828","nonce":33686},{"type":"v1","addr":"172.21.15.192:6829","nonce":33686}]},"public_addr":"172.21.15.192:6825/33686","cluster_addr":"172.21.15.192:6827/33686","heartbeat_back_addr":"172.21.15.192:6831/33686","heartbeat_front_addr":"172.21.15.192:6829/33686","state":["exists","up"]},{"osd":7,"uuid":"f1ebdfd5-b859-482b-9f97-c18c4a26f445","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6808","nonce":33685},{"type":"v1","addr":"172.21.15.192:6809","nonce":33685}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":33685},{"type":"v1","addr":"172.21.15.192:6811","nonce":33685}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":33685},{"type":"v1","addr":"172.21.15.192:6815","nonce":33685}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6812","nonce":33685},{"type":"v1","addr":"172.21.15.192:6813","nonce":33685}]},"public_addr":"172.21.15.192:6809/33685","cluster_addr":"172.21.15.192:6811/33685","heartbeat_back_addr":"172.21.15.192:6815/33685","heartbeat_front_addr":"172.21.15.192:6813/33685","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T03:29:18.002048+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T03:29:17.478576+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T03:29:18.034944+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T03:29:17.628513+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T03:29:18.047338+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T03:29:17.772349+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T03:29:17.883958+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T03:29:18.151838+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-06T03:29:57.562 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:57.557+0000 7fdfea1b9700 1 -- 172.21.15.110:0/4142435989 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fdfc4060550 msgr2=0x7fdfc4062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:57.562 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:57.557+0000 7fdfea1b9700 1 --2- 172.21.15.110:0/4142435989 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fdfc4060550 0x7fdfc4062a00 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fdfe406efc0 tx=0x7fdfd4017000 comp rx=0 tx=0).stop 2023-10-06T03:29:57.562 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:57.557+0000 7fdfea1b9700 1 -- 172.21.15.110:0/4142435989 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdfe40ce1b0 msgr2=0x7fdfe40c7070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:57.562 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:57.557+0000 7fdfea1b9700 1 --2- 172.21.15.110:0/4142435989 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdfe40ce1b0 0x7fdfe40c7070 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fdfcc00ec80 tx=0x7fdfcc0091a0 comp rx=0 tx=0).stop 2023-10-06T03:29:57.562 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:57.557+0000 7fdfea1b9700 1 -- 172.21.15.110:0/4142435989 shutdown_connections 2023-10-06T03:29:57.562 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:57.557+0000 7fdfea1b9700 1 --2- 172.21.15.110:0/4142435989 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fdfc4060550 0x7fdfc4062a00 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:57.562 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:57.557+0000 7fdfea1b9700 1 --2- 172.21.15.110:0/4142435989 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fdfe40cf930 0x7fdfe40b1560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:57.563 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:57.557+0000 7fdfea1b9700 1 --2- 172.21.15.110:0/4142435989 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdfe40ce1b0 0x7fdfe40c7070 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:57.563 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:57.557+0000 7fdfea1b9700 1 --2- 172.21.15.110:0/4142435989 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdfe40aefa0 0x7fdfe415c010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:57.563 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:57.557+0000 7fdfea1b9700 1 -- 172.21.15.110:0/4142435989 >> 172.21.15.110:0/4142435989 conn(0x7fdfe406dda0 msgr2=0x7fdfe4156ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:29:57.563 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:57.557+0000 7fdfea1b9700 1 -- 172.21.15.110:0/4142435989 shutdown_connections 2023-10-06T03:29:57.563 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:57.557+0000 7fdfea1b9700 1 -- 172.21.15.110:0/4142435989 wait complete. 2023-10-06T03:29:57.574 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-06T03:29:57.689 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:57.685+0000 7fa293fff700 1 Processor -- start 2023-10-06T03:29:57.689 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:57.685+0000 7fa293fff700 1 -- start start 2023-10-06T03:29:57.691 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:57.685+0000 7fa293fff700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa294158c30 0x7fa294159000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:57.691 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:57.685+0000 7fa293fff700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fa29406c180 0x7fa2940670a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:57.691 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:57.685+0000 7fa293fff700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2940c7710 con 0x7fa2940675e0 2023-10-06T03:29:57.692 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:57.685+0000 7fa293fff700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa29406cb20 con 0x7fa29406c180 2023-10-06T03:29:57.692 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:57.685+0000 7fa293fff700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa29406cf80 con 0x7fa294158c30 2023-10-06T03:29:57.692 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:57.685+0000 7fa2927fc700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fa29406c180 0x7fa2940670a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:57.692 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:57.685+0000 7fa2927fc700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fa29406c180 0x7fa2940670a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:47898/0 (socket says 172.21.15.110:47898) 2023-10-06T03:29:57.693 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:57.685+0000 7fa2927fc700 1 -- 172.21.15.110:0/1396875378 learned_addr learned my addr 172.21.15.110:0/1396875378 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:29:57.693 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:57.685+0000 7fa2937fe700 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7fa2940675e0 0x7fa2940c7ba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:40206/0 (socket says 172.21.15.110:40206) 2023-10-06T03:29:57.693 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:57.685+0000 7fa292ffd700 1 --2- 172.21.15.110:0/1396875378 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa294158c30 0x7fa294159000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:57.693 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:57.685+0000 7fa2927fc700 1 -- 172.21.15.110:0/1396875378 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa294158c30 msgr2=0x7fa294159000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:29:57.694 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:57.685+0000 7fa2927fc700 1 --2- 172.21.15.110:0/1396875378 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa294158c30 0x7fa294159000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:57.694 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:57.685+0000 7fa2927fc700 1 -- 172.21.15.110:0/1396875378 >> v1:172.21.15.110:6789/0 conn(0x7fa2940675e0 legacy=0x7fa2940c7ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:57.694 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:57.685+0000 7fa2927fc700 1 -- 172.21.15.110:0/1396875378 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa29406c5d0 con 0x7fa29406c180 2023-10-06T03:29:57.694 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:57.685+0000 7fa2927fc700 1 --2- 172.21.15.110:0/1396875378 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa29406c180 0x7fa2940670a0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fa28c009f50 tx=0x7fa28c009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be6e8a469cb91e0d server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:29:57.698 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:57.685+0000 7fa291ffb700 1 -- 172.21.15.110:0/1396875378 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa28c00e070 con 0x7fa29406c180 2023-10-06T03:29:57.698 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:57.685+0000 7fa291ffb700 1 -- 172.21.15.110:0/1396875378 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa28c00ba70 con 0x7fa29406c180 2023-10-06T03:29:57.698 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:57.685+0000 7fa291ffb700 1 -- 172.21.15.110:0/1396875378 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa28c0092e0 con 0x7fa29406c180 2023-10-06T03:29:57.699 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:57.685+0000 7fa293fff700 1 -- 172.21.15.110:0/1396875378 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa29406c180 msgr2=0x7fa2940670a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:57.699 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:57.685+0000 7fa293fff700 1 --2- 172.21.15.110:0/1396875378 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa29406c180 0x7fa2940670a0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fa28c009f50 tx=0x7fa28c009840 comp rx=0 tx=0).stop 2023-10-06T03:29:57.699 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:57.689+0000 7fa293fff700 1 -- 172.21.15.110:0/1396875378 shutdown_connections 2023-10-06T03:29:57.699 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:57.689+0000 7fa293fff700 1 --2- 172.21.15.110:0/1396875378 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa29406c180 0x7fa2940670a0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:57.699 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:57.689+0000 7fa293fff700 1 --2- 172.21.15.110:0/1396875378 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa294158c30 0x7fa294159000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:57.699 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:57.689+0000 7fa293fff700 1 -- 172.21.15.110:0/1396875378 >> 172.21.15.110:0/1396875378 conn(0x7fa2940c8d00 msgr2=0x7fa2940c9100 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:29:57.699 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:57.689+0000 7fa293fff700 1 -- 172.21.15.110:0/1396875378 shutdown_connections 2023-10-06T03:29:57.700 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:57.689+0000 7fa293fff700 1 -- 172.21.15.110:0/1396875378 wait complete. 2023-10-06T03:29:57.700 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:57.689+0000 7fa293fff700 1 Processor -- start 2023-10-06T03:29:57.700 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:57.689+0000 7fa293fff700 1 -- start start 2023-10-06T03:29:57.700 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:57.689+0000 7fa293fff700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa2940675e0 0x7fa294151180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:57.700 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:57.689+0000 7fa293fff700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa29406c180 0x7fa2941578f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:57.700 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:57.689+0000 7fa293fff700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa294158c30 0x7fa294155480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:57.700 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:57.689+0000 7fa293fff700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2941583f0 con 0x7fa2940675e0 2023-10-06T03:29:57.700 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:57.689+0000 7fa293fff700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa294169800 con 0x7fa294158c30 2023-10-06T03:29:57.700 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:57.689+0000 7fa293fff700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa294169ae0 con 0x7fa29406c180 2023-10-06T03:29:57.701 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:57.689+0000 7fa292ffd700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa2940675e0 0x7fa294151180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:57.701 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:57.689+0000 7fa292ffd700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa2940675e0 0x7fa294151180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:47908/0 (socket says 172.21.15.110:47908) 2023-10-06T03:29:57.701 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:57.689+0000 7fa292ffd700 1 -- 172.21.15.110:0/1607048538 learned_addr learned my addr 172.21.15.110:0/1607048538 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:29:57.701 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:57.689+0000 7fa2927fc700 1 --2- 172.21.15.110:0/1607048538 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa29406c180 0x7fa2941578f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:57.701 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:57.689+0000 7fa2937fe700 1 --2- 172.21.15.110:0/1607048538 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa294158c30 0x7fa294155480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:57.701 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:57.689+0000 7fa292ffd700 1 -- 172.21.15.110:0/1607048538 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa29406c180 msgr2=0x7fa2941578f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:57.701 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:57.689+0000 7fa292ffd700 1 --2- 172.21.15.110:0/1607048538 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa29406c180 0x7fa2941578f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:57.701 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:57.689+0000 7fa292ffd700 1 -- 172.21.15.110:0/1607048538 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa294158c30 msgr2=0x7fa294155480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:57.701 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:57.689+0000 7fa292ffd700 1 --2- 172.21.15.110:0/1607048538 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa294158c30 0x7fa294155480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:57.701 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:57.689+0000 7fa292ffd700 1 -- 172.21.15.110:0/1607048538 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa28c009ae0 con 0x7fa2940675e0 2023-10-06T03:29:57.702 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:57.689+0000 7fa2937fe700 1 --2- 172.21.15.110:0/1607048538 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa294158c30 0x7fa294155480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-06T03:29:57.702 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:57.689+0000 7fa292ffd700 1 --2- 172.21.15.110:0/1607048538 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa2940675e0 0x7fa294151180 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fa27c00d840 tx=0x7fa27c008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:29:57.702 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:57.689+0000 7fa28affd700 1 -- 172.21.15.110:0/1607048538 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa27c014620 con 0x7fa2940675e0 2023-10-06T03:29:57.702 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:57.689+0000 7fa28affd700 1 -- 172.21.15.110:0/1607048538 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa27c014780 con 0x7fa2940675e0 2023-10-06T03:29:57.702 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:57.689+0000 7fa293fff700 1 -- 172.21.15.110:0/1607048538 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa294169dc0 con 0x7fa2940675e0 2023-10-06T03:29:57.702 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:57.689+0000 7fa28affd700 1 -- 172.21.15.110:0/1607048538 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa27c014b40 con 0x7fa2940675e0 2023-10-06T03:29:57.702 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:57.689+0000 7fa293fff700 1 -- 172.21.15.110:0/1607048538 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2941516c0 con 0x7fa2940675e0 2023-10-06T03:29:57.702 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:57.693+0000 7fa293fff700 1 -- 172.21.15.110:0/1607048538 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa294070ff0 con 0x7fa2940675e0 2023-10-06T03:29:57.702 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:57.693+0000 7fa28affd700 1 -- 172.21.15.110:0/1607048538 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7fa27c01d400 con 0x7fa2940675e0 2023-10-06T03:29:57.702 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:57.693+0000 7fa28affd700 1 --2- 172.21.15.110:0/1607048538 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fa274060550 0x7fa274062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:57.703 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:57.693+0000 7fa28affd700 1 -- 172.21.15.110:0/1607048538 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6809+0+0 (secure 0 0 0) 0x7fa27c07d460 con 0x7fa2940675e0 2023-10-06T03:29:57.703 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:57.693+0000 7fa2927fc700 1 --2- 172.21.15.110:0/1607048538 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fa274060550 0x7fa274062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:57.703 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:57.693+0000 7fa2927fc700 1 --2- 172.21.15.110:0/1607048538 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fa274060550 0x7fa274062a00 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fa28c009880 tx=0x7fa28c018000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:29:57.703 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:57.701+0000 7fa28affd700 1 -- 172.21.15.110:0/1607048538 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fa27c04d660 con 0x7fa2940675e0 2023-10-06T03:29:57.899 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:57.897+0000 7fa293fff700 1 -- 172.21.15.110:0/1607048538 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa2940635e0 con 0x7fa2940675e0 2023-10-06T03:29:57.900 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:57.897+0000 7fa28affd700 1 -- 172.21.15.110:0/1607048538 <== mon.0 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4602 (secure 0 0 0) 0x7fa27c050c80 con 0x7fa2940675e0 2023-10-06T03:29:57.901 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-10-06T03:29:57.901 INFO:teuthology.orchestra.run.smithi110.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4279,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6833/1434505700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6832","nonce":1434505700},{"type":"v1","addr":"172.21.15.192:6833","nonce":1434505700}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4281,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/1168370789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6834","nonce":1168370789},{"type":"v1","addr":"172.21.15.110:6835","nonce":1168370789}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4288,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6835/1271810086","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":1271810086},{"type":"v1","addr":"172.21.15.192:6835","nonce":1271810086}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-06T03:29:33.894998+0000","modified":"2023-10-06T03:29:47.307075+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4293},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4293":{"gid":4293,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.110:6837/540320464","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6836","nonce":540320464},{"type":"v1","addr":"172.21.15.110:6837","nonce":540320464}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-06T03:29:57.904 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:57.897+0000 7fa293fff700 1 -- 172.21.15.110:0/1607048538 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fa274060550 msgr2=0x7fa274062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:57.904 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:57.897+0000 7fa293fff700 1 --2- 172.21.15.110:0/1607048538 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fa274060550 0x7fa274062a00 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fa28c009880 tx=0x7fa28c018000 comp rx=0 tx=0).stop 2023-10-06T03:29:57.904 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:57.897+0000 7fa293fff700 1 -- 172.21.15.110:0/1607048538 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa2940675e0 msgr2=0x7fa294151180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:57.904 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:57.897+0000 7fa293fff700 1 --2- 172.21.15.110:0/1607048538 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa2940675e0 0x7fa294151180 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fa27c00d840 tx=0x7fa27c008040 comp rx=0 tx=0).stop 2023-10-06T03:29:57.904 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:57.901+0000 7fa293fff700 1 -- 172.21.15.110:0/1607048538 shutdown_connections 2023-10-06T03:29:57.905 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:57.901+0000 7fa293fff700 1 --2- 172.21.15.110:0/1607048538 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fa274060550 0x7fa274062a00 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:57.905 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:57.901+0000 7fa293fff700 1 --2- 172.21.15.110:0/1607048538 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa294158c30 0x7fa294155480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:57.905 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:57.901+0000 7fa293fff700 1 --2- 172.21.15.110:0/1607048538 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa29406c180 0x7fa2941578f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:57.905 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:57.901+0000 7fa293fff700 1 --2- 172.21.15.110:0/1607048538 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa2940675e0 0x7fa294151180 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:57.905 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:57.901+0000 7fa293fff700 1 -- 172.21.15.110:0/1607048538 >> 172.21.15.110:0/1607048538 conn(0x7fa2940c8d00 msgr2=0x7fa29406a300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:29:57.905 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:57.901+0000 7fa293fff700 1 -- 172.21.15.110:0/1607048538 shutdown_connections 2023-10-06T03:29:57.905 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:57.901+0000 7fa293fff700 1 -- 172.21.15.110:0/1607048538 wait complete. 2023-10-06T03:29:57.906 INFO:teuthology.orchestra.run.smithi110.stderr:dumped fsmap epoch 8 2023-10-06T03:29:58.317 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 8, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-10-06T03:29:33.894998+0000', 'modified': '2023-10-06T03:29:47.307075+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4293}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4293': {'gid': 4293, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.110:6837/540320464', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.110:6836', 'nonce': 540320464}, {'type': 'v1', 'addr': '172.21.15.110:6837', 'nonce': 540320464}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138320759226367, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [4, 5], 'metadata_pool': 3, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2023-10-06T03:29:58.318 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2023-10-06T03:29:58.318 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2023-10-06T03:29:58.435 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.429+0000 7f2db782f700 1 Processor -- start 2023-10-06T03:29:58.435 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.429+0000 7f2db782f700 1 -- start start 2023-10-06T03:29:58.436 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.433+0000 7f2db782f700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2db0156d60 0x7f2db0157130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:58.436 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.433+0000 7f2db782f700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2db0158090 con 0x7f2db0069cf0 2023-10-06T03:29:58.436 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.433+0000 7f2db782f700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2db01577a0 con 0x7f2db00c6a90 2023-10-06T03:29:58.436 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.433+0000 7f2db782f700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2db006cc80 con 0x7f2db0156d60 2023-10-06T03:29:58.437 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.433+0000 7f2db5dcc700 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f2db0069cf0 0x7f2db015ac70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:40210/0 (socket says 172.21.15.110:40210) 2023-10-06T03:29:58.437 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.433+0000 7f2db5dcc700 1 -- 172.21.15.110:0/3488535805 learned_addr learned my addr 172.21.15.110:0/3488535805 (peer_addr_for_me v1:172.21.15.110:0/0) 2023-10-06T03:29:58.438 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.433+0000 7f2db55cb700 1 --2- 172.21.15.110:0/3488535805 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2db0156d60 0x7f2db0157130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:58.438 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.433+0000 7f2da3fff700 1 -- 172.21.15.110:0/3488535805 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3034758753 0 0) 0x7f2db0158090 con 0x7f2db0069cf0 2023-10-06T03:29:58.438 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.433+0000 7f2da3fff700 1 -- 172.21.15.110:0/3488535805 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2d9c003570 con 0x7f2db0069cf0 2023-10-06T03:29:58.438 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.433+0000 7f2da3fff700 1 -- 172.21.15.110:0/3488535805 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3765205873 0 0) 0x7f2db01577a0 con 0x7f2db00c6a90 2023-10-06T03:29:58.438 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.433+0000 7f2da3fff700 1 -- 172.21.15.110:0/3488535805 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2db0158090 con 0x7f2db00c6a90 2023-10-06T03:29:58.438 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.433+0000 7f2db55cb700 1 -- 172.21.15.110:0/3488535805 >> v1:172.21.15.192:6789/0 conn(0x7f2db00c6a90 legacy=0x7f2db00695e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:58.438 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.433+0000 7f2db55cb700 1 -- 172.21.15.110:0/3488535805 >> v1:172.21.15.110:6789/0 conn(0x7f2db0069cf0 legacy=0x7f2db015ac70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:58.438 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.433+0000 7f2db55cb700 1 -- 172.21.15.110:0/3488535805 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2db0157a40 con 0x7f2db0156d60 2023-10-06T03:29:58.438 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.433+0000 7f2db55cb700 1 --2- 172.21.15.110:0/3488535805 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2db0156d60 0x7f2db0157130 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f2dac009910 tx=0x7f2dac0091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=62abe9bffe507ef5 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:29:58.439 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.433+0000 7f2da3fff700 1 -- 172.21.15.110:0/3488535805 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2dac011630 con 0x7f2db0156d60 2023-10-06T03:29:58.439 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.433+0000 7f2da3fff700 1 -- 172.21.15.110:0/3488535805 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2dac011790 con 0x7f2db0156d60 2023-10-06T03:29:58.439 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.433+0000 7f2da3fff700 1 -- 172.21.15.110:0/3488535805 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2dac011b30 con 0x7f2db0156d60 2023-10-06T03:29:58.440 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.433+0000 7f2db782f700 1 -- 172.21.15.110:0/3488535805 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2db0156d60 msgr2=0x7f2db0157130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:58.440 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.433+0000 7f2db782f700 1 --2- 172.21.15.110:0/3488535805 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2db0156d60 0x7f2db0157130 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f2dac009910 tx=0x7f2dac0091a0 comp rx=0 tx=0).stop 2023-10-06T03:29:58.440 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.433+0000 7f2db782f700 1 -- 172.21.15.110:0/3488535805 shutdown_connections 2023-10-06T03:29:58.440 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.433+0000 7f2db782f700 1 --2- 172.21.15.110:0/3488535805 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2db0156d60 0x7f2db0157130 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:58.440 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.433+0000 7f2db782f700 1 -- 172.21.15.110:0/3488535805 >> 172.21.15.110:0/3488535805 conn(0x7f2db00cc710 msgr2=0x7f2db00c8670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:29:58.440 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.433+0000 7f2db782f700 1 -- 172.21.15.110:0/3488535805 shutdown_connections 2023-10-06T03:29:58.440 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.433+0000 7f2db782f700 1 -- 172.21.15.110:0/3488535805 wait complete. 2023-10-06T03:29:58.441 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.437+0000 7f2db782f700 1 Processor -- start 2023-10-06T03:29:58.441 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.437+0000 7f2db782f700 1 -- start start 2023-10-06T03:29:58.443 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.437+0000 7f2db782f700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2db0069cf0 0x7f2db01560b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:58.443 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.437+0000 7f2db782f700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2db00c6a90 0x7f2db00aa8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:58.443 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.437+0000 7f2db782f700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2db0156d60 0x7f2db00aae00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:58.443 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.437+0000 7f2db782f700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2db00c9730 con 0x7f2db0069cf0 2023-10-06T03:29:58.443 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.437+0000 7f2db782f700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2db00ca270 con 0x7f2db00c6a90 2023-10-06T03:29:58.443 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.437+0000 7f2db782f700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2db00c9940 con 0x7f2db0156d60 2023-10-06T03:29:58.443 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.437+0000 7f2db55cb700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2db0069cf0 0x7f2db01560b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:58.443 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.437+0000 7f2db5dcc700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2db0156d60 0x7f2db00aae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:58.443 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.437+0000 7f2db55cb700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2db0069cf0 0x7f2db01560b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:47916/0 (socket says 172.21.15.110:47916) 2023-10-06T03:29:58.444 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.437+0000 7f2db5dcc700 1 --2- 172.21.15.110:0/811219238 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2db0156d60 0x7f2db00aae00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3301/0 says I am v2:172.21.15.110:35008/0 (socket says 172.21.15.110:35008) 2023-10-06T03:29:58.444 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.437+0000 7f2db55cb700 1 -- 172.21.15.110:0/811219238 learned_addr learned my addr 172.21.15.110:0/811219238 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:29:58.444 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.437+0000 7f2db4dca700 1 --2- 172.21.15.110:0/811219238 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2db00c6a90 0x7f2db00aa8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:58.444 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.437+0000 7f2db55cb700 1 -- 172.21.15.110:0/811219238 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2db0156d60 msgr2=0x7f2db00aae00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:58.444 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.437+0000 7f2db55cb700 1 --2- 172.21.15.110:0/811219238 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2db0156d60 0x7f2db00aae00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:58.444 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.437+0000 7f2db55cb700 1 -- 172.21.15.110:0/811219238 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2db00c6a90 msgr2=0x7f2db00aa8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:58.444 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.437+0000 7f2db55cb700 1 --2- 172.21.15.110:0/811219238 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2db00c6a90 0x7f2db00aa8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:58.444 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.437+0000 7f2db55cb700 1 -- 172.21.15.110:0/811219238 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2dac009420 con 0x7f2db0069cf0 2023-10-06T03:29:58.444 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.437+0000 7f2db55cb700 1 --2- 172.21.15.110:0/811219238 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2db0069cf0 0x7f2db01560b0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f2dac009f00 tx=0x7f2dac017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:29:58.444 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.437+0000 7f2da17fa700 1 -- 172.21.15.110:0/811219238 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2dac01f070 con 0x7f2db0069cf0 2023-10-06T03:29:58.444 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.437+0000 7f2da17fa700 1 -- 172.21.15.110:0/811219238 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2dac002f70 con 0x7f2db0069cf0 2023-10-06T03:29:58.444 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.437+0000 7f2db782f700 1 -- 172.21.15.110:0/811219238 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2db00c9c20 con 0x7f2db0069cf0 2023-10-06T03:29:58.445 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.437+0000 7f2da17fa700 1 -- 172.21.15.110:0/811219238 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2dac006af0 con 0x7f2db0069cf0 2023-10-06T03:29:58.445 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.437+0000 7f2db782f700 1 -- 172.21.15.110:0/811219238 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2db01565f0 con 0x7f2db0069cf0 2023-10-06T03:29:58.447 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.441+0000 7f2da17fa700 1 -- 172.21.15.110:0/811219238 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7f2dac023070 con 0x7f2db0069cf0 2023-10-06T03:29:58.447 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.445+0000 7f2da17fa700 1 --2- 172.21.15.110:0/811219238 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f2d94060550 0x7f2d94062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:58.447 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.445+0000 7f2da17fa700 1 -- 172.21.15.110:0/811219238 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6809+0+0 (secure 0 0 0) 0x7f2dac07ddc0 con 0x7f2db0069cf0 2023-10-06T03:29:58.448 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.445+0000 7f2db4dca700 1 --2- 172.21.15.110:0/811219238 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f2d94060550 0x7f2d94062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:58.448 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.445+0000 7f2db4dca700 1 --2- 172.21.15.110:0/811219238 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f2d94060550 0x7f2d94062a00 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f2db00c2250 tx=0x7f2da4007f90 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:29:58.449 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.445+0000 7f2db9ad8740 1 Processor -- start 2023-10-06T03:29:58.449 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.445+0000 7f2db9ad8740 1 -- start start 2023-10-06T03:29:58.451 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.445+0000 7f2db9ad8740 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x1c11ae0 0x1e8ee50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:58.452 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.445+0000 7f2db9ad8740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x1c11710 0x1e8f390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:58.452 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.445+0000 7f2db9ad8740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x1c5c790 0x1e8f8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:58.452 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.445+0000 7f2db9ad8740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x1dbcf30 con 0x1c5c790 2023-10-06T03:29:58.452 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.445+0000 7f2db9ad8740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x1db8ed0 con 0x1c11710 2023-10-06T03:29:58.452 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.445+0000 7f2db9ad8740 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x1d7f0b0 con 0x1c11ae0 2023-10-06T03:29:58.452 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.445+0000 7f2db5dcc700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x1c5c790 0x1e8f8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:58.452 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.445+0000 7f2db5dcc700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x1c5c790 0x1e8f8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:47924/0 (socket says 172.21.15.110:47924) 2023-10-06T03:29:58.452 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.445+0000 7f2db5dcc700 1 -- 172.21.15.110:0/4133061640 learned_addr learned my addr 172.21.15.110:0/4133061640 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:29:58.452 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.445+0000 7f2db4dca700 1 --2- 172.21.15.110:0/4133061640 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x1c11710 0x1e8f390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:58.452 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.445+0000 7f2db55cb700 1 --2- 172.21.15.110:0/4133061640 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x1c11ae0 0x1e8ee50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:58.452 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.445+0000 7f2db5dcc700 1 -- 172.21.15.110:0/4133061640 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x1c11ae0 msgr2=0x1e8ee50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:29:58.452 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.445+0000 7f2db5dcc700 1 --2- 172.21.15.110:0/4133061640 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x1c11ae0 0x1e8ee50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:58.453 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.445+0000 7f2db5dcc700 1 -- 172.21.15.110:0/4133061640 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x1c11710 msgr2=0x1e8f390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:29:58.453 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.445+0000 7f2db5dcc700 1 --2- 172.21.15.110:0/4133061640 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x1c11710 0x1e8f390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:58.453 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.445+0000 7f2db5dcc700 1 -- 172.21.15.110:0/4133061640 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x1c5d9e0 con 0x1c5c790 2023-10-06T03:29:58.453 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.449+0000 7f2db4dca700 1 --2- 172.21.15.110:0/4133061640 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x1c11710 0x1e8f390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-06T03:29:58.453 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.449+0000 7f2db5dcc700 1 --2- 172.21.15.110:0/4133061640 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x1c5c790 0x1e8f8d0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f2db006a0f0 tx=0x7f2da8009040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca10c62dfe46330e server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:29:58.453 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.449+0000 7f2d9a7fc700 1 -- 172.21.15.110:0/4133061640 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2da8005b10 con 0x1c5c790 2023-10-06T03:29:58.453 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.449+0000 7f2d9a7fc700 1 -- 172.21.15.110:0/4133061640 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2da8005c90 con 0x1c5c790 2023-10-06T03:29:58.453 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.449+0000 7f2d9a7fc700 1 -- 172.21.15.110:0/4133061640 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2da8007320 con 0x1c5c790 2023-10-06T03:29:58.454 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.449+0000 7f2db9ad8740 1 -- 172.21.15.110:0/4133061640 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x1c5c790 msgr2=0x1e8f8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:58.454 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.449+0000 7f2db9ad8740 1 --2- 172.21.15.110:0/4133061640 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x1c5c790 0x1e8f8d0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f2db006a0f0 tx=0x7f2da8009040 comp rx=0 tx=0).stop 2023-10-06T03:29:58.454 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.449+0000 7f2db9ad8740 1 -- 172.21.15.110:0/4133061640 shutdown_connections 2023-10-06T03:29:58.454 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.449+0000 7f2db9ad8740 1 --2- 172.21.15.110:0/4133061640 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x1c5c790 0x1e8f8d0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:58.455 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.449+0000 7f2db9ad8740 1 --2- 172.21.15.110:0/4133061640 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x1c11710 0x1e8f390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:58.455 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.449+0000 7f2db9ad8740 1 --2- 172.21.15.110:0/4133061640 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x1c11ae0 0x1e8ee50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:58.455 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.449+0000 7f2db9ad8740 1 -- 172.21.15.110:0/4133061640 >> 172.21.15.110:0/4133061640 conn(0x1c11eb0 msgr2=0x1e4b880 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:29:58.455 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.449+0000 7f2db9ad8740 1 -- 172.21.15.110:0/4133061640 shutdown_connections 2023-10-06T03:29:58.455 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.449+0000 7f2db9ad8740 1 -- 172.21.15.110:0/4133061640 wait complete. 2023-10-06T03:29:58.455 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.449+0000 7f2db9ad8740 1 -- start start 2023-10-06T03:29:58.455 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.449+0000 7f2db9ad8740 1 Processor -- start 2023-10-06T03:29:58.461 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.453+0000 7f2db9ad8740 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x1c11710 0x1e4b740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:58.461 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.453+0000 7f2db9ad8740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x1c11ae0 0x1e74c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:58.461 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.453+0000 7f2db9ad8740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x1c5c790 0x1e75140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:58.461 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.453+0000 7f2db9ad8740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x1d7ed50 con 0x1c5c790 2023-10-06T03:29:58.461 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.453+0000 7f2db9ad8740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x1d57970 con 0x1c11ae0 2023-10-06T03:29:58.461 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.453+0000 7f2db9ad8740 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x1db86e0 con 0x1c11710 2023-10-06T03:29:58.461 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.453+0000 7f2db9ad8740 10 client.0 _ll_register_callbacks cb 0x1e02b00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-06T03:29:58.461 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.453+0000 7f2db5dcc700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x1c5c790 0x1e75140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:58.461 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.453+0000 7f2db5dcc700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x1c5c790 0x1e75140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:47930/0 (socket says 172.21.15.110:47930) 2023-10-06T03:29:58.461 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.453+0000 7f2db5dcc700 1 -- 172.21.15.110:0/1884022768 learned_addr learned my addr 172.21.15.110:0/1884022768 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:29:58.462 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.453+0000 7f2db4dca700 1 --2- 172.21.15.110:0/1884022768 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x1c11ae0 0x1e74c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:58.462 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.453+0000 7f2db55cb700 1 --2- 172.21.15.110:0/1884022768 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x1c11710 0x1e4b740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:58.462 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.453+0000 7f2db5dcc700 1 -- 172.21.15.110:0/1884022768 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x1c11710 msgr2=0x1e4b740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:29:58.462 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.453+0000 7f2db5dcc700 1 --2- 172.21.15.110:0/1884022768 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x1c11710 0x1e4b740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:58.462 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.453+0000 7f2db5dcc700 1 -- 172.21.15.110:0/1884022768 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x1c11ae0 msgr2=0x1e74c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:29:58.462 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.453+0000 7f2db5dcc700 1 --2- 172.21.15.110:0/1884022768 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x1c11ae0 0x1e74c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:58.462 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.453+0000 7f2db5dcc700 1 -- 172.21.15.110:0/1884022768 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d9c003570 con 0x1c5c790 2023-10-06T03:29:58.462 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.453+0000 7f2db55cb700 1 --2- 172.21.15.110:0/1884022768 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x1c11710 0x1e4b740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-06T03:29:58.462 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.453+0000 7f2db5dcc700 1 --2- 172.21.15.110:0/1884022768 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x1c5c790 0x1e75140 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x1d5b000 tx=0x7f2da8018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=34aeede23e4c6450 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:29:58.462 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.453+0000 7f2d8f7fe700 10 client.0 ms_handle_connect on v2:172.21.15.110:3300/0 2023-10-06T03:29:58.463 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.453+0000 7f2d8f7fe700 1 -- 172.21.15.110:0/1884022768 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2da80013f0 con 0x1c5c790 2023-10-06T03:29:58.463 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.453+0000 7f2d8f7fe700 1 -- 172.21.15.110:0/1884022768 <== mon.0 v2:172.21.15.110:3300/0 2 ==== osd_map(28..28 src has 1..28) v4 ==== 6809+0+0 (secure 0 0 0) 0x7f2da801d690 con 0x1c5c790 2023-10-06T03:29:58.463 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.453+0000 7f2db9ad8740 1 -- 172.21.15.110:0/1884022768 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x1d57cc0 con 0x1c5c790 2023-10-06T03:29:58.463 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.453+0000 7f2d8f7fe700 1 -- 172.21.15.110:0/1884022768 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_get_version_reply(handle=1 version=8) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2da801da80 con 0x1c5c790 2023-10-06T03:29:58.463 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.453+0000 7f2db9ad8740 10 client.4803 fetch_fsmap learned FSMap version 8 2023-10-06T03:29:58.463 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.453+0000 7f2db9ad8740 1 -- 172.21.15.110:0/1884022768 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,fsmap=8,monmap=2+}) v3 -- 0x1d87370 con 0x1c5c790 2023-10-06T03:29:58.463 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.453+0000 7f2d8f7fe700 1 -- 172.21.15.110:0/1884022768 <== mon.0 v2:172.21.15.110:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x1d57cc0 con 0x1c5c790 2023-10-06T03:29:58.463 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.453+0000 7f2d8f7fe700 1 -- 172.21.15.110:0/1884022768 <== mon.0 v2:172.21.15.110:3300/0 5 ==== fsmap(e 8) v1 ==== 3108+0+0 (secure 0 0 0) 0x7f2da8012070 con 0x1c5c790 2023-10-06T03:29:58.463 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.453+0000 7f2db9ad8740 10 client.4803 fetch_fsmap finished waiting for FSMap version 8 2023-10-06T03:29:58.463 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.453+0000 7f2db9ad8740 10 client.4803 resolve_mds: resolved name 'a' to daemon mds.a 2023-10-06T03:29:58.463 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.453+0000 7f2db9ad8740 20 client.4803 populate_metadata read hostname 'smithi110' 2023-10-06T03:29:58.464 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.453+0000 7f2db9ad8740 1 --2- 172.21.15.110:0/1884022768 >> [v2:172.21.15.110:6836/540320464,v1:172.21.15.110:6837/540320464] conn(0x1e9b6b0 0x1e9da90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:58.464 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.453+0000 7f2db9ad8740 4 client.4803 mds_command: new command op to 4293 tid=0[{"prefix": "get_command_descriptions"}] 2023-10-06T03:29:58.464 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.453+0000 7f2db9ad8740 1 -- 172.21.15.110:0/1884022768 --> [v2:172.21.15.110:6836/540320464,v1:172.21.15.110:6837/540320464] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x1d5b3e0 con 0x1e9b6b0 2023-10-06T03:29:58.464 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.453+0000 7f2db55cb700 1 --2- 172.21.15.110:0/1884022768 >> [v2:172.21.15.110:6836/540320464,v1:172.21.15.110:6837/540320464] conn(0x1e9b6b0 0x1e9da90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:58.464 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.453+0000 7f2d8f7fe700 10 client.4803 ms_handle_connect on v2:172.21.15.110:6836/540320464 2023-10-06T03:29:58.464 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.453+0000 7f2db55cb700 1 --2- 172.21.15.110:0/1884022768 >> [v2:172.21.15.110:6836/540320464,v1:172.21.15.110:6837/540320464] conn(0x1e9b6b0 0x1e9da90 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=26c255cd26155f50 server_cookie=5063a4973a0831b4 in_seq=0 out_seq=0 2023-10-06T03:29:58.464 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.457+0000 7f2d8f7fe700 1 -- 172.21.15.110:0/1884022768 <== mds.0 v2:172.21.15.110:6836/540320464 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+16954 (crc 0 0 0) 0x1d5b3e0 con 0x1e9b6b0 2023-10-06T03:29:58.464 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.457+0000 7f2d8f7fe700 10 client.4803 handle_command_reply: tid=0 2023-10-06T03:29:58.464 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.457+0000 7f2db9ad8740 1 client.4803 shutdown 2023-10-06T03:29:58.464 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.457+0000 7f2db9ad8740 1 -- 172.21.15.110:0/1884022768 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x1c5c790 msgr2=0x1e75140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:58.464 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.457+0000 7f2db9ad8740 1 --2- 172.21.15.110:0/1884022768 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x1c5c790 0x1e75140 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x1d5b000 tx=0x7f2da8018000 comp rx=0 tx=0).stop 2023-10-06T03:29:58.465 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.457+0000 7f2db9ad8740 1 -- 172.21.15.110:0/1884022768 shutdown_connections 2023-10-06T03:29:58.465 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.457+0000 7f2db9ad8740 1 --2- 172.21.15.110:0/1884022768 >> [v2:172.21.15.110:6836/540320464,v1:172.21.15.110:6837/540320464] conn(0x1e9b6b0 0x1e9da90 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:58.465 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.457+0000 7f2db9ad8740 1 --2- 172.21.15.110:0/1884022768 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x1c5c790 0x1e75140 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:58.465 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.457+0000 7f2d8f7fe700 0 client.4803 ms_handle_reset on v2:172.21.15.110:6836/540320464 2023-10-06T03:29:58.465 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.457+0000 7f2db9ad8740 1 --2- 172.21.15.110:0/1884022768 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x1c11ae0 0x1e74c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:58.465 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.457+0000 7f2db9ad8740 1 --2- 172.21.15.110:0/1884022768 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x1c11710 0x1e4b740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:58.465 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.457+0000 7f2db9ad8740 1 -- 172.21.15.110:0/1884022768 >> 172.21.15.110:0/1884022768 conn(0x1c11eb0 msgr2=0x1e43ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:29:58.465 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.457+0000 7f2db9ad8740 1 -- 172.21.15.110:0/1884022768 shutdown_connections 2023-10-06T03:29:58.465 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.457+0000 7f2db9ad8740 1 -- 172.21.15.110:0/1884022768 wait complete. 2023-10-06T03:29:58.465 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.457+0000 7f2db9ad8740 20 client.4803 trim_cache size 0 max 16384 2023-10-06T03:29:58.480 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.477+0000 7f2db9ad8740 1 Processor -- start 2023-10-06T03:29:58.480 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.477+0000 7f2db9ad8740 1 -- start start 2023-10-06T03:29:58.482 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.477+0000 7f2db9ad8740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x1c11710 0x1eecf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:58.482 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.477+0000 7f2db9ad8740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x1c11ae0 0x1eed460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:58.482 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.477+0000 7f2db9ad8740 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x1c5c790 0x1eed9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:58.482 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.477+0000 7f2db9ad8740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x1dbdef0 con 0x1c11ae0 2023-10-06T03:29:58.483 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.477+0000 7f2db9ad8740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x1dbc9d0 con 0x1c11710 2023-10-06T03:29:58.483 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.477+0000 7f2db9ad8740 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x1d59b80 con 0x1c5c790 2023-10-06T03:29:58.483 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.477+0000 7f2db4dca700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x1c11ae0 0x1eed460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:58.483 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.477+0000 7f2db4dca700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x1c11ae0 0x1eed460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:47942/0 (socket says 172.21.15.110:47942) 2023-10-06T03:29:58.483 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.477+0000 7f2db4dca700 1 -- 172.21.15.110:0/174730262 learned_addr learned my addr 172.21.15.110:0/174730262 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:29:58.483 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.477+0000 7f2db5dcc700 1 --2- 172.21.15.110:0/174730262 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x1c5c790 0x1eed9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:58.483 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.477+0000 7f2db55cb700 1 --2- 172.21.15.110:0/174730262 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x1c11710 0x1eecf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:58.483 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.477+0000 7f2db4dca700 1 -- 172.21.15.110:0/174730262 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x1c5c790 msgr2=0x1eed9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:29:58.483 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.477+0000 7f2db4dca700 1 --2- 172.21.15.110:0/174730262 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x1c5c790 0x1eed9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:58.483 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.477+0000 7f2db4dca700 1 -- 172.21.15.110:0/174730262 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x1c11710 msgr2=0x1eecf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:29:58.483 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.477+0000 7f2db4dca700 1 --2- 172.21.15.110:0/174730262 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x1c11710 0x1eecf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:58.484 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.477+0000 7f2db4dca700 1 -- 172.21.15.110:0/174730262 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2db00c0d90 con 0x1c11ae0 2023-10-06T03:29:58.484 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.477+0000 7f2db5dcc700 1 --2- 172.21.15.110:0/174730262 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x1c5c790 0x1eed9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-06T03:29:58.484 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.477+0000 7f2db4dca700 1 --2- 172.21.15.110:0/174730262 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x1c11ae0 0x1eed460 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f2da4002da0 tx=0x7f2da4015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e582c71ac3b3dab7 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:29:58.484 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.477+0000 7f2d9affd700 1 -- 172.21.15.110:0/174730262 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2da4002e80 con 0x1c11ae0 2023-10-06T03:29:58.484 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.477+0000 7f2d9affd700 1 -- 172.21.15.110:0/174730262 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2da400fe40 con 0x1c11ae0 2023-10-06T03:29:58.484 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.477+0000 7f2d9affd700 1 -- 172.21.15.110:0/174730262 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2da4003470 con 0x1c11ae0 2023-10-06T03:29:58.485 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.481+0000 7f2db9ad8740 1 -- 172.21.15.110:0/174730262 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x1c11ae0 msgr2=0x1eed460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:58.485 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.481+0000 7f2db9ad8740 1 --2- 172.21.15.110:0/174730262 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x1c11ae0 0x1eed460 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f2da4002da0 tx=0x7f2da4015000 comp rx=0 tx=0).stop 2023-10-06T03:29:58.485 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.481+0000 7f2db9ad8740 1 -- 172.21.15.110:0/174730262 shutdown_connections 2023-10-06T03:29:58.485 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.481+0000 7f2db9ad8740 1 --2- 172.21.15.110:0/174730262 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x1c5c790 0x1eed9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:58.485 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.481+0000 7f2db9ad8740 1 --2- 172.21.15.110:0/174730262 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x1c11ae0 0x1eed460 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:58.485 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.481+0000 7f2db9ad8740 1 --2- 172.21.15.110:0/174730262 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x1c11710 0x1eecf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:58.485 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.481+0000 7f2db9ad8740 1 -- 172.21.15.110:0/174730262 >> 172.21.15.110:0/174730262 conn(0x1c11eb0 msgr2=0x1e73c30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:29:58.485 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.481+0000 7f2db9ad8740 1 -- 172.21.15.110:0/174730262 shutdown_connections 2023-10-06T03:29:58.486 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.481+0000 7f2db9ad8740 1 -- 172.21.15.110:0/174730262 wait complete. 2023-10-06T03:29:58.486 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.481+0000 7f2db9ad8740 1 -- start start 2023-10-06T03:29:58.486 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.481+0000 7f2db9ad8740 1 Processor -- start 2023-10-06T03:29:58.488 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.485+0000 7f2db9ad8740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x1c11710 0x1e6fb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:58.488 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.485+0000 7f2db9ad8740 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x1c11ae0 0x1e700d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:58.488 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.485+0000 7f2db9ad8740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x1c5c790 0x1e70610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:58.489 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.485+0000 7f2db9ad8740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x1e26bb0 con 0x1c11710 2023-10-06T03:29:58.489 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.485+0000 7f2db9ad8740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x1dbd490 con 0x1c5c790 2023-10-06T03:29:58.489 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.485+0000 7f2db9ad8740 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x1d33410 con 0x1c11ae0 2023-10-06T03:29:58.489 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.485+0000 7f2db9ad8740 10 client.0 _ll_register_callbacks cb 0x1cd50e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-06T03:29:58.489 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.485+0000 7f2db55cb700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x1c11710 0x1e6fb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:58.489 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.485+0000 7f2db55cb700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x1c11710 0x1e6fb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:47944/0 (socket says 172.21.15.110:47944) 2023-10-06T03:29:58.489 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.485+0000 7f2db55cb700 1 -- 172.21.15.110:0/247571956 learned_addr learned my addr 172.21.15.110:0/247571956 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:29:58.489 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.485+0000 7f2db5dcc700 1 --2- 172.21.15.110:0/247571956 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x1c5c790 0x1e70610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:58.489 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.485+0000 7f2db4dca700 1 --2- 172.21.15.110:0/247571956 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x1c11ae0 0x1e700d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:58.489 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.485+0000 7f2db55cb700 1 -- 172.21.15.110:0/247571956 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x1c11ae0 msgr2=0x1e700d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:29:58.490 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.485+0000 7f2db55cb700 1 --2- 172.21.15.110:0/247571956 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x1c11ae0 0x1e700d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:58.490 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.485+0000 7f2db55cb700 1 -- 172.21.15.110:0/247571956 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x1c5c790 msgr2=0x1e70610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:29:58.490 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.485+0000 7f2db55cb700 1 --2- 172.21.15.110:0/247571956 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x1c5c790 0x1e70610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:58.490 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.485+0000 7f2db55cb700 1 -- 172.21.15.110:0/247571956 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2db0158090 con 0x1c11710 2023-10-06T03:29:58.493 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.485+0000 7f2db55cb700 1 --2- 172.21.15.110:0/247571956 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x1c11710 0x1e6fb90 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f2dac051880 tx=0x7f2dac054040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=537b7bcc520ccad8 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:29:58.493 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.485+0000 7f2d8f7fe700 10 client.0 ms_handle_connect on v2:172.21.15.110:3300/0 2023-10-06T03:29:58.493 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.485+0000 7f2d8f7fe700 1 -- 172.21.15.110:0/247571956 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2dac058b60 con 0x1c11710 2023-10-06T03:29:58.493 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.485+0000 7f2d8f7fe700 1 -- 172.21.15.110:0/247571956 <== mon.0 v2:172.21.15.110:3300/0 2 ==== osd_map(28..28 src has 1..28) v4 ==== 6809+0+0 (secure 0 0 0) 0x7f2dac07bf00 con 0x1c11710 2023-10-06T03:29:58.493 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.485+0000 7f2db9ad8740 1 -- 172.21.15.110:0/247571956 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f2da8005c90 con 0x1c11710 2023-10-06T03:29:58.493 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.485+0000 7f2d8f7fe700 1 -- 172.21.15.110:0/247571956 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_get_version_reply(handle=1 version=8) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2dac051070 con 0x1c11710 2023-10-06T03:29:58.494 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.485+0000 7f2db9ad8740 10 client.4809 fetch_fsmap learned FSMap version 8 2023-10-06T03:29:58.494 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.485+0000 7f2db9ad8740 1 -- 172.21.15.110:0/247571956 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,fsmap=8,monmap=2+}) v3 -- 0x1d6dd80 con 0x1c11710 2023-10-06T03:29:58.494 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.485+0000 7f2d8f7fe700 1 -- 172.21.15.110:0/247571956 <== mon.0 v2:172.21.15.110:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2dac06f030 con 0x1c11710 2023-10-06T03:29:58.494 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.485+0000 7f2d8f7fe700 1 -- 172.21.15.110:0/247571956 <== mon.0 v2:172.21.15.110:3300/0 5 ==== fsmap(e 8) v1 ==== 3108+0+0 (secure 0 0 0) 0x7f2dac052bd0 con 0x1c11710 2023-10-06T03:29:58.494 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.485+0000 7f2db9ad8740 10 client.4809 fetch_fsmap finished waiting for FSMap version 8 2023-10-06T03:29:58.494 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.485+0000 7f2db9ad8740 10 client.4809 resolve_mds: resolved name 'a' to daemon mds.a 2023-10-06T03:29:58.494 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.485+0000 7f2db9ad8740 20 client.4809 populate_metadata read hostname 'smithi110' 2023-10-06T03:29:58.494 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.485+0000 7f2db9ad8740 1 --2- 172.21.15.110:0/247571956 >> [v2:172.21.15.110:6836/540320464,v1:172.21.15.110:6837/540320464] conn(0x1ef71a0 0x1ef9580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:58.494 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.485+0000 7f2db9ad8740 4 client.4809 mds_command: new command op to 4293 tid=0[{"prefix": "status"}] 2023-10-06T03:29:58.494 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.485+0000 7f2db9ad8740 1 -- 172.21.15.110:0/247571956 --> [v2:172.21.15.110:6836/540320464,v1:172.21.15.110:6837/540320464] -- command(tid 0: {"prefix": "status"}) v1 -- 0x1d8d410 con 0x1ef71a0 2023-10-06T03:29:58.494 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.485+0000 7f2db4dca700 1 --2- 172.21.15.110:0/247571956 >> [v2:172.21.15.110:6836/540320464,v1:172.21.15.110:6837/540320464] conn(0x1ef71a0 0x1ef9580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:58.495 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.485+0000 7f2d8f7fe700 10 client.4809 ms_handle_connect on v2:172.21.15.110:6836/540320464 2023-10-06T03:29:58.495 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.485+0000 7f2db4dca700 1 --2- 172.21.15.110:0/247571956 >> [v2:172.21.15.110:6836/540320464,v1:172.21.15.110:6837/540320464] conn(0x1ef71a0 0x1ef9580 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=88bdb6f59b78db45 server_cookie=1d601d87ace58119 in_seq=0 out_seq=0 2023-10-06T03:29:58.495 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.489+0000 7f2d8f7fe700 1 -- 172.21.15.110:0/247571956 <== mds.0 v2:172.21.15.110:6836/540320464 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+324 (crc 0 0 0) 0x1d8d410 con 0x1ef71a0 2023-10-06T03:29:58.495 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.489+0000 7f2d8f7fe700 10 client.4809 handle_command_reply: tid=0 2023-10-06T03:29:58.495 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.489+0000 7f2db9ad8740 1 client.4809 shutdown 2023-10-06T03:29:58.495 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.489+0000 7f2db9ad8740 1 -- 172.21.15.110:0/247571956 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x1c11710 msgr2=0x1e6fb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:58.495 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.489+0000 7f2db9ad8740 1 --2- 172.21.15.110:0/247571956 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x1c11710 0x1e6fb90 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f2dac051880 tx=0x7f2dac054040 comp rx=0 tx=0).stop 2023-10-06T03:29:58.495 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.489+0000 7f2db9ad8740 1 -- 172.21.15.110:0/247571956 shutdown_connections 2023-10-06T03:29:58.495 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.489+0000 7f2db9ad8740 1 --2- 172.21.15.110:0/247571956 >> [v2:172.21.15.110:6836/540320464,v1:172.21.15.110:6837/540320464] conn(0x1ef71a0 0x1ef9580 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:58.495 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.489+0000 7f2db9ad8740 1 --2- 172.21.15.110:0/247571956 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x1c5c790 0x1e70610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:58.495 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.489+0000 7f2d8f7fe700 0 client.4809 ms_handle_reset on v2:172.21.15.110:6836/540320464 2023-10-06T03:29:58.495 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.489+0000 7f2db9ad8740 1 --2- 172.21.15.110:0/247571956 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x1c11ae0 0x1e700d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:58.496 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.489+0000 7f2db9ad8740 1 --2- 172.21.15.110:0/247571956 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x1c11710 0x1e6fb90 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:58.496 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.489+0000 7f2db9ad8740 1 -- 172.21.15.110:0/247571956 >> 172.21.15.110:0/247571956 conn(0x1c11eb0 msgr2=0x1e73c30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:29:58.496 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.489+0000 7f2db9ad8740 1 -- 172.21.15.110:0/247571956 shutdown_connections 2023-10-06T03:29:58.496 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.489+0000 7f2db9ad8740 1 -- 172.21.15.110:0/247571956 wait complete. 2023-10-06T03:29:58.496 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.489+0000 7f2db9ad8740 20 client.4809 trim_cache size 0 max 16384 2023-10-06T03:29:58.496 INFO:teuthology.orchestra.run.smithi110.stdout:{ 2023-10-06T03:29:58.496 INFO:teuthology.orchestra.run.smithi110.stdout: "cluster_fsid": "93b8f6ac-4c92-4b45-b8e4-b59617024aa2", 2023-10-06T03:29:58.496 INFO:teuthology.orchestra.run.smithi110.stdout: "whoami": 0, 2023-10-06T03:29:58.496 INFO:teuthology.orchestra.run.smithi110.stdout: "id": 4293, 2023-10-06T03:29:58.497 INFO:teuthology.orchestra.run.smithi110.stdout: "want_state": "up:active", 2023-10-06T03:29:58.497 INFO:teuthology.orchestra.run.smithi110.stdout: "state": "up:active", 2023-10-06T03:29:58.497 INFO:teuthology.orchestra.run.smithi110.stdout: "fs_name": "cephfs", 2023-10-06T03:29:58.497 INFO:teuthology.orchestra.run.smithi110.stdout: "rank_uptime": 23.973306402999999, 2023-10-06T03:29:58.497 INFO:teuthology.orchestra.run.smithi110.stdout: "mdsmap_epoch": 8, 2023-10-06T03:29:58.497 INFO:teuthology.orchestra.run.smithi110.stdout: "osdmap_epoch": 28, 2023-10-06T03:29:58.497 INFO:teuthology.orchestra.run.smithi110.stdout: "osdmap_epoch_barrier": 28, 2023-10-06T03:29:58.497 INFO:teuthology.orchestra.run.smithi110.stdout: "uptime": 29.004894599 2023-10-06T03:29:58.497 INFO:teuthology.orchestra.run.smithi110.stdout:} 2023-10-06T03:29:58.498 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.489+0000 7f2db782f700 1 -- 172.21.15.110:0/811219238 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f2d94060550 msgr2=0x7f2d94062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:58.498 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.489+0000 7f2db782f700 1 --2- 172.21.15.110:0/811219238 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f2d94060550 0x7f2d94062a00 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f2db00c2250 tx=0x7f2da4007f90 comp rx=0 tx=0).stop 2023-10-06T03:29:58.498 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.489+0000 7f2db782f700 1 -- 172.21.15.110:0/811219238 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2db0069cf0 msgr2=0x7f2db01560b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:58.498 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.489+0000 7f2db782f700 1 --2- 172.21.15.110:0/811219238 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2db0069cf0 0x7f2db01560b0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f2dac009f00 tx=0x7f2dac017000 comp rx=0 tx=0).stop 2023-10-06T03:29:58.498 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.489+0000 7f2db782f700 1 -- 172.21.15.110:0/811219238 shutdown_connections 2023-10-06T03:29:58.499 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.489+0000 7f2db782f700 1 --2- 172.21.15.110:0/811219238 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f2d94060550 0x7f2d94062a00 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:58.499 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.489+0000 7f2db782f700 1 --2- 172.21.15.110:0/811219238 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2db0156d60 0x7f2db00aae00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:58.499 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.489+0000 7f2db782f700 1 --2- 172.21.15.110:0/811219238 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2db00c6a90 0x7f2db00aa8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:58.499 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.489+0000 7f2db782f700 1 --2- 172.21.15.110:0/811219238 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2db0069cf0 0x7f2db01560b0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:58.499 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.489+0000 7f2db782f700 1 -- 172.21.15.110:0/811219238 >> 172.21.15.110:0/811219238 conn(0x7f2db00cc710 msgr2=0x7f2db01599b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:29:58.499 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.489+0000 7f2db782f700 1 -- 172.21.15.110:0/811219238 shutdown_connections 2023-10-06T03:29:58.499 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.489+0000 7f2db782f700 1 -- 172.21.15.110:0/811219238 wait complete. 2023-10-06T03:29:58.509 INFO:teuthology.run_tasks:Running task ceph-fuse... 2023-10-06T03:29:58.517 INFO:tasks.ceph_fuse:Running ceph_fuse task... 2023-10-06T03:29:58.517 INFO:tasks.ceph_fuse:config is {'client.0': {}, 'client.2': {}, 'client.1': {}, 'client.3': {}} 2023-10-06T03:29:58.517 INFO:tasks.ceph_fuse:client.0 config is {} 2023-10-06T03:29:58.517 INFO:tasks.ceph_fuse:client.2 config is {} 2023-10-06T03:29:58.518 INFO:tasks.ceph_fuse:client.1 config is {} 2023-10-06T03:29:58.518 INFO:tasks.ceph_fuse:client.3 config is {} 2023-10-06T03:29:58.518 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-06T03:29:58.518 DEBUG:teuthology.orchestra.run.smithi110:> ip netns list 2023-10-06T03:29:58.526 INFO:teuthology.orchestra.run.smithi110.stdout:ceph-ns--home-ubuntu-cephtest-mnt.admin (id: 0) 2023-10-06T03:29:58.526 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-06T03:29:58.527 DEBUG:teuthology.orchestra.run.smithi110:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.admin 2023-10-06T03:29:58.543 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-06T03:29:58.544 DEBUG:teuthology.orchestra.run.smithi110:> sudo ip link delete ceph-brx 2023-10-06T03:29:58.725 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-06T03:29:58.725 DEBUG:teuthology.orchestra.run.smithi192:> ip netns list 2023-10-06T03:29:58.733 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-06T03:29:58.733 DEBUG:teuthology.orchestra.run.smithi192:> sudo ip link delete ceph-brx 2023-10-06T03:29:58.753 INFO:teuthology.orchestra.run.smithi192.stderr:Cannot find device "ceph-brx" 2023-10-06T03:29:58.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-06T03:29:58.755 INFO:tasks.ceph_fuse:Mounting ceph-fuse clients... 2023-10-06T03:29:58.755 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-10-06T03:29:58.871 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.865+0000 7fdd462ec700 1 Processor -- start 2023-10-06T03:29:58.871 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.865+0000 7fdd462ec700 1 -- start start 2023-10-06T03:29:58.872 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.869+0000 7fdd462ec700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fdd400ccb30 0x7fdd4006ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:58.872 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.869+0000 7fdd462ec700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fdd400ccf00 0x7fdd4006ca00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:58.872 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.869+0000 7fdd462ec700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd400c8010 con 0x7fdd400bf970 2023-10-06T03:29:58.872 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.869+0000 7fdd462ec700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdd4015d920 con 0x7fdd400ccb30 2023-10-06T03:29:58.872 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.869+0000 7fdd462ec700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdd400cd380 con 0x7fdd400ccf00 2023-10-06T03:29:58.872 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.869+0000 7fdd452ea700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fdd400ccb30 0x7fdd4006ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:58.872 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.869+0000 7fdd452ea700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fdd400ccb30 0x7fdd4006ba60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:47950/0 (socket says 172.21.15.110:47950) 2023-10-06T03:29:58.872 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.869+0000 7fdd452ea700 1 -- 172.21.15.110:0/448535694 learned_addr learned my addr 172.21.15.110:0/448535694 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:29:58.874 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.869+0000 7fdd44ae9700 1 --2- 172.21.15.110:0/448535694 >> v2:172.21.15.192:3300/0 conn(0x7fdd400ccf00 0x7fdd4006ca00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:58.874 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.869+0000 7fdd452ea700 1 -- 172.21.15.110:0/448535694 >> v2:172.21.15.192:3300/0 conn(0x7fdd400ccf00 msgr2=0x7fdd4006ca00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:29:58.874 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.869+0000 7fdd452ea700 1 --2- 172.21.15.110:0/448535694 >> v2:172.21.15.192:3300/0 conn(0x7fdd400ccf00 0x7fdd4006ca00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:58.874 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.869+0000 7fdd452ea700 1 -- 172.21.15.110:0/448535694 >> v1:172.21.15.192:6789/0 conn(0x7fdd400bf970 legacy=0x7fdd400bfdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:58.874 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.869+0000 7fdd452ea700 1 -- 172.21.15.110:0/448535694 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd4006cf40 con 0x7fdd400ccb30 2023-10-06T03:29:58.874 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.869+0000 7fdd452ea700 1 --2- 172.21.15.110:0/448535694 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdd400ccb30 0x7fdd4006ba60 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fdd40067c80 tx=0x7fdd38009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa7c56528820fe6b server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:29:58.874 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.869+0000 7fdd37fff700 1 -- 172.21.15.110:0/448535694 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd3800e070 con 0x7fdd400ccb30 2023-10-06T03:29:58.874 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.869+0000 7fdd37fff700 1 -- 172.21.15.110:0/448535694 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd3800ba50 con 0x7fdd400ccb30 2023-10-06T03:29:58.874 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.869+0000 7fdd37fff700 1 -- 172.21.15.110:0/448535694 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd380092e0 con 0x7fdd400ccb30 2023-10-06T03:29:58.874 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.869+0000 7fdd462ec700 1 -- 172.21.15.110:0/448535694 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdd400ccb30 msgr2=0x7fdd4006ba60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:58.875 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.869+0000 7fdd462ec700 1 --2- 172.21.15.110:0/448535694 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdd400ccb30 0x7fdd4006ba60 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fdd40067c80 tx=0x7fdd38009840 comp rx=0 tx=0).stop 2023-10-06T03:29:58.875 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.869+0000 7fdd462ec700 1 -- 172.21.15.110:0/448535694 shutdown_connections 2023-10-06T03:29:58.875 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.869+0000 7fdd462ec700 1 --2- 172.21.15.110:0/448535694 >> v2:172.21.15.192:3300/0 conn(0x7fdd400ccf00 0x7fdd4006ca00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:58.875 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.869+0000 7fdd462ec700 1 --2- 172.21.15.110:0/448535694 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdd400ccb30 0x7fdd4006ba60 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:58.875 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.869+0000 7fdd462ec700 1 -- 172.21.15.110:0/448535694 >> 172.21.15.110:0/448535694 conn(0x7fdd400670a0 msgr2=0x7fdd400674a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:29:58.875 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.869+0000 7fdd462ec700 1 -- 172.21.15.110:0/448535694 shutdown_connections 2023-10-06T03:29:58.875 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.869+0000 7fdd462ec700 1 -- 172.21.15.110:0/448535694 wait complete. 2023-10-06T03:29:58.877 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.873+0000 7fdd462ec700 1 Processor -- start 2023-10-06T03:29:58.877 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.873+0000 7fdd462ec700 1 -- start start 2023-10-06T03:29:58.878 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.873+0000 7fdd462ec700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdd400bf970 0x7fdd401536e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:58.878 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.873+0000 7fdd462ec700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fdd400ccb30 0x7fdd40153f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:58.878 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.873+0000 7fdd462ec700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdd400ccf00 0x7fdd400af860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:58.878 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.873+0000 7fdd462ec700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd400b06d0 con 0x7fdd400bf970 2023-10-06T03:29:58.878 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.873+0000 7fdd462ec700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd400c0d10 con 0x7fdd400ccf00 2023-10-06T03:29:58.878 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.873+0000 7fdd462ec700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd400c12d0 con 0x7fdd400ccb30 2023-10-06T03:29:58.879 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.873+0000 7fdd452ea700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdd400bf970 0x7fdd401536e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:58.879 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.873+0000 7fdd452ea700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdd400bf970 0x7fdd401536e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:47954/0 (socket says 172.21.15.110:47954) 2023-10-06T03:29:58.879 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.873+0000 7fdd452ea700 1 -- 172.21.15.110:0/1627736019 learned_addr learned my addr 172.21.15.110:0/1627736019 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:29:58.879 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.873+0000 7fdd45aeb700 1 --2- 172.21.15.110:0/1627736019 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdd400ccf00 0x7fdd400af860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:58.879 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.873+0000 7fdd44ae9700 1 --2- 172.21.15.110:0/1627736019 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fdd400ccb30 0x7fdd40153f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:58.880 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.873+0000 7fdd452ea700 1 -- 172.21.15.110:0/1627736019 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fdd400ccb30 msgr2=0x7fdd40153f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:58.880 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.873+0000 7fdd452ea700 1 --2- 172.21.15.110:0/1627736019 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fdd400ccb30 0x7fdd40153f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:58.880 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.873+0000 7fdd452ea700 1 -- 172.21.15.110:0/1627736019 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdd400ccf00 msgr2=0x7fdd400af860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:58.880 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.873+0000 7fdd452ea700 1 --2- 172.21.15.110:0/1627736019 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdd400ccf00 0x7fdd400af860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:58.880 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.873+0000 7fdd452ea700 1 -- 172.21.15.110:0/1627736019 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd38009ae0 con 0x7fdd400bf970 2023-10-06T03:29:58.881 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.873+0000 7fdd452ea700 1 --2- 172.21.15.110:0/1627736019 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdd400bf970 0x7fdd401536e0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fdd38009fd0 tx=0x7fdd38018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:29:58.881 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.877+0000 7fdd357fa700 1 -- 172.21.15.110:0/1627736019 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd38007820 con 0x7fdd400bf970 2023-10-06T03:29:58.881 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.877+0000 7fdd357fa700 1 -- 172.21.15.110:0/1627736019 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd38007980 con 0x7fdd400bf970 2023-10-06T03:29:58.881 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.877+0000 7fdd462ec700 1 -- 172.21.15.110:0/1627736019 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd400c58f0 con 0x7fdd400bf970 2023-10-06T03:29:58.881 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.877+0000 7fdd357fa700 1 -- 172.21.15.110:0/1627736019 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd38007d40 con 0x7fdd400bf970 2023-10-06T03:29:58.881 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.877+0000 7fdd462ec700 1 -- 172.21.15.110:0/1627736019 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd400c0f60 con 0x7fdd400bf970 2023-10-06T03:29:58.881 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.877+0000 7fdd462ec700 1 -- 172.21.15.110:0/1627736019 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd40070ff0 con 0x7fdd400bf970 2023-10-06T03:29:58.882 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.877+0000 7fdd357fa700 1 -- 172.21.15.110:0/1627736019 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7fdd3801d420 con 0x7fdd400bf970 2023-10-06T03:29:58.882 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.877+0000 7fdd357fa700 1 --2- 172.21.15.110:0/1627736019 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fdd24060580 0x7fdd24062a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:29:58.883 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.877+0000 7fdd357fa700 1 -- 172.21.15.110:0/1627736019 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6809+0+0 (secure 0 0 0) 0x7fdd3807e7f0 con 0x7fdd400bf970 2023-10-06T03:29:58.883 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.877+0000 7fdd44ae9700 1 --2- 172.21.15.110:0/1627736019 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fdd24060580 0x7fdd24062a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:29:58.883 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.881+0000 7fdd44ae9700 1 --2- 172.21.15.110:0/1627736019 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fdd24060580 0x7fdd24062a30 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fdd400add90 tx=0x7fdd30008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:29:58.890 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:58.885+0000 7fdd357fa700 1 -- 172.21.15.110:0/1627736019 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fdd3804ea40 con 0x7fdd400bf970 2023-10-06T03:29:59.087 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:59.081+0000 7fdd462ec700 1 -- 172.21.15.110:0/1627736019 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fdd400c82b0 con 0x7fdd400bf970 2023-10-06T03:29:59.087 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:59.085+0000 7fdd357fa700 1 -- 172.21.15.110:0/1627736019 <== mon.0 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v8) v1 ==== 53+0+88 (secure 0 0 0) 0x7fdd38021070 con 0x7fdd400bf970 2023-10-06T03:29:59.087 INFO:teuthology.orchestra.run.smithi110.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2023-10-06T03:29:59.091 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:59.089+0000 7fdd462ec700 1 -- 172.21.15.110:0/1627736019 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fdd24060580 msgr2=0x7fdd24062a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:59.092 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:59.089+0000 7fdd462ec700 1 --2- 172.21.15.110:0/1627736019 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fdd24060580 0x7fdd24062a30 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fdd400add90 tx=0x7fdd30008040 comp rx=0 tx=0).stop 2023-10-06T03:29:59.092 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:59.089+0000 7fdd462ec700 1 -- 172.21.15.110:0/1627736019 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdd400bf970 msgr2=0x7fdd401536e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:29:59.092 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:59.089+0000 7fdd462ec700 1 --2- 172.21.15.110:0/1627736019 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdd400bf970 0x7fdd401536e0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fdd38009fd0 tx=0x7fdd38018000 comp rx=0 tx=0).stop 2023-10-06T03:29:59.092 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:59.089+0000 7fdd462ec700 1 -- 172.21.15.110:0/1627736019 shutdown_connections 2023-10-06T03:29:59.092 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:59.089+0000 7fdd462ec700 1 --2- 172.21.15.110:0/1627736019 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fdd24060580 0x7fdd24062a30 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:59.092 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:59.089+0000 7fdd462ec700 1 --2- 172.21.15.110:0/1627736019 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdd400ccf00 0x7fdd400af860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:59.092 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:59.089+0000 7fdd462ec700 1 --2- 172.21.15.110:0/1627736019 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fdd400ccb30 0x7fdd40153f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:59.092 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:59.089+0000 7fdd462ec700 1 --2- 172.21.15.110:0/1627736019 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdd400bf970 0x7fdd401536e0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:29:59.092 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:59.089+0000 7fdd462ec700 1 -- 172.21.15.110:0/1627736019 >> 172.21.15.110:0/1627736019 conn(0x7fdd400670a0 msgr2=0x7fdd400ae1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:29:59.093 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:59.089+0000 7fdd462ec700 1 -- 172.21.15.110:0/1627736019 shutdown_connections 2023-10-06T03:29:59.093 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:29:59.089+0000 7fdd462ec700 1 -- 172.21.15.110:0/1627736019 wait complete. 2023-10-06T03:29:59.107 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2023-10-06T03:29:59.107 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-10-06T03:29:59.107 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi110.front.sepia.ceph.com 2023-10-06T03:29:59.107 INFO:tasks.cephfs.mount:self.client.name = client.0 2023-10-06T03:29:59.107 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-10-06T03:29:59.107 INFO:tasks.cephfs.mount:self.cephfs_name = None 2023-10-06T03:29:59.107 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2023-10-06T03:29:59.108 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-10-06T03:29:59.108 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2023-10-06T03:29:59.108 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-06T03:29:59.108 DEBUG:teuthology.orchestra.run.smithi110:> ip addr 2023-10-06T03:29:59.116 INFO:teuthology.orchestra.run.smithi110.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-10-06T03:29:59.116 INFO:teuthology.orchestra.run.smithi110.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-10-06T03:29:59.116 INFO:teuthology.orchestra.run.smithi110.stdout: inet 127.0.0.1/8 scope host lo 2023-10-06T03:29:59.116 INFO:teuthology.orchestra.run.smithi110.stdout: valid_lft forever preferred_lft forever 2023-10-06T03:29:59.116 INFO:teuthology.orchestra.run.smithi110.stdout: inet6 ::1/128 scope host 2023-10-06T03:29:59.116 INFO:teuthology.orchestra.run.smithi110.stdout: valid_lft forever preferred_lft forever 2023-10-06T03:29:59.116 INFO:teuthology.orchestra.run.smithi110.stdout:2: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2023-10-06T03:29:59.117 INFO:teuthology.orchestra.run.smithi110.stdout: link/ether 0c:c4:7a:d8:db:14 brd ff:ff:ff:ff:ff:ff 2023-10-06T03:29:59.117 INFO:teuthology.orchestra.run.smithi110.stdout:3: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000 2023-10-06T03:29:59.117 INFO:teuthology.orchestra.run.smithi110.stdout: link/ether 0c:c4:7a:d8:db:15 brd ff:ff:ff:ff:ff:ff 2023-10-06T03:29:59.117 INFO:teuthology.orchestra.run.smithi110.stdout:4: enp3s0f0: mtu 1500 qdisc noop state DOWN group default qlen 1000 2023-10-06T03:29:59.117 INFO:teuthology.orchestra.run.smithi110.stdout: link/ether 0c:c4:7a:8f:cc:34 brd ff:ff:ff:ff:ff:ff 2023-10-06T03:29:59.117 INFO:teuthology.orchestra.run.smithi110.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2023-10-06T03:29:59.117 INFO:teuthology.orchestra.run.smithi110.stdout: link/ether 0c:c4:7a:8f:cc:35 brd ff:ff:ff:ff:ff:ff 2023-10-06T03:29:59.117 INFO:teuthology.orchestra.run.smithi110.stdout: inet 172.21.15.110/20 brd 172.21.15.255 scope global dynamic enp3s0f1 2023-10-06T03:29:59.117 INFO:teuthology.orchestra.run.smithi110.stdout: valid_lft 42591sec preferred_lft 42591sec 2023-10-06T03:29:59.117 INFO:teuthology.orchestra.run.smithi110.stdout: inet6 fe80::ec4:7aff:fe8f:cc35/64 scope link 2023-10-06T03:29:59.117 INFO:teuthology.orchestra.run.smithi110.stdout: valid_lft forever preferred_lft forever 2023-10-06T03:29:59.118 INFO:teuthology.orchestra.run.smithi110.stdout:6: docker0: mtu 1500 qdisc noqueue state DOWN group default 2023-10-06T03:29:59.118 INFO:teuthology.orchestra.run.smithi110.stdout: link/ether 02:42:dc:de:aa:fb brd ff:ff:ff:ff:ff:ff 2023-10-06T03:29:59.118 INFO:teuthology.orchestra.run.smithi110.stdout: inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 2023-10-06T03:29:59.118 INFO:teuthology.orchestra.run.smithi110.stdout: valid_lft forever preferred_lft forever 2023-10-06T03:29:59.118 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2023-10-06T03:29:59.118 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-06T03:29:59.118 DEBUG:teuthology.orchestra.run.smithi110:> (cd / && exec bash -c ' 2023-10-06T03:29:59.118 DEBUG:teuthology.orchestra.run.smithi110:> set -e 2023-10-06T03:29:59.118 DEBUG:teuthology.orchestra.run.smithi110:> sudo ip link add name ceph-brx type bridge 2023-10-06T03:29:59.119 DEBUG:teuthology.orchestra.run.smithi110:> sudo ip addr flush dev ceph-brx 2023-10-06T03:29:59.119 DEBUG:teuthology.orchestra.run.smithi110:> sudo ip link set ceph-brx up 2023-10-06T03:29:59.119 DEBUG:teuthology.orchestra.run.smithi110:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2023-10-06T03:29:59.119 DEBUG:teuthology.orchestra.run.smithi110:> ') 2023-10-06T03:29:59.200 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-06T03:29:59.201 DEBUG:teuthology.orchestra.run.smithi110:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-10-06T03:29:59.210 INFO:teuthology.orchestra.run.smithi110.stdout:1 2023-10-06T03:29:59.211 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-06T03:29:59.211 DEBUG:teuthology.orchestra.run.smithi110:> route 2023-10-06T03:29:59.218 INFO:teuthology.orchestra.run.smithi110.stdout:Kernel IP routing table 2023-10-06T03:29:59.218 INFO:teuthology.orchestra.run.smithi110.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-10-06T03:29:59.218 INFO:teuthology.orchestra.run.smithi110.stdout:default _gateway 0.0.0.0 UG 0 0 0 enp3s0f1 2023-10-06T03:29:59.218 INFO:teuthology.orchestra.run.smithi110.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2023-10-06T03:29:59.218 INFO:teuthology.orchestra.run.smithi110.stdout:172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 2023-10-06T03:29:59.219 INFO:teuthology.orchestra.run.smithi110.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 0 0 0 enp3s0f1 2023-10-06T03:29:59.219 INFO:teuthology.orchestra.run.smithi110.stdout:_gateway 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2023-10-06T03:29:59.219 INFO:teuthology.orchestra.run.smithi110.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-10-06T03:29:59.219 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-06T03:29:59.219 DEBUG:teuthology.orchestra.run.smithi110:> (cd / && exec bash -c ' 2023-10-06T03:29:59.219 DEBUG:teuthology.orchestra.run.smithi110:> set -e 2023-10-06T03:29:59.219 DEBUG:teuthology.orchestra.run.smithi110:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2023-10-06T03:29:59.220 DEBUG:teuthology.orchestra.run.smithi110:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2023-10-06T03:29:59.220 DEBUG:teuthology.orchestra.run.smithi110:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2023-10-06T03:29:59.220 DEBUG:teuthology.orchestra.run.smithi110:> ') 2023-10-06T03:29:59.257 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-06T03:29:59.257 DEBUG:teuthology.orchestra.run.smithi110:> ip netns list 2023-10-06T03:29:59.310 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-06T03:29:59.311 DEBUG:teuthology.orchestra.run.smithi110:> ip netns list-id 2023-10-06T03:29:59.318 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-06T03:29:59.318 DEBUG:teuthology.orchestra.run.smithi110:> (cd / && exec bash -c ' 2023-10-06T03:29:59.318 DEBUG:teuthology.orchestra.run.smithi110:> set -e 2023-10-06T03:29:59.319 DEBUG:teuthology.orchestra.run.smithi110:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2023-10-06T03:29:59.319 DEBUG:teuthology.orchestra.run.smithi110:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2023-10-06T03:29:59.319 DEBUG:teuthology.orchestra.run.smithi110:> ') 2023-10-06T03:29:59.349 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2023-10-06T03:29:59.349 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-06T03:29:59.349 DEBUG:teuthology.orchestra.run.smithi110:> (cd / && exec bash -c ' 2023-10-06T03:29:59.349 DEBUG:teuthology.orchestra.run.smithi110:> set -e 2023-10-06T03:29:59.349 DEBUG:teuthology.orchestra.run.smithi110:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2023-10-06T03:29:59.350 DEBUG:teuthology.orchestra.run.smithi110:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2023-10-06T03:29:59.350 DEBUG:teuthology.orchestra.run.smithi110:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2023-10-06T03:29:59.350 DEBUG:teuthology.orchestra.run.smithi110:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2023-10-06T03:29:59.350 DEBUG:teuthology.orchestra.run.smithi110:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2023-10-06T03:29:59.350 DEBUG:teuthology.orchestra.run.smithi110:> ') 2023-10-06T03:29:59.444 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-06T03:29:59.444 DEBUG:teuthology.orchestra.run.smithi110:> (cd / && exec bash -c ' 2023-10-06T03:29:59.444 DEBUG:teuthology.orchestra.run.smithi110:> set -e 2023-10-06T03:29:59.444 DEBUG:teuthology.orchestra.run.smithi110:> sudo ip link set brx.0 up 2023-10-06T03:29:59.444 DEBUG:teuthology.orchestra.run.smithi110:> sudo ip link set dev brx.0 master ceph-brx 2023-10-06T03:29:59.444 DEBUG:teuthology.orchestra.run.smithi110:> ') 2023-10-06T03:29:59.473 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2023-10-06T03:29:59.473 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-06T03:29:59.473 DEBUG:teuthology.orchestra.run.smithi110:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2023-10-06T03:29:59.502 INFO:teuthology.orchestra.run.smithi110.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2023-10-06T03:29:59.503 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-06T03:29:59.503 DEBUG:teuthology.orchestra.run.smithi110:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2023-10-06T03:29:59.509 DEBUG:teuthology.orchestra.run.smithi110:> sudo modprobe fuse 2023-10-06T03:29:59.568 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-10-06T03:29:59.569 DEBUG:teuthology.orchestra.run.smithi110:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-10-06T03:29:59.582 INFO:teuthology.orchestra.run.smithi110.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-10-06T03:29:59.583 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-10-06T03:29:59.584 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-06T03:29:59.584 DEBUG:teuthology.orchestra.run.smithi110:> ls /sys/fs/fuse/connections 2023-10-06T03:29:59.589 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2023-10-06T03:29:59.589 DEBUG:teuthology.orchestra.run.smithi110:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2023-10-06T03:29:59.637 DEBUG:teuthology.orchestra.run.smithi110:> sudo modprobe fuse 2023-10-06T03:29:59.655 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-10-06T03:29:59.655 DEBUG:teuthology.orchestra.run.smithi110:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-10-06T03:29:59.664 INFO:teuthology.orchestra.run.smithi110.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-10-06T03:29:59.664 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-10-06T03:29:59.665 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-06T03:29:59.665 DEBUG:teuthology.orchestra.run.smithi110:> ls /sys/fs/fuse/connections 2023-10-06T03:29:59.725 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi110.stderr:2023-10-06T03:29:59.721+0000 7fa72bfbc380 -1 init, newargv = 0x563e416fb2e0 newargc=15 2023-10-06T03:29:59.725 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi110.stderr:ceph-fuse[40003]: starting ceph client 2023-10-06T03:29:59.734 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi110.stderr:ceph-fuse[40003]: starting fuse 2023-10-06T03:30:00.815 DEBUG:teuthology.orchestra.run.smithi110:> sudo modprobe fuse 2023-10-06T03:30:00.834 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-10-06T03:30:00.834 DEBUG:teuthology.orchestra.run.smithi110:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-10-06T03:30:00.844 INFO:teuthology.orchestra.run.smithi110.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-10-06T03:30:00.845 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-10-06T03:30:00.846 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-06T03:30:00.846 DEBUG:teuthology.orchestra.run.smithi110:> ls /sys/fs/fuse/connections 2023-10-06T03:30:00.894 INFO:teuthology.orchestra.run.smithi110.stdout:52 2023-10-06T03:30:00.894 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [52] 2023-10-06T03:30:00.894 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2023-10-06T03:30:00.894 DEBUG:teuthology.orchestra.run.smithi110:> import glob 2023-10-06T03:30:00.894 DEBUG:teuthology.orchestra.run.smithi110:> import re 2023-10-06T03:30:00.894 DEBUG:teuthology.orchestra.run.smithi110:> import os 2023-10-06T03:30:00.894 DEBUG:teuthology.orchestra.run.smithi110:> import subprocess 2023-10-06T03:30:00.895 DEBUG:teuthology.orchestra.run.smithi110:> 2023-10-06T03:30:00.895 DEBUG:teuthology.orchestra.run.smithi110:> def _find_admin_socket(client_name): 2023-10-06T03:30:00.895 DEBUG:teuthology.orchestra.run.smithi110:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2023-10-06T03:30:00.895 DEBUG:teuthology.orchestra.run.smithi110:> files = glob.glob(asok_path) 2023-10-06T03:30:00.895 DEBUG:teuthology.orchestra.run.smithi110:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2023-10-06T03:30:00.895 DEBUG:teuthology.orchestra.run.smithi110:> 2023-10-06T03:30:00.895 DEBUG:teuthology.orchestra.run.smithi110:> # Given a non-glob path, it better be there 2023-10-06T03:30:00.895 DEBUG:teuthology.orchestra.run.smithi110:> if "*" not in asok_path: 2023-10-06T03:30:00.895 DEBUG:teuthology.orchestra.run.smithi110:> assert(len(files) == 1) 2023-10-06T03:30:00.895 DEBUG:teuthology.orchestra.run.smithi110:> return files[0] 2023-10-06T03:30:00.895 DEBUG:teuthology.orchestra.run.smithi110:> 2023-10-06T03:30:00.896 DEBUG:teuthology.orchestra.run.smithi110:> for f in files: 2023-10-06T03:30:00.896 DEBUG:teuthology.orchestra.run.smithi110:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2023-10-06T03:30:00.896 DEBUG:teuthology.orchestra.run.smithi110:> if os.path.exists("/proc/{0}".format(pid)): 2023-10-06T03:30:00.896 DEBUG:teuthology.orchestra.run.smithi110:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2023-10-06T03:30:00.896 DEBUG:teuthology.orchestra.run.smithi110:> contents = proc_f.read() 2023-10-06T03:30:00.896 DEBUG:teuthology.orchestra.run.smithi110:> if mountpoint in contents: 2023-10-06T03:30:00.896 DEBUG:teuthology.orchestra.run.smithi110:> return f 2023-10-06T03:30:00.896 DEBUG:teuthology.orchestra.run.smithi110:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2023-10-06T03:30:00.896 DEBUG:teuthology.orchestra.run.smithi110:> 2023-10-06T03:30:00.896 DEBUG:teuthology.orchestra.run.smithi110:> print(_find_admin_socket("client.0")) 2023-10-06T03:30:00.896 DEBUG:teuthology.orchestra.run.smithi110:> ' 2023-10-06T03:30:00.979 INFO:teuthology.orchestra.run.smithi110.stdout:/var/run/ceph/ceph-client.0.40003.asok 2023-10-06T03:30:01.163 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.40003.asok 2023-10-06T03:30:01.164 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-06T03:30:01.164 DEBUG:teuthology.orchestra.run.smithi110:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.40003.asok status 2023-10-06T03:30:01.258 INFO:teuthology.orchestra.run.smithi110.stdout:{ 2023-10-06T03:30:01.258 INFO:teuthology.orchestra.run.smithi110.stdout: "metadata": { 2023-10-06T03:30:01.258 INFO:teuthology.orchestra.run.smithi110.stdout: "ceph_sha1": "736ac5ba72cb47af7d6f6666b1e100f7e20a20e2", 2023-10-06T03:30:01.258 INFO:teuthology.orchestra.run.smithi110.stdout: "ceph_version": "ceph version 17.2.6-1296-g736ac5ba (736ac5ba72cb47af7d6f6666b1e100f7e20a20e2) quincy (stable)", 2023-10-06T03:30:01.258 INFO:teuthology.orchestra.run.smithi110.stdout: "entity_id": "0", 2023-10-06T03:30:01.259 INFO:teuthology.orchestra.run.smithi110.stdout: "hostname": "smithi110", 2023-10-06T03:30:01.259 INFO:teuthology.orchestra.run.smithi110.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2023-10-06T03:30:01.259 INFO:teuthology.orchestra.run.smithi110.stdout: "pid": "40003", 2023-10-06T03:30:01.259 INFO:teuthology.orchestra.run.smithi110.stdout: "root": "/" 2023-10-06T03:30:01.259 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2023-10-06T03:30:01.259 INFO:teuthology.orchestra.run.smithi110.stdout: "dentry_count": 0, 2023-10-06T03:30:01.259 INFO:teuthology.orchestra.run.smithi110.stdout: "dentry_pinned_count": 0, 2023-10-06T03:30:01.259 INFO:teuthology.orchestra.run.smithi110.stdout: "id": 4821, 2023-10-06T03:30:01.259 INFO:teuthology.orchestra.run.smithi110.stdout: "inst": { 2023-10-06T03:30:01.259 INFO:teuthology.orchestra.run.smithi110.stdout: "name": { 2023-10-06T03:30:01.259 INFO:teuthology.orchestra.run.smithi110.stdout: "type": "client", 2023-10-06T03:30:01.260 INFO:teuthology.orchestra.run.smithi110.stdout: "num": 4821 2023-10-06T03:30:01.260 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2023-10-06T03:30:01.260 INFO:teuthology.orchestra.run.smithi110.stdout: "addr": { 2023-10-06T03:30:01.260 INFO:teuthology.orchestra.run.smithi110.stdout: "type": "v1", 2023-10-06T03:30:01.260 INFO:teuthology.orchestra.run.smithi110.stdout: "addr": "192.168.0.1:0", 2023-10-06T03:30:01.260 INFO:teuthology.orchestra.run.smithi110.stdout: "nonce": 2708030271 2023-10-06T03:30:01.260 INFO:teuthology.orchestra.run.smithi110.stdout: } 2023-10-06T03:30:01.260 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2023-10-06T03:30:01.260 INFO:teuthology.orchestra.run.smithi110.stdout: "addr": { 2023-10-06T03:30:01.260 INFO:teuthology.orchestra.run.smithi110.stdout: "type": "v1", 2023-10-06T03:30:01.260 INFO:teuthology.orchestra.run.smithi110.stdout: "addr": "192.168.0.1:0", 2023-10-06T03:30:01.261 INFO:teuthology.orchestra.run.smithi110.stdout: "nonce": 2708030271 2023-10-06T03:30:01.261 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2023-10-06T03:30:01.261 INFO:teuthology.orchestra.run.smithi110.stdout: "inst_str": "client.4821 192.168.0.1:0/2708030271", 2023-10-06T03:30:01.261 INFO:teuthology.orchestra.run.smithi110.stdout: "addr_str": "192.168.0.1:0/2708030271", 2023-10-06T03:30:01.261 INFO:teuthology.orchestra.run.smithi110.stdout: "inode_count": 1, 2023-10-06T03:30:01.261 INFO:teuthology.orchestra.run.smithi110.stdout: "mds_epoch": 8, 2023-10-06T03:30:01.261 INFO:teuthology.orchestra.run.smithi110.stdout: "osd_epoch": 28, 2023-10-06T03:30:01.261 INFO:teuthology.orchestra.run.smithi110.stdout: "osd_epoch_barrier": 0, 2023-10-06T03:30:01.261 INFO:teuthology.orchestra.run.smithi110.stdout: "blocklisted": false, 2023-10-06T03:30:01.261 INFO:teuthology.orchestra.run.smithi110.stdout: "fs_name": "cephfs" 2023-10-06T03:30:01.261 INFO:teuthology.orchestra.run.smithi110.stdout:} 2023-10-06T03:30:01.267 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-10-06T03:30:01.377 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:01.373+0000 7fb41121c700 1 Processor -- start 2023-10-06T03:30:01.377 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:01.373+0000 7fb41121c700 1 -- start start 2023-10-06T03:30:01.378 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:01.373+0000 7fb41121c700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fb40c0ca020 0x7fb40c159140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:01.378 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:01.373+0000 7fb41121c700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb40c0ca500 con 0x7fb40c0adcd0 2023-10-06T03:30:01.378 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:01.373+0000 7fb41121c700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb40c0ca7e0 con 0x7fb40c06f340 2023-10-06T03:30:01.378 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:01.373+0000 7fb41121c700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb40c156440 con 0x7fb40c0ca020 2023-10-06T03:30:01.380 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:01.373+0000 7fb40bfff700 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7fb40c0adcd0 0x7fb40c158a30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:59414/0 (socket says 172.21.15.110:59414) 2023-10-06T03:30:01.380 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:01.373+0000 7fb40bfff700 1 -- 172.21.15.110:0/2412257537 learned_addr learned my addr 172.21.15.110:0/2412257537 (peer_addr_for_me v1:172.21.15.110:0/0) 2023-10-06T03:30:01.380 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:01.373+0000 7fb40b7fe700 1 --2- 172.21.15.110:0/2412257537 >> v2:172.21.15.192:3300/0 conn(0x7fb40c0ca020 0x7fb40c159140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:01.380 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:01.373+0000 7fb40affd700 1 -- 172.21.15.110:0/2412257537 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1934302411 0 0) 0x7fb40c0ca500 con 0x7fb40c0adcd0 2023-10-06T03:30:01.380 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:01.373+0000 7fb40affd700 1 -- 172.21.15.110:0/2412257537 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb3f8003570 con 0x7fb40c0adcd0 2023-10-06T03:30:01.380 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:01.373+0000 7fb40affd700 1 -- 172.21.15.110:0/2412257537 <== mon.0 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3676247735 0 0) 0x7fb3f8003570 con 0x7fb40c0adcd0 2023-10-06T03:30:01.381 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:01.373+0000 7fb40affd700 1 -- 172.21.15.110:0/2412257537 >> v2:172.21.15.192:3300/0 conn(0x7fb40c0ca020 msgr2=0x7fb40c159140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:30:01.381 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:01.373+0000 7fb40affd700 1 --2- 172.21.15.110:0/2412257537 >> v2:172.21.15.192:3300/0 conn(0x7fb40c0ca020 0x7fb40c159140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:01.381 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:01.373+0000 7fb40affd700 1 -- 172.21.15.110:0/2412257537 >> v1:172.21.15.192:6789/0 conn(0x7fb40c06f340 legacy=0x7fb40c0c6c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:01.381 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:01.373+0000 7fb40affd700 1 -- 172.21.15.110:0/2412257537 --> v1:172.21.15.110:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb40c0caac0 con 0x7fb40c0adcd0 2023-10-06T03:30:01.381 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:01.373+0000 7fb40affd700 1 -- 172.21.15.110:0/2412257537 <== mon.0 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 118603703 0 0) 0x7fb3f4001470 con 0x7fb40c0adcd0 2023-10-06T03:30:01.381 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:01.373+0000 7fb40affd700 1 -- 172.21.15.110:0/2412257537 >> v1:172.21.15.110:6789/0 conn(0x7fb40c0adcd0 legacy=0x7fb40c158a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:01.381 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:01.373+0000 7fb40affd700 1 --2- 172.21.15.110:0/2412257537 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb3f8003a60 0x7fb3f8003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:01.381 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:01.373+0000 7fb40affd700 1 --2- 172.21.15.110:0/2412257537 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb3f8004810 0x7fb3f8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:01.381 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:01.373+0000 7fb40affd700 1 --2- 172.21.15.110:0/2412257537 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb3f80051c0 0x7fb3f800dc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:01.381 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:01.373+0000 7fb40affd700 1 -- 172.21.15.110:0/2412257537 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb40c156440 con 0x7fb3f8003a60 2023-10-06T03:30:01.381 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:01.373+0000 7fb40affd700 1 -- 172.21.15.110:0/2412257537 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3f800e370 con 0x7fb3f8004810 2023-10-06T03:30:01.382 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:01.373+0000 7fb40affd700 1 -- 172.21.15.110:0/2412257537 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3f800e530 con 0x7fb3f80051c0 2023-10-06T03:30:01.382 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:01.377+0000 7fb40bfff700 1 --2- 172.21.15.110:0/2412257537 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb3f8003a60 0x7fb3f8003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:01.382 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:01.377+0000 7fb410a1b700 1 --2- 172.21.15.110:0/2412257537 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb3f80051c0 0x7fb3f800dc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:01.382 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:01.377+0000 7fb40b7fe700 1 --2- 172.21.15.110:0/2412257537 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb3f8004810 0x7fb3f8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:01.382 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:01.377+0000 7fb40b7fe700 1 -- 172.21.15.110:0/2412257537 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb3f80051c0 msgr2=0x7fb3f800dc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:30:01.382 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:01.377+0000 7fb40b7fe700 1 --2- 172.21.15.110:0/2412257537 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb3f80051c0 0x7fb3f800dc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:01.382 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:01.377+0000 7fb40b7fe700 1 -- 172.21.15.110:0/2412257537 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb3f8003a60 msgr2=0x7fb3f8003e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:30:01.382 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:01.377+0000 7fb40b7fe700 1 --2- 172.21.15.110:0/2412257537 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb3f8003a60 0x7fb3f8003e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:01.382 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:01.377+0000 7fb40b7fe700 1 -- 172.21.15.110:0/2412257537 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3f8003570 con 0x7fb3f8004810 2023-10-06T03:30:01.382 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:01.377+0000 7fb410a1b700 1 -- 172.21.15.110:0/2412257537 reap_dead start 2023-10-06T03:30:01.382 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:01.377+0000 7fb40b7fe700 1 --2- 172.21.15.110:0/2412257537 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb3f8004810 0x7fb3f8004c80 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fb40c06d350 tx=0x7fb3fc00a890 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a024bec99842f434 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:30:01.385 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:01.377+0000 7fb40affd700 1 -- 172.21.15.110:0/2412257537 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3fc00acc0 con 0x7fb3f8004810 2023-10-06T03:30:01.385 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:01.377+0000 7fb40affd700 1 -- 172.21.15.110:0/2412257537 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3fc010040 con 0x7fb3f8004810 2023-10-06T03:30:01.385 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:01.377+0000 7fb40affd700 1 -- 172.21.15.110:0/2412257537 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3fc009dc0 con 0x7fb3f8004810 2023-10-06T03:30:01.385 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:01.377+0000 7fb41121c700 1 -- 172.21.15.110:0/2412257537 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb3f8004810 msgr2=0x7fb3f8004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:01.385 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:01.377+0000 7fb41121c700 1 --2- 172.21.15.110:0/2412257537 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb3f8004810 0x7fb3f8004c80 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fb40c06d350 tx=0x7fb3fc00a890 comp rx=0 tx=0).stop 2023-10-06T03:30:01.385 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:01.377+0000 7fb41121c700 1 -- 172.21.15.110:0/2412257537 shutdown_connections 2023-10-06T03:30:01.385 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:01.377+0000 7fb41121c700 1 --2- 172.21.15.110:0/2412257537 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb3f8004810 0x7fb3f8004c80 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:01.386 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:01.377+0000 7fb41121c700 1 -- 172.21.15.110:0/2412257537 >> 172.21.15.110:0/2412257537 conn(0x7fb40c0cc470 msgr2=0x7fb40c0c8410 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:30:01.386 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:01.377+0000 7fb41121c700 1 -- 172.21.15.110:0/2412257537 shutdown_connections 2023-10-06T03:30:01.386 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:01.377+0000 7fb41121c700 1 -- 172.21.15.110:0/2412257537 wait complete. 2023-10-06T03:30:01.386 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:01.377+0000 7fb41121c700 1 Processor -- start 2023-10-06T03:30:01.386 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:01.377+0000 7fb41121c700 1 -- start start 2023-10-06T03:30:01.386 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:01.381+0000 7fb41121c700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb3f8004810 0x7fb40c0c1e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:01.386 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:01.381+0000 7fb41121c700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb40c0c2390 0x7fb40c150db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:01.386 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:01.381+0000 7fb41121c700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb40c1512f0 0x7fb40c14e960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:01.387 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:01.381+0000 7fb41121c700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb40c1688d0 con 0x7fb3f8004810 2023-10-06T03:30:01.387 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:01.381+0000 7fb41121c700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb40c153300 con 0x7fb40c1512f0 2023-10-06T03:30:01.387 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:01.381+0000 7fb41121c700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb40c151770 con 0x7fb40c0c2390 2023-10-06T03:30:01.387 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:01.381+0000 7fb40bfff700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb3f8004810 0x7fb40c0c1e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:01.387 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:01.381+0000 7fb40bfff700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb3f8004810 0x7fb40c0c1e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:40402/0 (socket says 172.21.15.110:40402) 2023-10-06T03:30:01.387 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:01.381+0000 7fb40bfff700 1 -- 172.21.15.110:0/275312677 learned_addr learned my addr 172.21.15.110:0/275312677 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:30:01.387 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:01.381+0000 7fb40b7fe700 1 --2- 172.21.15.110:0/275312677 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb40c0c2390 0x7fb40c150db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:01.387 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:01.381+0000 7fb410a1b700 1 --2- 172.21.15.110:0/275312677 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb40c1512f0 0x7fb40c14e960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:01.387 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:01.381+0000 7fb40bfff700 1 -- 172.21.15.110:0/275312677 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb40c0c2390 msgr2=0x7fb40c150db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:01.387 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:01.381+0000 7fb40bfff700 1 --2- 172.21.15.110:0/275312677 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb40c0c2390 0x7fb40c150db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:01.387 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:01.381+0000 7fb40bfff700 1 -- 172.21.15.110:0/275312677 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb40c1512f0 msgr2=0x7fb40c14e960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:01.388 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:01.381+0000 7fb40bfff700 1 --2- 172.21.15.110:0/275312677 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb40c1512f0 0x7fb40c14e960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:01.388 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:01.381+0000 7fb40bfff700 1 -- 172.21.15.110:0/275312677 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb40c151a20 con 0x7fb3f8004810 2023-10-06T03:30:01.388 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:01.381+0000 7fb410a1b700 1 --2- 172.21.15.110:0/275312677 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb40c1512f0 0x7fb40c14e960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-06T03:30:01.388 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:01.381+0000 7fb40bfff700 1 --2- 172.21.15.110:0/275312677 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb3f8004810 0x7fb40c0c1e50 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fb3f4007390 tx=0x7fb3f4001e50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:30:01.389 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:01.381+0000 7fb403fff700 1 -- 172.21.15.110:0/275312677 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3f4018030 con 0x7fb3f8004810 2023-10-06T03:30:01.389 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:01.381+0000 7fb403fff700 1 -- 172.21.15.110:0/275312677 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3f4007750 con 0x7fb3f8004810 2023-10-06T03:30:01.389 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:01.381+0000 7fb41121c700 1 -- 172.21.15.110:0/275312677 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb40c151d00 con 0x7fb3f8004810 2023-10-06T03:30:01.389 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:01.381+0000 7fb403fff700 1 -- 172.21.15.110:0/275312677 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3f400fb20 con 0x7fb3f8004810 2023-10-06T03:30:01.389 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:01.381+0000 7fb41121c700 1 -- 172.21.15.110:0/275312677 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb40c1522c0 con 0x7fb3f8004810 2023-10-06T03:30:01.389 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:01.385+0000 7fb41121c700 1 -- 172.21.15.110:0/275312677 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb40c070ff0 con 0x7fb3f8004810 2023-10-06T03:30:01.390 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:01.385+0000 7fb403fff700 1 -- 172.21.15.110:0/275312677 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7fb3f401b070 con 0x7fb3f8004810 2023-10-06T03:30:01.390 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:01.385+0000 7fb403fff700 1 --2- 172.21.15.110:0/275312677 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fb3ec060580 0x7fb3ec062a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:01.390 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:01.385+0000 7fb403fff700 1 -- 172.21.15.110:0/275312677 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6809+0+0 (secure 0 0 0) 0x7fb3f4019070 con 0x7fb3f8004810 2023-10-06T03:30:01.390 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:01.385+0000 7fb40b7fe700 1 --2- 172.21.15.110:0/275312677 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fb3ec060580 0x7fb3ec062a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:01.390 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:01.385+0000 7fb40b7fe700 1 --2- 172.21.15.110:0/275312677 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fb3ec060580 0x7fb3ec062a30 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fb3fc00a8d0 tx=0x7fb3fc019000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:30:01.394 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:01.389+0000 7fb403fff700 1 -- 172.21.15.110:0/275312677 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fb3f404cbc0 con 0x7fb3f8004810 2023-10-06T03:30:01.587 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:01.581+0000 7fb41121c700 1 -- 172.21.15.110:0/275312677 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fb40c0cada0 con 0x7fb3f8004810 2023-10-06T03:30:01.587 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:01.585+0000 7fb403fff700 1 -- 172.21.15.110:0/275312677 <== mon.0 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v8) v1 ==== 53+0+88 (secure 0 0 0) 0x7fb3f407ae60 con 0x7fb3f8004810 2023-10-06T03:30:01.588 INFO:teuthology.orchestra.run.smithi110.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2023-10-06T03:30:01.590 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:01.585+0000 7fb41121c700 1 -- 172.21.15.110:0/275312677 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fb3ec060580 msgr2=0x7fb3ec062a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:01.590 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:01.585+0000 7fb41121c700 1 --2- 172.21.15.110:0/275312677 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fb3ec060580 0x7fb3ec062a30 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fb3fc00a8d0 tx=0x7fb3fc019000 comp rx=0 tx=0).stop 2023-10-06T03:30:01.591 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:01.585+0000 7fb41121c700 1 -- 172.21.15.110:0/275312677 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb3f8004810 msgr2=0x7fb40c0c1e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:01.591 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:01.585+0000 7fb41121c700 1 --2- 172.21.15.110:0/275312677 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb3f8004810 0x7fb40c0c1e50 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fb3f4007390 tx=0x7fb3f4001e50 comp rx=0 tx=0).stop 2023-10-06T03:30:01.591 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:01.585+0000 7fb41121c700 1 -- 172.21.15.110:0/275312677 shutdown_connections 2023-10-06T03:30:01.591 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:01.585+0000 7fb41121c700 1 --2- 172.21.15.110:0/275312677 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fb3ec060580 0x7fb3ec062a30 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:01.591 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:01.585+0000 7fb41121c700 1 --2- 172.21.15.110:0/275312677 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb40c1512f0 0x7fb40c14e960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:01.591 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:01.585+0000 7fb41121c700 1 --2- 172.21.15.110:0/275312677 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb40c0c2390 0x7fb40c150db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:01.591 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:01.585+0000 7fb41121c700 1 --2- 172.21.15.110:0/275312677 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb3f8004810 0x7fb40c0c1e50 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:01.591 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:01.585+0000 7fb41121c700 1 -- 172.21.15.110:0/275312677 >> 172.21.15.110:0/275312677 conn(0x7fb40c0cc470 msgr2=0x7fb40c06b850 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:30:01.592 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:01.585+0000 7fb41121c700 1 -- 172.21.15.110:0/275312677 shutdown_connections 2023-10-06T03:30:01.592 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:01.585+0000 7fb41121c700 1 -- 172.21.15.110:0/275312677 wait complete. 2023-10-06T03:30:01.601 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2023-10-06T03:30:01.601 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-10-06T03:30:01.601 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi110.front.sepia.ceph.com 2023-10-06T03:30:01.601 INFO:tasks.cephfs.mount:self.client.name = client.2 2023-10-06T03:30:01.601 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.2 2023-10-06T03:30:01.601 INFO:tasks.cephfs.mount:self.cephfs_name = None 2023-10-06T03:30:01.602 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2023-10-06T03:30:01.602 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-10-06T03:30:01.602 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.2' 2023-10-06T03:30:01.602 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-06T03:30:01.602 DEBUG:teuthology.orchestra.run.smithi110:> ip addr 2023-10-06T03:30:01.611 INFO:teuthology.orchestra.run.smithi110.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-10-06T03:30:01.611 INFO:teuthology.orchestra.run.smithi110.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-10-06T03:30:01.611 INFO:teuthology.orchestra.run.smithi110.stdout: inet 127.0.0.1/8 scope host lo 2023-10-06T03:30:01.611 INFO:teuthology.orchestra.run.smithi110.stdout: valid_lft forever preferred_lft forever 2023-10-06T03:30:01.611 INFO:teuthology.orchestra.run.smithi110.stdout: inet6 ::1/128 scope host 2023-10-06T03:30:01.611 INFO:teuthology.orchestra.run.smithi110.stdout: valid_lft forever preferred_lft forever 2023-10-06T03:30:01.611 INFO:teuthology.orchestra.run.smithi110.stdout:2: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2023-10-06T03:30:01.611 INFO:teuthology.orchestra.run.smithi110.stdout: link/ether 0c:c4:7a:d8:db:14 brd ff:ff:ff:ff:ff:ff 2023-10-06T03:30:01.612 INFO:teuthology.orchestra.run.smithi110.stdout:3: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000 2023-10-06T03:30:01.612 INFO:teuthology.orchestra.run.smithi110.stdout: link/ether 0c:c4:7a:d8:db:15 brd ff:ff:ff:ff:ff:ff 2023-10-06T03:30:01.612 INFO:teuthology.orchestra.run.smithi110.stdout:4: enp3s0f0: mtu 1500 qdisc noop state DOWN group default qlen 1000 2023-10-06T03:30:01.612 INFO:teuthology.orchestra.run.smithi110.stdout: link/ether 0c:c4:7a:8f:cc:34 brd ff:ff:ff:ff:ff:ff 2023-10-06T03:30:01.612 INFO:teuthology.orchestra.run.smithi110.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2023-10-06T03:30:01.612 INFO:teuthology.orchestra.run.smithi110.stdout: link/ether 0c:c4:7a:8f:cc:35 brd ff:ff:ff:ff:ff:ff 2023-10-06T03:30:01.612 INFO:teuthology.orchestra.run.smithi110.stdout: inet 172.21.15.110/20 brd 172.21.15.255 scope global dynamic enp3s0f1 2023-10-06T03:30:01.613 INFO:teuthology.orchestra.run.smithi110.stdout: valid_lft 42589sec preferred_lft 42589sec 2023-10-06T03:30:01.613 INFO:teuthology.orchestra.run.smithi110.stdout: inet6 fe80::ec4:7aff:fe8f:cc35/64 scope link 2023-10-06T03:30:01.613 INFO:teuthology.orchestra.run.smithi110.stdout: valid_lft forever preferred_lft forever 2023-10-06T03:30:01.613 INFO:teuthology.orchestra.run.smithi110.stdout:6: docker0: mtu 1500 qdisc noqueue state DOWN group default 2023-10-06T03:30:01.613 INFO:teuthology.orchestra.run.smithi110.stdout: link/ether 02:42:dc:de:aa:fb brd ff:ff:ff:ff:ff:ff 2023-10-06T03:30:01.613 INFO:teuthology.orchestra.run.smithi110.stdout: inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 2023-10-06T03:30:01.613 INFO:teuthology.orchestra.run.smithi110.stdout: valid_lft forever preferred_lft forever 2023-10-06T03:30:01.614 INFO:teuthology.orchestra.run.smithi110.stdout:9: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-10-06T03:30:01.614 INFO:teuthology.orchestra.run.smithi110.stdout: link/ether 5e:32:2b:b4:0f:e6 brd ff:ff:ff:ff:ff:ff 2023-10-06T03:30:01.614 INFO:teuthology.orchestra.run.smithi110.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-10-06T03:30:01.614 INFO:teuthology.orchestra.run.smithi110.stdout: valid_lft forever preferred_lft forever 2023-10-06T03:30:01.614 INFO:teuthology.orchestra.run.smithi110.stdout: inet6 fe80::34e3:85ff:feb6:e924/64 scope link 2023-10-06T03:30:01.614 INFO:teuthology.orchestra.run.smithi110.stdout: valid_lft forever preferred_lft forever 2023-10-06T03:30:01.614 INFO:teuthology.orchestra.run.smithi110.stdout:10: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-10-06T03:30:01.614 INFO:teuthology.orchestra.run.smithi110.stdout: link/ether 5e:32:2b:b4:0f:e6 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2023-10-06T03:30:01.614 INFO:teuthology.orchestra.run.smithi110.stdout: inet6 fe80::5c32:2bff:feb4:fe6/64 scope link 2023-10-06T03:30:01.614 INFO:teuthology.orchestra.run.smithi110.stdout: valid_lft forever preferred_lft forever 2023-10-06T03:30:01.615 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-06T03:30:01.615 DEBUG:teuthology.orchestra.run.smithi110:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-10-06T03:30:01.630 INFO:teuthology.orchestra.run.smithi110.stdout:1 2023-10-06T03:30:01.632 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-06T03:30:01.632 DEBUG:teuthology.orchestra.run.smithi110:> route 2023-10-06T03:30:01.683 INFO:teuthology.orchestra.run.smithi110.stdout:Kernel IP routing table 2023-10-06T03:30:01.683 INFO:teuthology.orchestra.run.smithi110.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-10-06T03:30:01.684 INFO:teuthology.orchestra.run.smithi110.stdout:default _gateway 0.0.0.0 UG 0 0 0 enp3s0f1 2023-10-06T03:30:01.684 INFO:teuthology.orchestra.run.smithi110.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2023-10-06T03:30:01.684 INFO:teuthology.orchestra.run.smithi110.stdout:172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 2023-10-06T03:30:01.684 INFO:teuthology.orchestra.run.smithi110.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 0 0 0 enp3s0f1 2023-10-06T03:30:01.684 INFO:teuthology.orchestra.run.smithi110.stdout:_gateway 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2023-10-06T03:30:01.684 INFO:teuthology.orchestra.run.smithi110.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-10-06T03:30:01.684 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-06T03:30:01.685 DEBUG:teuthology.orchestra.run.smithi110:> (cd / && exec bash -c ' 2023-10-06T03:30:01.685 DEBUG:teuthology.orchestra.run.smithi110:> set -e 2023-10-06T03:30:01.685 DEBUG:teuthology.orchestra.run.smithi110:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2023-10-06T03:30:01.685 DEBUG:teuthology.orchestra.run.smithi110:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2023-10-06T03:30:01.685 DEBUG:teuthology.orchestra.run.smithi110:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2023-10-06T03:30:01.685 DEBUG:teuthology.orchestra.run.smithi110:> ') 2023-10-06T03:30:01.723 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-06T03:30:01.723 DEBUG:teuthology.orchestra.run.smithi110:> ip netns list 2023-10-06T03:30:01.731 INFO:teuthology.orchestra.run.smithi110.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2023-10-06T03:30:01.731 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-06T03:30:01.731 DEBUG:teuthology.orchestra.run.smithi110:> ip netns list-id 2023-10-06T03:30:01.739 INFO:teuthology.orchestra.run.smithi110.stdout:nsid 0 (iproute2 netns name: ceph-ns--home-ubuntu-cephtest-mnt.0) 2023-10-06T03:30:01.741 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-06T03:30:01.741 DEBUG:teuthology.orchestra.run.smithi110:> (cd / && exec bash -c ' 2023-10-06T03:30:01.741 DEBUG:teuthology.orchestra.run.smithi110:> set -e 2023-10-06T03:30:01.742 DEBUG:teuthology.orchestra.run.smithi110:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.2 2023-10-06T03:30:01.742 DEBUG:teuthology.orchestra.run.smithi110:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.2 1 2023-10-06T03:30:01.742 DEBUG:teuthology.orchestra.run.smithi110:> ') 2023-10-06T03:30:01.811 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-06T03:30:01.811 DEBUG:teuthology.orchestra.run.smithi110:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr 2023-10-06T03:30:01.830 INFO:teuthology.orchestra.run.smithi110.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-10-06T03:30:01.830 INFO:teuthology.orchestra.run.smithi110.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-10-06T03:30:01.830 INFO:teuthology.orchestra.run.smithi110.stdout: inet 127.0.0.1/8 scope host lo 2023-10-06T03:30:01.830 INFO:teuthology.orchestra.run.smithi110.stdout: valid_lft forever preferred_lft forever 2023-10-06T03:30:01.831 INFO:teuthology.orchestra.run.smithi110.stdout: inet6 ::1/128 scope host 2023-10-06T03:30:01.831 INFO:teuthology.orchestra.run.smithi110.stdout: valid_lft forever preferred_lft forever 2023-10-06T03:30:01.831 INFO:teuthology.orchestra.run.smithi110.stdout:2: veth0@if10: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-10-06T03:30:01.831 INFO:teuthology.orchestra.run.smithi110.stdout: link/ether 7a:24:00:77:4b:7c brd ff:ff:ff:ff:ff:ff link-netnsid 0 2023-10-06T03:30:01.831 INFO:teuthology.orchestra.run.smithi110.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2023-10-06T03:30:01.831 INFO:teuthology.orchestra.run.smithi110.stdout: valid_lft forever preferred_lft forever 2023-10-06T03:30:01.831 INFO:teuthology.orchestra.run.smithi110.stdout: inet6 fe80::7824:ff:fe77:4b7c/64 scope link 2023-10-06T03:30:01.831 INFO:teuthology.orchestra.run.smithi110.stdout: valid_lft forever preferred_lft forever 2023-10-06T03:30:01.832 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-06T03:30:01.832 DEBUG:teuthology.orchestra.run.smithi110:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr 2023-10-06T03:30:01.844 INFO:teuthology.orchestra.run.smithi110.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-10-06T03:30:01.844 INFO:teuthology.orchestra.run.smithi110.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-10-06T03:30:01.844 INFO:teuthology.orchestra.run.smithi110.stdout: inet 127.0.0.1/8 scope host lo 2023-10-06T03:30:01.844 INFO:teuthology.orchestra.run.smithi110.stdout: valid_lft forever preferred_lft forever 2023-10-06T03:30:01.844 INFO:teuthology.orchestra.run.smithi110.stdout: inet6 ::1/128 scope host 2023-10-06T03:30:01.844 INFO:teuthology.orchestra.run.smithi110.stdout: valid_lft forever preferred_lft forever 2023-10-06T03:30:01.845 INFO:teuthology.orchestra.run.smithi110.stdout:2: veth0@if10: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-10-06T03:30:01.845 INFO:teuthology.orchestra.run.smithi110.stdout: link/ether 7a:24:00:77:4b:7c brd ff:ff:ff:ff:ff:ff link-netnsid 0 2023-10-06T03:30:01.845 INFO:teuthology.orchestra.run.smithi110.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2023-10-06T03:30:01.845 INFO:teuthology.orchestra.run.smithi110.stdout: valid_lft forever preferred_lft forever 2023-10-06T03:30:01.845 INFO:teuthology.orchestra.run.smithi110.stdout: inet6 fe80::7824:ff:fe77:4b7c/64 scope link 2023-10-06T03:30:01.845 INFO:teuthology.orchestra.run.smithi110.stdout: valid_lft forever preferred_lft forever 2023-10-06T03:30:01.846 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.2' with 192.168.0.2/16 2023-10-06T03:30:01.846 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-06T03:30:01.846 DEBUG:teuthology.orchestra.run.smithi110:> (cd / && exec bash -c ' 2023-10-06T03:30:01.847 DEBUG:teuthology.orchestra.run.smithi110:> set -e 2023-10-06T03:30:01.847 DEBUG:teuthology.orchestra.run.smithi110:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.2 type veth peer name brx.1 2023-10-06T03:30:01.847 DEBUG:teuthology.orchestra.run.smithi110:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip addr add 192.168.0.2/16 brd 192.168.255.255 dev veth0 2023-10-06T03:30:01.847 DEBUG:teuthology.orchestra.run.smithi110:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip link set veth0 up 2023-10-06T03:30:01.847 DEBUG:teuthology.orchestra.run.smithi110:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip link set lo up 2023-10-06T03:30:01.847 DEBUG:teuthology.orchestra.run.smithi110:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip route add default via 192.168.255.254 2023-10-06T03:30:01.847 DEBUG:teuthology.orchestra.run.smithi110:> ') 2023-10-06T03:30:01.946 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-06T03:30:01.947 DEBUG:teuthology.orchestra.run.smithi110:> (cd / && exec bash -c ' 2023-10-06T03:30:01.947 DEBUG:teuthology.orchestra.run.smithi110:> set -e 2023-10-06T03:30:01.947 DEBUG:teuthology.orchestra.run.smithi110:> sudo ip link set brx.1 up 2023-10-06T03:30:01.947 DEBUG:teuthology.orchestra.run.smithi110:> sudo ip link set dev brx.1 master ceph-brx 2023-10-06T03:30:01.947 DEBUG:teuthology.orchestra.run.smithi110:> ') 2023-10-06T03:30:01.974 INFO:tasks.cephfs.fuse_mount:Client client.2 config is {} 2023-10-06T03:30:01.974 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-06T03:30:01.974 DEBUG:teuthology.orchestra.run.smithi110:> mkdir -p -v /home/ubuntu/cephtest/mnt.2 2023-10-06T03:30:01.981 INFO:teuthology.orchestra.run.smithi110.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.2' 2023-10-06T03:30:01.981 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-06T03:30:01.981 DEBUG:teuthology.orchestra.run.smithi110:> chmod 0000 /home/ubuntu/cephtest/mnt.2 2023-10-06T03:30:02.029 DEBUG:teuthology.orchestra.run.smithi110:> sudo modprobe fuse 2023-10-06T03:30:02.045 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-10-06T03:30:02.045 DEBUG:teuthology.orchestra.run.smithi110:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-10-06T03:30:02.055 INFO:teuthology.orchestra.run.smithi110.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-10-06T03:30:02.056 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-10-06T03:30:02.057 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-06T03:30:02.057 DEBUG:teuthology.orchestra.run.smithi110:> ls /sys/fs/fuse/connections 2023-10-06T03:30:02.064 INFO:teuthology.orchestra.run.smithi110.stdout:52 2023-10-06T03:30:02.064 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [52] 2023-10-06T03:30:02.064 DEBUG:teuthology.orchestra.run.smithi110:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.2 --id 2) 2023-10-06T03:30:02.066 DEBUG:teuthology.orchestra.run.smithi110:> sudo modprobe fuse 2023-10-06T03:30:02.079 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-10-06T03:30:02.079 DEBUG:teuthology.orchestra.run.smithi110:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-10-06T03:30:02.090 INFO:teuthology.orchestra.run.smithi110.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-10-06T03:30:02.091 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-10-06T03:30:02.091 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-06T03:30:02.091 DEBUG:teuthology.orchestra.run.smithi110:> ls /sys/fs/fuse/connections 2023-10-06T03:30:02.146 INFO:teuthology.orchestra.run.smithi110.stdout:52 2023-10-06T03:30:02.147 INFO:tasks.cephfs.fuse_mount.ceph-fuse.2.smithi110.stderr:2023-10-06T03:30:02.141+0000 7f1450c39380 -1 init, newargv = 0x55d18d1a4620 newargc=15 2023-10-06T03:30:02.147 INFO:tasks.cephfs.fuse_mount.ceph-fuse.2.smithi110.stderr:ceph-fuse[40144]: starting ceph client 2023-10-06T03:30:02.155 INFO:tasks.cephfs.fuse_mount.ceph-fuse.2.smithi110.stderr:ceph-fuse[40144]: starting fuse 2023-10-06T03:30:03.248 DEBUG:teuthology.orchestra.run.smithi110:> sudo modprobe fuse 2023-10-06T03:30:03.267 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-10-06T03:30:03.267 DEBUG:teuthology.orchestra.run.smithi110:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-10-06T03:30:03.325 INFO:teuthology.orchestra.run.smithi110.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-10-06T03:30:03.325 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-10-06T03:30:03.326 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-06T03:30:03.326 DEBUG:teuthology.orchestra.run.smithi110:> ls /sys/fs/fuse/connections 2023-10-06T03:30:03.333 INFO:teuthology.orchestra.run.smithi110.stdout:52 2023-10-06T03:30:03.334 INFO:teuthology.orchestra.run.smithi110.stdout:53 2023-10-06T03:30:03.334 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [52, 53] 2023-10-06T03:30:03.334 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2023-10-06T03:30:03.334 DEBUG:teuthology.orchestra.run.smithi110:> import glob 2023-10-06T03:30:03.334 DEBUG:teuthology.orchestra.run.smithi110:> import re 2023-10-06T03:30:03.335 DEBUG:teuthology.orchestra.run.smithi110:> import os 2023-10-06T03:30:03.335 DEBUG:teuthology.orchestra.run.smithi110:> import subprocess 2023-10-06T03:30:03.335 DEBUG:teuthology.orchestra.run.smithi110:> 2023-10-06T03:30:03.335 DEBUG:teuthology.orchestra.run.smithi110:> def _find_admin_socket(client_name): 2023-10-06T03:30:03.335 DEBUG:teuthology.orchestra.run.smithi110:> asok_path = "/var/run/ceph/ceph-client.2.*.asok" 2023-10-06T03:30:03.335 DEBUG:teuthology.orchestra.run.smithi110:> files = glob.glob(asok_path) 2023-10-06T03:30:03.335 DEBUG:teuthology.orchestra.run.smithi110:> mountpoint = "/home/ubuntu/cephtest/mnt.2" 2023-10-06T03:30:03.335 DEBUG:teuthology.orchestra.run.smithi110:> 2023-10-06T03:30:03.336 DEBUG:teuthology.orchestra.run.smithi110:> # Given a non-glob path, it better be there 2023-10-06T03:30:03.336 DEBUG:teuthology.orchestra.run.smithi110:> if "*" not in asok_path: 2023-10-06T03:30:03.336 DEBUG:teuthology.orchestra.run.smithi110:> assert(len(files) == 1) 2023-10-06T03:30:03.336 DEBUG:teuthology.orchestra.run.smithi110:> return files[0] 2023-10-06T03:30:03.336 DEBUG:teuthology.orchestra.run.smithi110:> 2023-10-06T03:30:03.336 DEBUG:teuthology.orchestra.run.smithi110:> for f in files: 2023-10-06T03:30:03.336 DEBUG:teuthology.orchestra.run.smithi110:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2023-10-06T03:30:03.336 DEBUG:teuthology.orchestra.run.smithi110:> if os.path.exists("/proc/{0}".format(pid)): 2023-10-06T03:30:03.336 DEBUG:teuthology.orchestra.run.smithi110:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2023-10-06T03:30:03.336 DEBUG:teuthology.orchestra.run.smithi110:> contents = proc_f.read() 2023-10-06T03:30:03.337 DEBUG:teuthology.orchestra.run.smithi110:> if mountpoint in contents: 2023-10-06T03:30:03.337 DEBUG:teuthology.orchestra.run.smithi110:> return f 2023-10-06T03:30:03.337 DEBUG:teuthology.orchestra.run.smithi110:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2023-10-06T03:30:03.337 DEBUG:teuthology.orchestra.run.smithi110:> 2023-10-06T03:30:03.337 DEBUG:teuthology.orchestra.run.smithi110:> print(_find_admin_socket("client.2")) 2023-10-06T03:30:03.337 DEBUG:teuthology.orchestra.run.smithi110:> ' 2023-10-06T03:30:03.419 INFO:teuthology.orchestra.run.smithi110.stdout:/var/run/ceph/ceph-client.2.40144.asok 2023-10-06T03:30:03.604 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.2.40144.asok 2023-10-06T03:30:03.604 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-06T03:30:03.605 DEBUG:teuthology.orchestra.run.smithi110:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.2.40144.asok status 2023-10-06T03:30:03.752 INFO:teuthology.orchestra.run.smithi110.stdout:{ 2023-10-06T03:30:03.752 INFO:teuthology.orchestra.run.smithi110.stdout: "metadata": { 2023-10-06T03:30:03.752 INFO:teuthology.orchestra.run.smithi110.stdout: "ceph_sha1": "736ac5ba72cb47af7d6f6666b1e100f7e20a20e2", 2023-10-06T03:30:03.753 INFO:teuthology.orchestra.run.smithi110.stdout: "ceph_version": "ceph version 17.2.6-1296-g736ac5ba (736ac5ba72cb47af7d6f6666b1e100f7e20a20e2) quincy (stable)", 2023-10-06T03:30:03.753 INFO:teuthology.orchestra.run.smithi110.stdout: "entity_id": "2", 2023-10-06T03:30:03.753 INFO:teuthology.orchestra.run.smithi110.stdout: "hostname": "smithi110", 2023-10-06T03:30:03.753 INFO:teuthology.orchestra.run.smithi110.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.2", 2023-10-06T03:30:03.753 INFO:teuthology.orchestra.run.smithi110.stdout: "pid": "40144", 2023-10-06T03:30:03.753 INFO:teuthology.orchestra.run.smithi110.stdout: "root": "/" 2023-10-06T03:30:03.753 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2023-10-06T03:30:03.753 INFO:teuthology.orchestra.run.smithi110.stdout: "dentry_count": 0, 2023-10-06T03:30:03.753 INFO:teuthology.orchestra.run.smithi110.stdout: "dentry_pinned_count": 0, 2023-10-06T03:30:03.753 INFO:teuthology.orchestra.run.smithi110.stdout: "id": 4836, 2023-10-06T03:30:03.753 INFO:teuthology.orchestra.run.smithi110.stdout: "inst": { 2023-10-06T03:30:03.753 INFO:teuthology.orchestra.run.smithi110.stdout: "name": { 2023-10-06T03:30:03.754 INFO:teuthology.orchestra.run.smithi110.stdout: "type": "client", 2023-10-06T03:30:03.754 INFO:teuthology.orchestra.run.smithi110.stdout: "num": 4836 2023-10-06T03:30:03.754 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2023-10-06T03:30:03.754 INFO:teuthology.orchestra.run.smithi110.stdout: "addr": { 2023-10-06T03:30:03.754 INFO:teuthology.orchestra.run.smithi110.stdout: "type": "v1", 2023-10-06T03:30:03.754 INFO:teuthology.orchestra.run.smithi110.stdout: "addr": "192.168.0.2:0", 2023-10-06T03:30:03.754 INFO:teuthology.orchestra.run.smithi110.stdout: "nonce": 2269791849 2023-10-06T03:30:03.754 INFO:teuthology.orchestra.run.smithi110.stdout: } 2023-10-06T03:30:03.754 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2023-10-06T03:30:03.754 INFO:teuthology.orchestra.run.smithi110.stdout: "addr": { 2023-10-06T03:30:03.754 INFO:teuthology.orchestra.run.smithi110.stdout: "type": "v1", 2023-10-06T03:30:03.755 INFO:teuthology.orchestra.run.smithi110.stdout: "addr": "192.168.0.2:0", 2023-10-06T03:30:03.755 INFO:teuthology.orchestra.run.smithi110.stdout: "nonce": 2269791849 2023-10-06T03:30:03.755 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2023-10-06T03:30:03.755 INFO:teuthology.orchestra.run.smithi110.stdout: "inst_str": "client.4836 192.168.0.2:0/2269791849", 2023-10-06T03:30:03.755 INFO:teuthology.orchestra.run.smithi110.stdout: "addr_str": "192.168.0.2:0/2269791849", 2023-10-06T03:30:03.755 INFO:teuthology.orchestra.run.smithi110.stdout: "inode_count": 1, 2023-10-06T03:30:03.755 INFO:teuthology.orchestra.run.smithi110.stdout: "mds_epoch": 8, 2023-10-06T03:30:03.755 INFO:teuthology.orchestra.run.smithi110.stdout: "osd_epoch": 28, 2023-10-06T03:30:03.755 INFO:teuthology.orchestra.run.smithi110.stdout: "osd_epoch_barrier": 0, 2023-10-06T03:30:03.755 INFO:teuthology.orchestra.run.smithi110.stdout: "blocklisted": false, 2023-10-06T03:30:03.755 INFO:teuthology.orchestra.run.smithi110.stdout: "fs_name": "cephfs" 2023-10-06T03:30:03.755 INFO:teuthology.orchestra.run.smithi110.stdout:} 2023-10-06T03:30:03.762 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-10-06T03:30:03.864 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:03.861+0000 7f4384f42700 1 Processor -- start 2023-10-06T03:30:03.864 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:03.861+0000 7f4384f42700 1 -- start start 2023-10-06T03:30:03.865 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:03.861+0000 7f4384f42700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f438015d420 0x7f43800c7b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:03.865 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:03.861+0000 7f4384f42700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4380069d70 0x7f43800ca270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:03.865 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:03.861+0000 7f4384f42700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f438015d7f0 0x7f43800ca7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:03.865 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:03.861+0000 7f4384f42700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f438006aef0 con 0x7f438015d7f0 2023-10-06T03:30:03.866 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:03.861+0000 7f4384f42700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f438006a2e0 con 0x7f438015d420 2023-10-06T03:30:03.866 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:03.861+0000 7f4384f42700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f438006a5c0 con 0x7f4380069d70 2023-10-06T03:30:03.866 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:03.861+0000 7f437ed9d700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f438015d7f0 0x7f43800ca7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:03.866 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:03.861+0000 7f437ed9d700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f438015d7f0 0x7f43800ca7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:40406/0 (socket says 172.21.15.110:40406) 2023-10-06T03:30:03.866 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:03.861+0000 7f437ed9d700 1 -- 172.21.15.110:0/3496106519 learned_addr learned my addr 172.21.15.110:0/3496106519 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:30:03.866 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:03.861+0000 7f437e59c700 1 --2- 172.21.15.110:0/3496106519 >> v2:172.21.15.192:3300/0 conn(0x7f438015d420 0x7f43800c7b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:03.866 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:03.861+0000 7f437dd9b700 1 --2- 172.21.15.110:0/3496106519 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4380069d70 0x7f43800ca270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:03.867 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:03.861+0000 7f437ed9d700 1 -- 172.21.15.110:0/3496106519 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4380069d70 msgr2=0x7f43800ca270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:30:03.867 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:03.861+0000 7f437ed9d700 1 --2- 172.21.15.110:0/3496106519 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4380069d70 0x7f43800ca270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:03.867 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:03.861+0000 7f437ed9d700 1 -- 172.21.15.110:0/3496106519 >> v2:172.21.15.192:3300/0 conn(0x7f438015d420 msgr2=0x7f43800c7b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:30:03.867 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:03.861+0000 7f437ed9d700 1 --2- 172.21.15.110:0/3496106519 >> v2:172.21.15.192:3300/0 conn(0x7f438015d420 0x7f43800c7b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:03.867 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:03.861+0000 7f437ed9d700 1 -- 172.21.15.110:0/3496106519 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f438006a8a0 con 0x7f438015d7f0 2023-10-06T03:30:03.867 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:03.861+0000 7f437ed9d700 1 --2- 172.21.15.110:0/3496106519 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f438015d7f0 0x7f43800ca7b0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f43800640e0 tx=0x7f4370009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=693e27efba68cded server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:30:03.868 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:03.861+0000 7f437d59a700 1 -- 172.21.15.110:0/3496106519 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f437000b640 con 0x7f438015d7f0 2023-10-06T03:30:03.868 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:03.861+0000 7f437d59a700 1 -- 172.21.15.110:0/3496106519 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f437000b7a0 con 0x7f438015d7f0 2023-10-06T03:30:03.868 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:03.861+0000 7f437d59a700 1 -- 172.21.15.110:0/3496106519 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f437000bb60 con 0x7f438015d7f0 2023-10-06T03:30:03.869 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:03.865+0000 7f4384f42700 1 -- 172.21.15.110:0/3496106519 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f438015d7f0 msgr2=0x7f43800ca7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:03.869 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:03.865+0000 7f4384f42700 1 --2- 172.21.15.110:0/3496106519 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f438015d7f0 0x7f43800ca7b0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f43800640e0 tx=0x7f4370009250 comp rx=0 tx=0).stop 2023-10-06T03:30:03.869 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:03.865+0000 7f4384f42700 1 -- 172.21.15.110:0/3496106519 shutdown_connections 2023-10-06T03:30:03.869 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:03.865+0000 7f4384f42700 1 --2- 172.21.15.110:0/3496106519 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f438015d7f0 0x7f43800ca7b0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:03.869 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:03.865+0000 7f4384f42700 1 --2- 172.21.15.110:0/3496106519 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4380069d70 0x7f43800ca270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:03.869 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:03.865+0000 7f4384f42700 1 --2- 172.21.15.110:0/3496106519 >> v2:172.21.15.192:3300/0 conn(0x7f438015d420 0x7f43800c7b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:03.869 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:03.865+0000 7f4384f42700 1 -- 172.21.15.110:0/3496106519 >> 172.21.15.110:0/3496106519 conn(0x7f43800cbd00 msgr2=0x7f43800cc100 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:30:03.869 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:03.865+0000 7f4384f42700 1 -- 172.21.15.110:0/3496106519 shutdown_connections 2023-10-06T03:30:03.869 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:03.865+0000 7f4384f42700 1 -- 172.21.15.110:0/3496106519 wait complete. 2023-10-06T03:30:03.871 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:03.865+0000 7f4384f42700 1 Processor -- start 2023-10-06T03:30:03.871 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:03.865+0000 7f4384f42700 1 -- start start 2023-10-06T03:30:03.872 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:03.869+0000 7f4384f42700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4380069d70 0x7f43801580a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:03.872 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:03.869+0000 7f4384f42700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f438015d420 0x7f43801585e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:03.872 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:03.869+0000 7f4384f42700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f438015d7f0 0x7f43800c4c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:03.872 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:03.869+0000 7f4384f42700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f438015e6c0 con 0x7f438015d7f0 2023-10-06T03:30:03.872 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:03.869+0000 7f4384f42700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43800c4050 con 0x7f438015d420 2023-10-06T03:30:03.873 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:03.869+0000 7f4384f42700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f438015ebb0 con 0x7f4380069d70 2023-10-06T03:30:03.873 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:03.869+0000 7f437ed9d700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f438015d7f0 0x7f43800c4c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:03.873 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:03.869+0000 7f437dd9b700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f438015d420 0x7f43801585e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:03.873 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:03.869+0000 7f437ed9d700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f438015d7f0 0x7f43800c4c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:40422/0 (socket says 172.21.15.110:40422) 2023-10-06T03:30:03.873 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:03.869+0000 7f437ed9d700 1 -- 172.21.15.110:0/3066421537 learned_addr learned my addr 172.21.15.110:0/3066421537 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:30:03.873 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:03.869+0000 7f437e59c700 1 --2- 172.21.15.110:0/3066421537 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4380069d70 0x7f43801580a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:03.874 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:03.869+0000 7f437ed9d700 1 -- 172.21.15.110:0/3066421537 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4380069d70 msgr2=0x7f43801580a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:03.874 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:03.869+0000 7f437ed9d700 1 --2- 172.21.15.110:0/3066421537 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4380069d70 0x7f43801580a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:03.874 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:03.869+0000 7f437ed9d700 1 -- 172.21.15.110:0/3066421537 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f438015d420 msgr2=0x7f43801585e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:03.874 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:03.869+0000 7f437ed9d700 1 --2- 172.21.15.110:0/3066421537 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f438015d420 0x7f43801585e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:03.874 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:03.869+0000 7f437ed9d700 1 -- 172.21.15.110:0/3066421537 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43700094f0 con 0x7f438015d7f0 2023-10-06T03:30:03.874 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:03.869+0000 7f437ed9d700 1 --2- 172.21.15.110:0/3066421537 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f438015d7f0 0x7f43800c4c00 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f438006da30 tx=0x7f4370017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:30:03.875 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:03.869+0000 7f43767fc700 1 -- 172.21.15.110:0/3066421537 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f437000b910 con 0x7f438015d7f0 2023-10-06T03:30:03.875 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:03.869+0000 7f43767fc700 1 -- 172.21.15.110:0/3066421537 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f437000ba90 con 0x7f438015d7f0 2023-10-06T03:30:03.875 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:03.869+0000 7f4384f42700 1 -- 172.21.15.110:0/3066421537 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4380158b80 con 0x7f438015d7f0 2023-10-06T03:30:03.875 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:03.869+0000 7f43767fc700 1 -- 172.21.15.110:0/3066421537 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4370025670 con 0x7f438015d7f0 2023-10-06T03:30:03.875 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:03.869+0000 7f4384f42700 1 -- 172.21.15.110:0/3066421537 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f438015e8d0 con 0x7f438015d7f0 2023-10-06T03:30:03.875 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:03.869+0000 7f4384f42700 1 -- 172.21.15.110:0/3066421537 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f438006ff60 con 0x7f438015d7f0 2023-10-06T03:30:03.876 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:03.873+0000 7f43767fc700 1 -- 172.21.15.110:0/3066421537 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7f4370023070 con 0x7f438015d7f0 2023-10-06T03:30:03.876 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:03.873+0000 7f43767fc700 1 --2- 172.21.15.110:0/3066421537 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f4360060550 0x7f4360062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:03.876 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:03.873+0000 7f43767fc700 1 -- 172.21.15.110:0/3066421537 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6809+0+0 (secure 0 0 0) 0x7f437007e010 con 0x7f438015d7f0 2023-10-06T03:30:03.876 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:03.873+0000 7f437e59c700 1 --2- 172.21.15.110:0/3066421537 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f4360060550 0x7f4360062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:03.877 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:03.873+0000 7f437e59c700 1 --2- 172.21.15.110:0/3066421537 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f4360060550 0x7f4360062a00 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f4368009db0 tx=0x7f43680091a0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:30:03.885 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:03.881+0000 7f43767fc700 1 -- 172.21.15.110:0/3066421537 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f437004e260 con 0x7f438015d7f0 2023-10-06T03:30:04.078 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:04.073+0000 7f4384f42700 1 -- 172.21.15.110:0/3066421537 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f438006ab80 con 0x7f438015d7f0 2023-10-06T03:30:04.078 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:04.073+0000 7f43767fc700 1 -- 172.21.15.110:0/3066421537 <== mon.0 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v8) v1 ==== 53+0+88 (secure 0 0 0) 0x7f4370051880 con 0x7f438015d7f0 2023-10-06T03:30:04.078 INFO:teuthology.orchestra.run.smithi110.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2023-10-06T03:30:04.082 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:04.077+0000 7f4384f42700 1 -- 172.21.15.110:0/3066421537 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f4360060550 msgr2=0x7f4360062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:04.082 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:04.077+0000 7f4384f42700 1 --2- 172.21.15.110:0/3066421537 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f4360060550 0x7f4360062a00 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f4368009db0 tx=0x7f43680091a0 comp rx=0 tx=0).stop 2023-10-06T03:30:04.083 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:04.077+0000 7f4384f42700 1 -- 172.21.15.110:0/3066421537 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f438015d7f0 msgr2=0x7f43800c4c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:04.083 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:04.077+0000 7f4384f42700 1 --2- 172.21.15.110:0/3066421537 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f438015d7f0 0x7f43800c4c00 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f438006da30 tx=0x7f4370017000 comp rx=0 tx=0).stop 2023-10-06T03:30:04.083 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:04.077+0000 7f4384f42700 1 -- 172.21.15.110:0/3066421537 shutdown_connections 2023-10-06T03:30:04.083 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:04.077+0000 7f4384f42700 1 --2- 172.21.15.110:0/3066421537 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f4360060550 0x7f4360062a00 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:04.083 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:04.077+0000 7f4384f42700 1 --2- 172.21.15.110:0/3066421537 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f438015d7f0 0x7f43800c4c00 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:04.083 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:04.077+0000 7f4384f42700 1 --2- 172.21.15.110:0/3066421537 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f438015d420 0x7f43801585e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:04.083 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:04.077+0000 7f4384f42700 1 --2- 172.21.15.110:0/3066421537 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4380069d70 0x7f43801580a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:04.083 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:04.077+0000 7f4384f42700 1 -- 172.21.15.110:0/3066421537 >> 172.21.15.110:0/3066421537 conn(0x7f43800cbd00 msgr2=0x7f43800c1580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:30:04.083 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:04.077+0000 7f4384f42700 1 -- 172.21.15.110:0/3066421537 shutdown_connections 2023-10-06T03:30:04.083 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:04.077+0000 7f4384f42700 1 -- 172.21.15.110:0/3066421537 wait complete. 2023-10-06T03:30:04.098 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2023-10-06T03:30:04.099 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-10-06T03:30:04.099 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi192.front.sepia.ceph.com 2023-10-06T03:30:04.099 INFO:tasks.cephfs.mount:self.client.name = client.1 2023-10-06T03:30:04.099 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2023-10-06T03:30:04.099 INFO:tasks.cephfs.mount:self.cephfs_name = None 2023-10-06T03:30:04.099 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2023-10-06T03:30:04.099 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-10-06T03:30:04.099 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.1' 2023-10-06T03:30:04.099 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-06T03:30:04.099 DEBUG:teuthology.orchestra.run.smithi192:> ip addr 2023-10-06T03:30:04.107 INFO:teuthology.orchestra.run.smithi192.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-10-06T03:30:04.107 INFO:teuthology.orchestra.run.smithi192.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-10-06T03:30:04.107 INFO:teuthology.orchestra.run.smithi192.stdout: inet 127.0.0.1/8 scope host lo 2023-10-06T03:30:04.107 INFO:teuthology.orchestra.run.smithi192.stdout: valid_lft forever preferred_lft forever 2023-10-06T03:30:04.107 INFO:teuthology.orchestra.run.smithi192.stdout: inet6 ::1/128 scope host 2023-10-06T03:30:04.107 INFO:teuthology.orchestra.run.smithi192.stdout: valid_lft forever preferred_lft forever 2023-10-06T03:30:04.107 INFO:teuthology.orchestra.run.smithi192.stdout:2: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2023-10-06T03:30:04.107 INFO:teuthology.orchestra.run.smithi192.stdout: link/ether 0c:c4:7a:d9:38:4e brd ff:ff:ff:ff:ff:ff 2023-10-06T03:30:04.108 INFO:teuthology.orchestra.run.smithi192.stdout:3: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000 2023-10-06T03:30:04.108 INFO:teuthology.orchestra.run.smithi192.stdout: link/ether 0c:c4:7a:d9:38:4f brd ff:ff:ff:ff:ff:ff 2023-10-06T03:30:04.108 INFO:teuthology.orchestra.run.smithi192.stdout:4: enp3s0f0: mtu 1500 qdisc noop state DOWN group default qlen 1000 2023-10-06T03:30:04.108 INFO:teuthology.orchestra.run.smithi192.stdout: link/ether 0c:c4:7a:88:80:48 brd ff:ff:ff:ff:ff:ff 2023-10-06T03:30:04.108 INFO:teuthology.orchestra.run.smithi192.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2023-10-06T03:30:04.108 INFO:teuthology.orchestra.run.smithi192.stdout: link/ether 0c:c4:7a:88:80:49 brd ff:ff:ff:ff:ff:ff 2023-10-06T03:30:04.108 INFO:teuthology.orchestra.run.smithi192.stdout: inet 172.21.15.192/20 brd 172.21.15.255 scope global dynamic enp3s0f1 2023-10-06T03:30:04.108 INFO:teuthology.orchestra.run.smithi192.stdout: valid_lft 42578sec preferred_lft 42578sec 2023-10-06T03:30:04.108 INFO:teuthology.orchestra.run.smithi192.stdout: inet6 fe80::ec4:7aff:fe88:8049/64 scope link 2023-10-06T03:30:04.108 INFO:teuthology.orchestra.run.smithi192.stdout: valid_lft forever preferred_lft forever 2023-10-06T03:30:04.108 INFO:teuthology.orchestra.run.smithi192.stdout:6: docker0: mtu 1500 qdisc noqueue state DOWN group default 2023-10-06T03:30:04.108 INFO:teuthology.orchestra.run.smithi192.stdout: link/ether 02:42:fb:a6:b4:5a brd ff:ff:ff:ff:ff:ff 2023-10-06T03:30:04.109 INFO:teuthology.orchestra.run.smithi192.stdout: inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 2023-10-06T03:30:04.109 INFO:teuthology.orchestra.run.smithi192.stdout: valid_lft forever preferred_lft forever 2023-10-06T03:30:04.109 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2023-10-06T03:30:04.110 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-06T03:30:04.110 DEBUG:teuthology.orchestra.run.smithi192:> (cd / && exec bash -c ' 2023-10-06T03:30:04.110 DEBUG:teuthology.orchestra.run.smithi192:> set -e 2023-10-06T03:30:04.110 DEBUG:teuthology.orchestra.run.smithi192:> sudo ip link add name ceph-brx type bridge 2023-10-06T03:30:04.110 DEBUG:teuthology.orchestra.run.smithi192:> sudo ip addr flush dev ceph-brx 2023-10-06T03:30:04.110 DEBUG:teuthology.orchestra.run.smithi192:> sudo ip link set ceph-brx up 2023-10-06T03:30:04.110 DEBUG:teuthology.orchestra.run.smithi192:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2023-10-06T03:30:04.110 DEBUG:teuthology.orchestra.run.smithi192:> ') 2023-10-06T03:30:04.203 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-06T03:30:04.203 DEBUG:teuthology.orchestra.run.smithi192:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-10-06T03:30:04.216 INFO:teuthology.orchestra.run.smithi192.stdout:1 2023-10-06T03:30:04.217 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-06T03:30:04.218 DEBUG:teuthology.orchestra.run.smithi192:> route 2023-10-06T03:30:04.239 INFO:teuthology.orchestra.run.smithi192.stdout:Kernel IP routing table 2023-10-06T03:30:04.239 INFO:teuthology.orchestra.run.smithi192.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-10-06T03:30:04.239 INFO:teuthology.orchestra.run.smithi192.stdout:default _gateway 0.0.0.0 UG 0 0 0 enp3s0f1 2023-10-06T03:30:04.240 INFO:teuthology.orchestra.run.smithi192.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2023-10-06T03:30:04.240 INFO:teuthology.orchestra.run.smithi192.stdout:172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 2023-10-06T03:30:04.240 INFO:teuthology.orchestra.run.smithi192.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 0 0 0 enp3s0f1 2023-10-06T03:30:04.240 INFO:teuthology.orchestra.run.smithi192.stdout:_gateway 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2023-10-06T03:30:04.240 INFO:teuthology.orchestra.run.smithi192.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-10-06T03:30:04.240 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-06T03:30:04.241 DEBUG:teuthology.orchestra.run.smithi192:> (cd / && exec bash -c ' 2023-10-06T03:30:04.241 DEBUG:teuthology.orchestra.run.smithi192:> set -e 2023-10-06T03:30:04.241 DEBUG:teuthology.orchestra.run.smithi192:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2023-10-06T03:30:04.241 DEBUG:teuthology.orchestra.run.smithi192:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2023-10-06T03:30:04.241 DEBUG:teuthology.orchestra.run.smithi192:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2023-10-06T03:30:04.241 DEBUG:teuthology.orchestra.run.smithi192:> ') 2023-10-06T03:30:04.278 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-06T03:30:04.278 DEBUG:teuthology.orchestra.run.smithi192:> ip netns list 2023-10-06T03:30:04.329 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-06T03:30:04.329 DEBUG:teuthology.orchestra.run.smithi192:> ip netns list-id 2023-10-06T03:30:04.337 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-06T03:30:04.338 DEBUG:teuthology.orchestra.run.smithi192:> (cd / && exec bash -c ' 2023-10-06T03:30:04.338 DEBUG:teuthology.orchestra.run.smithi192:> set -e 2023-10-06T03:30:04.338 DEBUG:teuthology.orchestra.run.smithi192:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.1 2023-10-06T03:30:04.338 DEBUG:teuthology.orchestra.run.smithi192:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.1 0 2023-10-06T03:30:04.338 DEBUG:teuthology.orchestra.run.smithi192:> ') 2023-10-06T03:30:04.367 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.1' with 192.168.0.1/16 2023-10-06T03:30:04.367 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-06T03:30:04.367 DEBUG:teuthology.orchestra.run.smithi192:> (cd / && exec bash -c ' 2023-10-06T03:30:04.367 DEBUG:teuthology.orchestra.run.smithi192:> set -e 2023-10-06T03:30:04.367 DEBUG:teuthology.orchestra.run.smithi192:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.1 type veth peer name brx.0 2023-10-06T03:30:04.367 DEBUG:teuthology.orchestra.run.smithi192:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2023-10-06T03:30:04.367 DEBUG:teuthology.orchestra.run.smithi192:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set veth0 up 2023-10-06T03:30:04.368 DEBUG:teuthology.orchestra.run.smithi192:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set lo up 2023-10-06T03:30:04.368 DEBUG:teuthology.orchestra.run.smithi192:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip route add default via 192.168.255.254 2023-10-06T03:30:04.368 DEBUG:teuthology.orchestra.run.smithi192:> ') 2023-10-06T03:30:04.483 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-06T03:30:04.484 DEBUG:teuthology.orchestra.run.smithi192:> (cd / && exec bash -c ' 2023-10-06T03:30:04.484 DEBUG:teuthology.orchestra.run.smithi192:> set -e 2023-10-06T03:30:04.484 DEBUG:teuthology.orchestra.run.smithi192:> sudo ip link set brx.0 up 2023-10-06T03:30:04.484 DEBUG:teuthology.orchestra.run.smithi192:> sudo ip link set dev brx.0 master ceph-brx 2023-10-06T03:30:04.484 DEBUG:teuthology.orchestra.run.smithi192:> ') 2023-10-06T03:30:04.507 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {} 2023-10-06T03:30:04.507 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-06T03:30:04.507 DEBUG:teuthology.orchestra.run.smithi192:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2023-10-06T03:30:04.512 INFO:teuthology.orchestra.run.smithi192.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2023-10-06T03:30:04.512 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-06T03:30:04.512 DEBUG:teuthology.orchestra.run.smithi192:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2023-10-06T03:30:04.516 DEBUG:teuthology.orchestra.run.smithi192:> sudo modprobe fuse 2023-10-06T03:30:04.528 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-10-06T03:30:04.528 DEBUG:teuthology.orchestra.run.smithi192:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-10-06T03:30:04.538 INFO:teuthology.orchestra.run.smithi192.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-10-06T03:30:04.539 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-10-06T03:30:04.539 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-06T03:30:04.539 DEBUG:teuthology.orchestra.run.smithi192:> ls /sys/fs/fuse/connections 2023-10-06T03:30:04.546 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2023-10-06T03:30:04.546 DEBUG:teuthology.orchestra.run.smithi192:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.1 --id 1) 2023-10-06T03:30:04.596 DEBUG:teuthology.orchestra.run.smithi192:> sudo modprobe fuse 2023-10-06T03:30:04.612 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-10-06T03:30:04.613 DEBUG:teuthology.orchestra.run.smithi192:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-10-06T03:30:04.623 INFO:teuthology.orchestra.run.smithi192.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-10-06T03:30:04.623 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-10-06T03:30:04.624 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-06T03:30:04.624 DEBUG:teuthology.orchestra.run.smithi192:> ls /sys/fs/fuse/connections 2023-10-06T03:30:04.684 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi192.stderr:2023-10-06T03:30:04.681+0000 7f99ef86a380 -1 init, newargv = 0x56005ad9dfb0 newargc=15 2023-10-06T03:30:04.684 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi192.stderr:ceph-fuse[39249]: starting ceph client 2023-10-06T03:30:04.692 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi192.stderr:ceph-fuse[39249]: starting fuse 2023-10-06T03:30:05.730 DEBUG:teuthology.orchestra.run.smithi192:> sudo modprobe fuse 2023-10-06T03:30:05.749 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-10-06T03:30:05.749 DEBUG:teuthology.orchestra.run.smithi192:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-10-06T03:30:05.810 INFO:teuthology.orchestra.run.smithi192.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-10-06T03:30:05.810 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-10-06T03:30:05.811 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-06T03:30:05.811 DEBUG:teuthology.orchestra.run.smithi192:> ls /sys/fs/fuse/connections 2023-10-06T03:30:05.818 INFO:teuthology.orchestra.run.smithi192.stdout:52 2023-10-06T03:30:05.818 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [52] 2023-10-06T03:30:05.818 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2023-10-06T03:30:05.818 DEBUG:teuthology.orchestra.run.smithi192:> import glob 2023-10-06T03:30:05.818 DEBUG:teuthology.orchestra.run.smithi192:> import re 2023-10-06T03:30:05.818 DEBUG:teuthology.orchestra.run.smithi192:> import os 2023-10-06T03:30:05.818 DEBUG:teuthology.orchestra.run.smithi192:> import subprocess 2023-10-06T03:30:05.818 DEBUG:teuthology.orchestra.run.smithi192:> 2023-10-06T03:30:05.819 DEBUG:teuthology.orchestra.run.smithi192:> def _find_admin_socket(client_name): 2023-10-06T03:30:05.819 DEBUG:teuthology.orchestra.run.smithi192:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2023-10-06T03:30:05.819 DEBUG:teuthology.orchestra.run.smithi192:> files = glob.glob(asok_path) 2023-10-06T03:30:05.819 DEBUG:teuthology.orchestra.run.smithi192:> mountpoint = "/home/ubuntu/cephtest/mnt.1" 2023-10-06T03:30:05.819 DEBUG:teuthology.orchestra.run.smithi192:> 2023-10-06T03:30:05.819 DEBUG:teuthology.orchestra.run.smithi192:> # Given a non-glob path, it better be there 2023-10-06T03:30:05.819 DEBUG:teuthology.orchestra.run.smithi192:> if "*" not in asok_path: 2023-10-06T03:30:05.819 DEBUG:teuthology.orchestra.run.smithi192:> assert(len(files) == 1) 2023-10-06T03:30:05.819 DEBUG:teuthology.orchestra.run.smithi192:> return files[0] 2023-10-06T03:30:05.819 DEBUG:teuthology.orchestra.run.smithi192:> 2023-10-06T03:30:05.819 DEBUG:teuthology.orchestra.run.smithi192:> for f in files: 2023-10-06T03:30:05.819 DEBUG:teuthology.orchestra.run.smithi192:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2023-10-06T03:30:05.820 DEBUG:teuthology.orchestra.run.smithi192:> if os.path.exists("/proc/{0}".format(pid)): 2023-10-06T03:30:05.820 DEBUG:teuthology.orchestra.run.smithi192:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2023-10-06T03:30:05.820 DEBUG:teuthology.orchestra.run.smithi192:> contents = proc_f.read() 2023-10-06T03:30:05.820 DEBUG:teuthology.orchestra.run.smithi192:> if mountpoint in contents: 2023-10-06T03:30:05.820 DEBUG:teuthology.orchestra.run.smithi192:> return f 2023-10-06T03:30:05.820 DEBUG:teuthology.orchestra.run.smithi192:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2023-10-06T03:30:05.820 DEBUG:teuthology.orchestra.run.smithi192:> 2023-10-06T03:30:05.820 DEBUG:teuthology.orchestra.run.smithi192:> print(_find_admin_socket("client.1")) 2023-10-06T03:30:05.820 DEBUG:teuthology.orchestra.run.smithi192:> ' 2023-10-06T03:30:05.901 INFO:teuthology.orchestra.run.smithi192.stdout:/var/run/ceph/ceph-client.1.39249.asok 2023-10-06T03:30:06.085 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.39249.asok 2023-10-06T03:30:06.085 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-06T03:30:06.085 DEBUG:teuthology.orchestra.run.smithi192:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.39249.asok status 2023-10-06T03:30:06.222 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2023-10-06T03:30:06.222 INFO:teuthology.orchestra.run.smithi192.stdout: "metadata": { 2023-10-06T03:30:06.222 INFO:teuthology.orchestra.run.smithi192.stdout: "ceph_sha1": "736ac5ba72cb47af7d6f6666b1e100f7e20a20e2", 2023-10-06T03:30:06.222 INFO:teuthology.orchestra.run.smithi192.stdout: "ceph_version": "ceph version 17.2.6-1296-g736ac5ba (736ac5ba72cb47af7d6f6666b1e100f7e20a20e2) quincy (stable)", 2023-10-06T03:30:06.222 INFO:teuthology.orchestra.run.smithi192.stdout: "entity_id": "1", 2023-10-06T03:30:06.222 INFO:teuthology.orchestra.run.smithi192.stdout: "hostname": "smithi192", 2023-10-06T03:30:06.222 INFO:teuthology.orchestra.run.smithi192.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2023-10-06T03:30:06.222 INFO:teuthology.orchestra.run.smithi192.stdout: "pid": "39249", 2023-10-06T03:30:06.223 INFO:teuthology.orchestra.run.smithi192.stdout: "root": "/" 2023-10-06T03:30:06.223 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2023-10-06T03:30:06.223 INFO:teuthology.orchestra.run.smithi192.stdout: "dentry_count": 0, 2023-10-06T03:30:06.223 INFO:teuthology.orchestra.run.smithi192.stdout: "dentry_pinned_count": 0, 2023-10-06T03:30:06.223 INFO:teuthology.orchestra.run.smithi192.stdout: "id": 4679, 2023-10-06T03:30:06.223 INFO:teuthology.orchestra.run.smithi192.stdout: "inst": { 2023-10-06T03:30:06.223 INFO:teuthology.orchestra.run.smithi192.stdout: "name": { 2023-10-06T03:30:06.223 INFO:teuthology.orchestra.run.smithi192.stdout: "type": "client", 2023-10-06T03:30:06.223 INFO:teuthology.orchestra.run.smithi192.stdout: "num": 4679 2023-10-06T03:30:06.223 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2023-10-06T03:30:06.223 INFO:teuthology.orchestra.run.smithi192.stdout: "addr": { 2023-10-06T03:30:06.223 INFO:teuthology.orchestra.run.smithi192.stdout: "type": "v1", 2023-10-06T03:30:06.224 INFO:teuthology.orchestra.run.smithi192.stdout: "addr": "192.168.0.1:0", 2023-10-06T03:30:06.224 INFO:teuthology.orchestra.run.smithi192.stdout: "nonce": 1712639905 2023-10-06T03:30:06.224 INFO:teuthology.orchestra.run.smithi192.stdout: } 2023-10-06T03:30:06.224 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2023-10-06T03:30:06.224 INFO:teuthology.orchestra.run.smithi192.stdout: "addr": { 2023-10-06T03:30:06.224 INFO:teuthology.orchestra.run.smithi192.stdout: "type": "v1", 2023-10-06T03:30:06.224 INFO:teuthology.orchestra.run.smithi192.stdout: "addr": "192.168.0.1:0", 2023-10-06T03:30:06.224 INFO:teuthology.orchestra.run.smithi192.stdout: "nonce": 1712639905 2023-10-06T03:30:06.224 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2023-10-06T03:30:06.224 INFO:teuthology.orchestra.run.smithi192.stdout: "inst_str": "client.4679 192.168.0.1:0/1712639905", 2023-10-06T03:30:06.224 INFO:teuthology.orchestra.run.smithi192.stdout: "addr_str": "192.168.0.1:0/1712639905", 2023-10-06T03:30:06.224 INFO:teuthology.orchestra.run.smithi192.stdout: "inode_count": 1, 2023-10-06T03:30:06.225 INFO:teuthology.orchestra.run.smithi192.stdout: "mds_epoch": 8, 2023-10-06T03:30:06.225 INFO:teuthology.orchestra.run.smithi192.stdout: "osd_epoch": 28, 2023-10-06T03:30:06.225 INFO:teuthology.orchestra.run.smithi192.stdout: "osd_epoch_barrier": 0, 2023-10-06T03:30:06.225 INFO:teuthology.orchestra.run.smithi192.stdout: "blocklisted": false, 2023-10-06T03:30:06.225 INFO:teuthology.orchestra.run.smithi192.stdout: "fs_name": "cephfs" 2023-10-06T03:30:06.225 INFO:teuthology.orchestra.run.smithi192.stdout:} 2023-10-06T03:30:06.231 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-10-06T03:30:06.345 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:06.341+0000 7f2c85f31700 1 Processor -- start 2023-10-06T03:30:06.346 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:06.341+0000 7f2c85f31700 1 -- start start 2023-10-06T03:30:06.346 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:06.341+0000 7f2c85f31700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2c801408d0 0x7f2c8006b040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:06.347 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:06.341+0000 7f2c85f31700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f2c800c6ce0 0x7f2c800c7140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:06.347 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:06.341+0000 7f2c85f31700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c800c7680 con 0x7f2c80140ca0 2023-10-06T03:30:06.347 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:06.341+0000 7f2c85f31700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2c8006c9b0 con 0x7f2c800c6ce0 2023-10-06T03:30:06.347 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:06.341+0000 7f2c85f31700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c8006cc90 con 0x7f2c801408d0 2023-10-06T03:30:06.348 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:06.341+0000 7f2c7ffff700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f2c800c6ce0 0x7f2c800c7140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:06.348 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:06.345+0000 7f2c77fff700 1 --1- >> v1:172.21.15.192:6789/0 conn(0x7f2c80140ca0 0x7f2c8006c180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.110:44838/0 (socket says 172.21.15.110:44838) 2023-10-06T03:30:06.348 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:06.345+0000 7f2c7f7fe700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2c801408d0 0x7f2c8006b040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:06.348 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:06.345+0000 7f2c77fff700 1 -- 172.21.15.110:0/3905234422 learned_addr learned my addr 172.21.15.110:0/3905234422 (peer_addr_for_me v1:172.21.15.110:0/0) 2023-10-06T03:30:06.348 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:06.345+0000 7f2c7effd700 1 -- 172.21.15.110:0/3905234422 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1319393509 0 0) 0x7f2c68001110 con 0x7f2c80140ca0 2023-10-06T03:30:06.348 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:06.345+0000 7f2c7ffff700 1 -- 172.21.15.110:0/3905234422 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2c801408d0 msgr2=0x7f2c8006b040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:30:06.348 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:06.345+0000 7f2c7ffff700 1 --2- 172.21.15.110:0/3905234422 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2c801408d0 0x7f2c8006b040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:06.348 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:06.345+0000 7f2c7ffff700 1 -- 172.21.15.110:0/3905234422 >> v1:172.21.15.192:6789/0 conn(0x7f2c80140ca0 legacy=0x7f2c8006c180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:06.348 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:06.345+0000 7f2c7ffff700 1 -- 172.21.15.110:0/3905234422 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c8006cf70 con 0x7f2c800c6ce0 2023-10-06T03:30:06.349 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:06.345+0000 7f2c7ffff700 1 --2- 172.21.15.110:0/3905234422 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2c800c6ce0 0x7f2c800c7140 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f2c78009e60 tx=0x7f2c78009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=53ff892d8b114c69 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:30:06.349 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:06.345+0000 7f2c7effd700 1 -- 172.21.15.110:0/3905234422 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c7800e070 con 0x7f2c800c6ce0 2023-10-06T03:30:06.349 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:06.345+0000 7f2c7effd700 1 -- 172.21.15.110:0/3905234422 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c7800bb60 con 0x7f2c800c6ce0 2023-10-06T03:30:06.349 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:06.345+0000 7f2c7effd700 1 -- 172.21.15.110:0/3905234422 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c78012430 con 0x7f2c800c6ce0 2023-10-06T03:30:06.350 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:06.345+0000 7f2c85f31700 1 -- 172.21.15.110:0/3905234422 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2c800c6ce0 msgr2=0x7f2c800c7140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:06.350 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:06.345+0000 7f2c85f31700 1 --2- 172.21.15.110:0/3905234422 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2c800c6ce0 0x7f2c800c7140 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f2c78009e60 tx=0x7f2c78009780 comp rx=0 tx=0).stop 2023-10-06T03:30:06.350 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:06.345+0000 7f2c85f31700 1 -- 172.21.15.110:0/3905234422 shutdown_connections 2023-10-06T03:30:06.350 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:06.345+0000 7f2c85f31700 1 --2- 172.21.15.110:0/3905234422 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2c800c6ce0 0x7f2c800c7140 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:06.351 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:06.345+0000 7f2c85f31700 1 --2- 172.21.15.110:0/3905234422 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2c801408d0 0x7f2c8006b040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:06.351 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:06.345+0000 7f2c85f31700 1 -- 172.21.15.110:0/3905234422 >> 172.21.15.110:0/3905234422 conn(0x7f2c800cbd00 msgr2=0x7f2c800cc100 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:30:06.351 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:06.345+0000 7f2c85f31700 1 -- 172.21.15.110:0/3905234422 shutdown_connections 2023-10-06T03:30:06.351 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:06.345+0000 7f2c85f31700 1 -- 172.21.15.110:0/3905234422 wait complete. 2023-10-06T03:30:06.353 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:06.349+0000 7f2c85f31700 1 Processor -- start 2023-10-06T03:30:06.353 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:06.349+0000 7f2c85f31700 1 -- start start 2023-10-06T03:30:06.355 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:06.349+0000 7f2c85f31700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2c800c6ce0 0x7f2c80155c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:06.355 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:06.349+0000 7f2c85f31700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2c801408d0 0x7f2c80139980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:06.355 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:06.349+0000 7f2c85f31700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2c80140ca0 0x7f2c80139ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:06.355 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:06.349+0000 7f2c85f31700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c8015c860 con 0x7f2c800c6ce0 2023-10-06T03:30:06.356 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:06.349+0000 7f2c85f31700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c8015bfc0 con 0x7f2c801408d0 2023-10-06T03:30:06.356 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:06.349+0000 7f2c85f31700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c8015c2a0 con 0x7f2c80140ca0 2023-10-06T03:30:06.356 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:06.349+0000 7f2c7f7fe700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2c800c6ce0 0x7f2c80155c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:06.356 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:06.349+0000 7f2c7f7fe700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2c800c6ce0 0x7f2c80155c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:40436/0 (socket says 172.21.15.110:40436) 2023-10-06T03:30:06.356 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:06.349+0000 7f2c7f7fe700 1 -- 172.21.15.110:0/1272940693 learned_addr learned my addr 172.21.15.110:0/1272940693 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:30:06.356 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:06.349+0000 7f2c7ffff700 1 --2- 172.21.15.110:0/1272940693 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2c80140ca0 0x7f2c80139ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:06.356 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:06.349+0000 7f2c77fff700 1 --2- 172.21.15.110:0/1272940693 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2c801408d0 0x7f2c80139980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:06.356 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:06.353+0000 7f2c7f7fe700 1 -- 172.21.15.110:0/1272940693 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2c80140ca0 msgr2=0x7f2c80139ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:06.356 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:06.353+0000 7f2c7f7fe700 1 --2- 172.21.15.110:0/1272940693 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2c80140ca0 0x7f2c80139ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:06.356 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:06.353+0000 7f2c7f7fe700 1 -- 172.21.15.110:0/1272940693 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2c801408d0 msgr2=0x7f2c80139980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:06.356 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:06.353+0000 7f2c7f7fe700 1 --2- 172.21.15.110:0/1272940693 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2c801408d0 0x7f2c80139980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:06.357 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:06.353+0000 7f2c7f7fe700 1 -- 172.21.15.110:0/1272940693 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c78009a20 con 0x7f2c800c6ce0 2023-10-06T03:30:06.357 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:06.353+0000 7f2c7ffff700 1 --2- 172.21.15.110:0/1272940693 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2c80140ca0 0x7f2c80139ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-06T03:30:06.357 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:06.353+0000 7f2c7f7fe700 1 --2- 172.21.15.110:0/1272940693 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2c800c6ce0 0x7f2c80155c70 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f2c7000ec80 tx=0x7f2c700091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:30:06.357 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:06.353+0000 7f2c777fe700 1 -- 172.21.15.110:0/1272940693 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c7000c9d0 con 0x7f2c800c6ce0 2023-10-06T03:30:06.358 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:06.353+0000 7f2c777fe700 1 -- 172.21.15.110:0/1272940693 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c7000cb30 con 0x7f2c800c6ce0 2023-10-06T03:30:06.358 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:06.353+0000 7f2c85f31700 1 -- 172.21.15.110:0/1272940693 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c8015c580 con 0x7f2c800c6ce0 2023-10-06T03:30:06.358 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:06.353+0000 7f2c777fe700 1 -- 172.21.15.110:0/1272940693 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c70013470 con 0x7f2c800c6ce0 2023-10-06T03:30:06.358 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:06.353+0000 7f2c85f31700 1 -- 172.21.15.110:0/1272940693 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c801561b0 con 0x7f2c800c6ce0 2023-10-06T03:30:06.358 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:06.353+0000 7f2c85f31700 1 -- 172.21.15.110:0/1272940693 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c80157a70 con 0x7f2c800c6ce0 2023-10-06T03:30:06.359 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:06.353+0000 7f2c777fe700 1 -- 172.21.15.110:0/1272940693 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7f2c70013820 con 0x7f2c800c6ce0 2023-10-06T03:30:06.360 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:06.357+0000 7f2c777fe700 1 --2- 172.21.15.110:0/1272940693 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f2c60060550 0x7f2c60062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:06.360 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:06.357+0000 7f2c777fe700 1 -- 172.21.15.110:0/1272940693 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6809+0+0 (secure 0 0 0) 0x7f2c7007d710 con 0x7f2c800c6ce0 2023-10-06T03:30:06.360 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:06.357+0000 7f2c77fff700 1 --2- 172.21.15.110:0/1272940693 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f2c60060550 0x7f2c60062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:06.360 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:06.357+0000 7f2c77fff700 1 --2- 172.21.15.110:0/1272940693 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f2c60060550 0x7f2c60062a00 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f2c800c6b60 tx=0x7f2c68009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:30:06.367 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:06.365+0000 7f2c777fe700 1 -- 172.21.15.110:0/1272940693 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f2c7004d930 con 0x7f2c800c6ce0 2023-10-06T03:30:06.564 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:06.561+0000 7f2c85f31700 1 -- 172.21.15.110:0/1272940693 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f2c8006ff60 con 0x7f2c800c6ce0 2023-10-06T03:30:06.564 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:06.561+0000 7f2c777fe700 1 -- 172.21.15.110:0/1272940693 <== mon.0 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v8) v1 ==== 53+0+88 (secure 0 0 0) 0x7f2c70050f50 con 0x7f2c800c6ce0 2023-10-06T03:30:06.564 INFO:teuthology.orchestra.run.smithi110.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2023-10-06T03:30:06.567 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:06.561+0000 7f2c85f31700 1 -- 172.21.15.110:0/1272940693 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f2c60060550 msgr2=0x7f2c60062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:06.567 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:06.561+0000 7f2c85f31700 1 --2- 172.21.15.110:0/1272940693 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f2c60060550 0x7f2c60062a00 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f2c800c6b60 tx=0x7f2c68009040 comp rx=0 tx=0).stop 2023-10-06T03:30:06.567 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:06.561+0000 7f2c85f31700 1 -- 172.21.15.110:0/1272940693 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2c800c6ce0 msgr2=0x7f2c80155c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:06.568 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:06.561+0000 7f2c85f31700 1 --2- 172.21.15.110:0/1272940693 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2c800c6ce0 0x7f2c80155c70 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f2c7000ec80 tx=0x7f2c700091a0 comp rx=0 tx=0).stop 2023-10-06T03:30:06.568 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:06.561+0000 7f2c85f31700 1 -- 172.21.15.110:0/1272940693 shutdown_connections 2023-10-06T03:30:06.568 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:06.561+0000 7f2c85f31700 1 --2- 172.21.15.110:0/1272940693 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f2c60060550 0x7f2c60062a00 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:06.568 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:06.561+0000 7f2c85f31700 1 --2- 172.21.15.110:0/1272940693 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2c80140ca0 0x7f2c80139ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:06.568 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:06.561+0000 7f2c85f31700 1 --2- 172.21.15.110:0/1272940693 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2c801408d0 0x7f2c80139980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:06.568 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:06.561+0000 7f2c85f31700 1 --2- 172.21.15.110:0/1272940693 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2c800c6ce0 0x7f2c80155c70 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:06.568 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:06.561+0000 7f2c85f31700 1 -- 172.21.15.110:0/1272940693 >> 172.21.15.110:0/1272940693 conn(0x7f2c800cbd00 msgr2=0x7f2c800c0aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:30:06.568 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:06.565+0000 7f2c85f31700 1 -- 172.21.15.110:0/1272940693 shutdown_connections 2023-10-06T03:30:06.568 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:06.565+0000 7f2c85f31700 1 -- 172.21.15.110:0/1272940693 wait complete. 2023-10-06T03:30:06.578 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2023-10-06T03:30:06.579 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-10-06T03:30:06.579 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi192.front.sepia.ceph.com 2023-10-06T03:30:06.579 INFO:tasks.cephfs.mount:self.client.name = client.3 2023-10-06T03:30:06.579 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.3 2023-10-06T03:30:06.579 INFO:tasks.cephfs.mount:self.cephfs_name = None 2023-10-06T03:30:06.579 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2023-10-06T03:30:06.579 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-10-06T03:30:06.579 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.3' 2023-10-06T03:30:06.579 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-06T03:30:06.580 DEBUG:teuthology.orchestra.run.smithi192:> ip addr 2023-10-06T03:30:06.588 INFO:teuthology.orchestra.run.smithi192.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-10-06T03:30:06.588 INFO:teuthology.orchestra.run.smithi192.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-10-06T03:30:06.588 INFO:teuthology.orchestra.run.smithi192.stdout: inet 127.0.0.1/8 scope host lo 2023-10-06T03:30:06.588 INFO:teuthology.orchestra.run.smithi192.stdout: valid_lft forever preferred_lft forever 2023-10-06T03:30:06.588 INFO:teuthology.orchestra.run.smithi192.stdout: inet6 ::1/128 scope host 2023-10-06T03:30:06.588 INFO:teuthology.orchestra.run.smithi192.stdout: valid_lft forever preferred_lft forever 2023-10-06T03:30:06.589 INFO:teuthology.orchestra.run.smithi192.stdout:2: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2023-10-06T03:30:06.589 INFO:teuthology.orchestra.run.smithi192.stdout: link/ether 0c:c4:7a:d9:38:4e brd ff:ff:ff:ff:ff:ff 2023-10-06T03:30:06.589 INFO:teuthology.orchestra.run.smithi192.stdout:3: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000 2023-10-06T03:30:06.589 INFO:teuthology.orchestra.run.smithi192.stdout: link/ether 0c:c4:7a:d9:38:4f brd ff:ff:ff:ff:ff:ff 2023-10-06T03:30:06.589 INFO:teuthology.orchestra.run.smithi192.stdout:4: enp3s0f0: mtu 1500 qdisc noop state DOWN group default qlen 1000 2023-10-06T03:30:06.589 INFO:teuthology.orchestra.run.smithi192.stdout: link/ether 0c:c4:7a:88:80:48 brd ff:ff:ff:ff:ff:ff 2023-10-06T03:30:06.589 INFO:teuthology.orchestra.run.smithi192.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2023-10-06T03:30:06.589 INFO:teuthology.orchestra.run.smithi192.stdout: link/ether 0c:c4:7a:88:80:49 brd ff:ff:ff:ff:ff:ff 2023-10-06T03:30:06.589 INFO:teuthology.orchestra.run.smithi192.stdout: inet 172.21.15.192/20 brd 172.21.15.255 scope global dynamic enp3s0f1 2023-10-06T03:30:06.589 INFO:teuthology.orchestra.run.smithi192.stdout: valid_lft 42576sec preferred_lft 42576sec 2023-10-06T03:30:06.589 INFO:teuthology.orchestra.run.smithi192.stdout: inet6 fe80::ec4:7aff:fe88:8049/64 scope link 2023-10-06T03:30:06.589 INFO:teuthology.orchestra.run.smithi192.stdout: valid_lft forever preferred_lft forever 2023-10-06T03:30:06.590 INFO:teuthology.orchestra.run.smithi192.stdout:6: docker0: mtu 1500 qdisc noqueue state DOWN group default 2023-10-06T03:30:06.590 INFO:teuthology.orchestra.run.smithi192.stdout: link/ether 02:42:fb:a6:b4:5a brd ff:ff:ff:ff:ff:ff 2023-10-06T03:30:06.590 INFO:teuthology.orchestra.run.smithi192.stdout: inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 2023-10-06T03:30:06.590 INFO:teuthology.orchestra.run.smithi192.stdout: valid_lft forever preferred_lft forever 2023-10-06T03:30:06.590 INFO:teuthology.orchestra.run.smithi192.stdout:7: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-10-06T03:30:06.590 INFO:teuthology.orchestra.run.smithi192.stdout: link/ether 0a:20:8d:91:de:4f brd ff:ff:ff:ff:ff:ff 2023-10-06T03:30:06.590 INFO:teuthology.orchestra.run.smithi192.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-10-06T03:30:06.590 INFO:teuthology.orchestra.run.smithi192.stdout: valid_lft forever preferred_lft forever 2023-10-06T03:30:06.590 INFO:teuthology.orchestra.run.smithi192.stdout: inet6 fe80::4445:7dff:fe96:545f/64 scope link 2023-10-06T03:30:06.590 INFO:teuthology.orchestra.run.smithi192.stdout: valid_lft forever preferred_lft forever 2023-10-06T03:30:06.590 INFO:teuthology.orchestra.run.smithi192.stdout:8: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-10-06T03:30:06.591 INFO:teuthology.orchestra.run.smithi192.stdout: link/ether 0a:20:8d:91:de:4f brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2023-10-06T03:30:06.591 INFO:teuthology.orchestra.run.smithi192.stdout: inet6 fe80::820:8dff:fe91:de4f/64 scope link 2023-10-06T03:30:06.591 INFO:teuthology.orchestra.run.smithi192.stdout: valid_lft forever preferred_lft forever 2023-10-06T03:30:06.591 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-06T03:30:06.591 DEBUG:teuthology.orchestra.run.smithi192:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-10-06T03:30:06.606 INFO:teuthology.orchestra.run.smithi192.stdout:1 2023-10-06T03:30:06.608 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-06T03:30:06.608 DEBUG:teuthology.orchestra.run.smithi192:> route 2023-10-06T03:30:06.658 INFO:teuthology.orchestra.run.smithi192.stdout:Kernel IP routing table 2023-10-06T03:30:06.658 INFO:teuthology.orchestra.run.smithi192.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-10-06T03:30:06.659 INFO:teuthology.orchestra.run.smithi192.stdout:default _gateway 0.0.0.0 UG 0 0 0 enp3s0f1 2023-10-06T03:30:06.659 INFO:teuthology.orchestra.run.smithi192.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2023-10-06T03:30:06.659 INFO:teuthology.orchestra.run.smithi192.stdout:172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 2023-10-06T03:30:06.659 INFO:teuthology.orchestra.run.smithi192.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 0 0 0 enp3s0f1 2023-10-06T03:30:06.659 INFO:teuthology.orchestra.run.smithi192.stdout:_gateway 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2023-10-06T03:30:06.659 INFO:teuthology.orchestra.run.smithi192.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-10-06T03:30:06.660 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-06T03:30:06.660 DEBUG:teuthology.orchestra.run.smithi192:> (cd / && exec bash -c ' 2023-10-06T03:30:06.660 DEBUG:teuthology.orchestra.run.smithi192:> set -e 2023-10-06T03:30:06.660 DEBUG:teuthology.orchestra.run.smithi192:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2023-10-06T03:30:06.660 DEBUG:teuthology.orchestra.run.smithi192:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2023-10-06T03:30:06.660 DEBUG:teuthology.orchestra.run.smithi192:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2023-10-06T03:30:06.660 DEBUG:teuthology.orchestra.run.smithi192:> ') 2023-10-06T03:30:06.698 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-06T03:30:06.698 DEBUG:teuthology.orchestra.run.smithi192:> ip netns list 2023-10-06T03:30:06.705 INFO:teuthology.orchestra.run.smithi192.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2023-10-06T03:30:06.705 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-06T03:30:06.706 DEBUG:teuthology.orchestra.run.smithi192:> ip netns list-id 2023-10-06T03:30:06.757 INFO:teuthology.orchestra.run.smithi192.stdout:nsid 0 (iproute2 netns name: ceph-ns--home-ubuntu-cephtest-mnt.1) 2023-10-06T03:30:06.758 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-06T03:30:06.758 DEBUG:teuthology.orchestra.run.smithi192:> (cd / && exec bash -c ' 2023-10-06T03:30:06.758 DEBUG:teuthology.orchestra.run.smithi192:> set -e 2023-10-06T03:30:06.758 DEBUG:teuthology.orchestra.run.smithi192:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.3 2023-10-06T03:30:06.758 DEBUG:teuthology.orchestra.run.smithi192:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.3 1 2023-10-06T03:30:06.758 DEBUG:teuthology.orchestra.run.smithi192:> ') 2023-10-06T03:30:06.788 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-06T03:30:06.788 DEBUG:teuthology.orchestra.run.smithi192:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr 2023-10-06T03:30:06.805 INFO:teuthology.orchestra.run.smithi192.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-10-06T03:30:06.805 INFO:teuthology.orchestra.run.smithi192.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-10-06T03:30:06.805 INFO:teuthology.orchestra.run.smithi192.stdout: inet 127.0.0.1/8 scope host lo 2023-10-06T03:30:06.806 INFO:teuthology.orchestra.run.smithi192.stdout: valid_lft forever preferred_lft forever 2023-10-06T03:30:06.806 INFO:teuthology.orchestra.run.smithi192.stdout: inet6 ::1/128 scope host 2023-10-06T03:30:06.806 INFO:teuthology.orchestra.run.smithi192.stdout: valid_lft forever preferred_lft forever 2023-10-06T03:30:06.806 INFO:teuthology.orchestra.run.smithi192.stdout:2: veth0@if8: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-10-06T03:30:06.806 INFO:teuthology.orchestra.run.smithi192.stdout: link/ether fe:39:40:05:71:19 brd ff:ff:ff:ff:ff:ff link-netnsid 0 2023-10-06T03:30:06.806 INFO:teuthology.orchestra.run.smithi192.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2023-10-06T03:30:06.806 INFO:teuthology.orchestra.run.smithi192.stdout: valid_lft forever preferred_lft forever 2023-10-06T03:30:06.806 INFO:teuthology.orchestra.run.smithi192.stdout: inet6 fe80::fc39:40ff:fe05:7119/64 scope link 2023-10-06T03:30:06.806 INFO:teuthology.orchestra.run.smithi192.stdout: valid_lft forever preferred_lft forever 2023-10-06T03:30:06.807 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-06T03:30:06.807 DEBUG:teuthology.orchestra.run.smithi192:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr 2023-10-06T03:30:06.867 INFO:teuthology.orchestra.run.smithi192.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-10-06T03:30:06.867 INFO:teuthology.orchestra.run.smithi192.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-10-06T03:30:06.867 INFO:teuthology.orchestra.run.smithi192.stdout: inet 127.0.0.1/8 scope host lo 2023-10-06T03:30:06.867 INFO:teuthology.orchestra.run.smithi192.stdout: valid_lft forever preferred_lft forever 2023-10-06T03:30:06.867 INFO:teuthology.orchestra.run.smithi192.stdout: inet6 ::1/128 scope host 2023-10-06T03:30:06.867 INFO:teuthology.orchestra.run.smithi192.stdout: valid_lft forever preferred_lft forever 2023-10-06T03:30:06.867 INFO:teuthology.orchestra.run.smithi192.stdout:2: veth0@if8: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-10-06T03:30:06.867 INFO:teuthology.orchestra.run.smithi192.stdout: link/ether fe:39:40:05:71:19 brd ff:ff:ff:ff:ff:ff link-netnsid 0 2023-10-06T03:30:06.867 INFO:teuthology.orchestra.run.smithi192.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2023-10-06T03:30:06.868 INFO:teuthology.orchestra.run.smithi192.stdout: valid_lft forever preferred_lft forever 2023-10-06T03:30:06.868 INFO:teuthology.orchestra.run.smithi192.stdout: inet6 fe80::fc39:40ff:fe05:7119/64 scope link 2023-10-06T03:30:06.868 INFO:teuthology.orchestra.run.smithi192.stdout: valid_lft forever preferred_lft forever 2023-10-06T03:30:06.869 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.3' with 192.168.0.2/16 2023-10-06T03:30:06.869 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-06T03:30:06.869 DEBUG:teuthology.orchestra.run.smithi192:> (cd / && exec bash -c ' 2023-10-06T03:30:06.869 DEBUG:teuthology.orchestra.run.smithi192:> set -e 2023-10-06T03:30:06.869 DEBUG:teuthology.orchestra.run.smithi192:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.3 type veth peer name brx.1 2023-10-06T03:30:06.870 DEBUG:teuthology.orchestra.run.smithi192:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip addr add 192.168.0.2/16 brd 192.168.255.255 dev veth0 2023-10-06T03:30:06.870 DEBUG:teuthology.orchestra.run.smithi192:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip link set veth0 up 2023-10-06T03:30:06.870 DEBUG:teuthology.orchestra.run.smithi192:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip link set lo up 2023-10-06T03:30:06.870 DEBUG:teuthology.orchestra.run.smithi192:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip route add default via 192.168.255.254 2023-10-06T03:30:06.870 DEBUG:teuthology.orchestra.run.smithi192:> ') 2023-10-06T03:30:06.926 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-06T03:30:06.926 DEBUG:teuthology.orchestra.run.smithi192:> (cd / && exec bash -c ' 2023-10-06T03:30:06.926 DEBUG:teuthology.orchestra.run.smithi192:> set -e 2023-10-06T03:30:06.926 DEBUG:teuthology.orchestra.run.smithi192:> sudo ip link set brx.1 up 2023-10-06T03:30:06.926 DEBUG:teuthology.orchestra.run.smithi192:> sudo ip link set dev brx.1 master ceph-brx 2023-10-06T03:30:06.926 DEBUG:teuthology.orchestra.run.smithi192:> ') 2023-10-06T03:30:06.953 INFO:tasks.cephfs.fuse_mount:Client client.3 config is {} 2023-10-06T03:30:06.954 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-06T03:30:06.954 DEBUG:teuthology.orchestra.run.smithi192:> mkdir -p -v /home/ubuntu/cephtest/mnt.3 2023-10-06T03:30:07.004 INFO:teuthology.orchestra.run.smithi192.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.3' 2023-10-06T03:30:07.004 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-06T03:30:07.005 DEBUG:teuthology.orchestra.run.smithi192:> chmod 0000 /home/ubuntu/cephtest/mnt.3 2023-10-06T03:30:07.011 DEBUG:teuthology.orchestra.run.smithi192:> sudo modprobe fuse 2023-10-06T03:30:07.026 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-10-06T03:30:07.026 DEBUG:teuthology.orchestra.run.smithi192:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-10-06T03:30:07.037 INFO:teuthology.orchestra.run.smithi192.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-10-06T03:30:07.038 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-10-06T03:30:07.038 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-06T03:30:07.039 DEBUG:teuthology.orchestra.run.smithi192:> ls /sys/fs/fuse/connections 2023-10-06T03:30:07.044 INFO:teuthology.orchestra.run.smithi192.stdout:52 2023-10-06T03:30:07.044 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [52] 2023-10-06T03:30:07.045 DEBUG:teuthology.orchestra.run.smithi192:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.3 --id 3) 2023-10-06T03:30:07.047 DEBUG:teuthology.orchestra.run.smithi192:> sudo modprobe fuse 2023-10-06T03:30:07.061 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-10-06T03:30:07.061 DEBUG:teuthology.orchestra.run.smithi192:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-10-06T03:30:07.117 INFO:teuthology.orchestra.run.smithi192.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-10-06T03:30:07.118 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-10-06T03:30:07.119 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-06T03:30:07.119 DEBUG:teuthology.orchestra.run.smithi192:> ls /sys/fs/fuse/connections 2023-10-06T03:30:07.126 INFO:teuthology.orchestra.run.smithi192.stdout:52 2023-10-06T03:30:07.128 INFO:tasks.cephfs.fuse_mount.ceph-fuse.3.smithi192.stderr:2023-10-06T03:30:07.125+0000 7f1afee24380 -1 init, newargv = 0x5576e0c569a0 newargc=15 2023-10-06T03:30:07.128 INFO:tasks.cephfs.fuse_mount.ceph-fuse.3.smithi192.stderr:ceph-fuse[39360]: starting ceph client 2023-10-06T03:30:07.137 INFO:tasks.cephfs.fuse_mount.ceph-fuse.3.smithi192.stderr:ceph-fuse[39360]: starting fuse 2023-10-06T03:30:07.831 DEBUG:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-06T03:30:07.834 DEBUG:teuthology.orchestra.run.smithi192:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-06T03:30:08.227 DEBUG:teuthology.orchestra.run.smithi192:> sudo modprobe fuse 2023-10-06T03:30:08.245 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-10-06T03:30:08.245 DEBUG:teuthology.orchestra.run.smithi192:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-10-06T03:30:08.257 INFO:teuthology.orchestra.run.smithi192.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-10-06T03:30:08.258 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-10-06T03:30:08.258 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-06T03:30:08.258 DEBUG:teuthology.orchestra.run.smithi192:> ls /sys/fs/fuse/connections 2023-10-06T03:30:08.312 INFO:teuthology.orchestra.run.smithi192.stdout:52 2023-10-06T03:30:08.312 INFO:teuthology.orchestra.run.smithi192.stdout:53 2023-10-06T03:30:08.312 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [52, 53] 2023-10-06T03:30:08.312 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2023-10-06T03:30:08.313 DEBUG:teuthology.orchestra.run.smithi192:> import glob 2023-10-06T03:30:08.313 DEBUG:teuthology.orchestra.run.smithi192:> import re 2023-10-06T03:30:08.313 DEBUG:teuthology.orchestra.run.smithi192:> import os 2023-10-06T03:30:08.313 DEBUG:teuthology.orchestra.run.smithi192:> import subprocess 2023-10-06T03:30:08.313 DEBUG:teuthology.orchestra.run.smithi192:> 2023-10-06T03:30:08.313 DEBUG:teuthology.orchestra.run.smithi192:> def _find_admin_socket(client_name): 2023-10-06T03:30:08.313 DEBUG:teuthology.orchestra.run.smithi192:> asok_path = "/var/run/ceph/ceph-client.3.*.asok" 2023-10-06T03:30:08.313 DEBUG:teuthology.orchestra.run.smithi192:> files = glob.glob(asok_path) 2023-10-06T03:30:08.313 DEBUG:teuthology.orchestra.run.smithi192:> mountpoint = "/home/ubuntu/cephtest/mnt.3" 2023-10-06T03:30:08.313 DEBUG:teuthology.orchestra.run.smithi192:> 2023-10-06T03:30:08.313 DEBUG:teuthology.orchestra.run.smithi192:> # Given a non-glob path, it better be there 2023-10-06T03:30:08.313 DEBUG:teuthology.orchestra.run.smithi192:> if "*" not in asok_path: 2023-10-06T03:30:08.314 DEBUG:teuthology.orchestra.run.smithi192:> assert(len(files) == 1) 2023-10-06T03:30:08.314 DEBUG:teuthology.orchestra.run.smithi192:> return files[0] 2023-10-06T03:30:08.314 DEBUG:teuthology.orchestra.run.smithi192:> 2023-10-06T03:30:08.314 DEBUG:teuthology.orchestra.run.smithi192:> for f in files: 2023-10-06T03:30:08.314 DEBUG:teuthology.orchestra.run.smithi192:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2023-10-06T03:30:08.314 DEBUG:teuthology.orchestra.run.smithi192:> if os.path.exists("/proc/{0}".format(pid)): 2023-10-06T03:30:08.314 DEBUG:teuthology.orchestra.run.smithi192:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2023-10-06T03:30:08.314 DEBUG:teuthology.orchestra.run.smithi192:> contents = proc_f.read() 2023-10-06T03:30:08.314 DEBUG:teuthology.orchestra.run.smithi192:> if mountpoint in contents: 2023-10-06T03:30:08.314 DEBUG:teuthology.orchestra.run.smithi192:> return f 2023-10-06T03:30:08.314 DEBUG:teuthology.orchestra.run.smithi192:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2023-10-06T03:30:08.314 DEBUG:teuthology.orchestra.run.smithi192:> 2023-10-06T03:30:08.314 DEBUG:teuthology.orchestra.run.smithi192:> print(_find_admin_socket("client.3")) 2023-10-06T03:30:08.315 DEBUG:teuthology.orchestra.run.smithi192:> ' 2023-10-06T03:30:08.395 INFO:teuthology.orchestra.run.smithi192.stdout:/var/run/ceph/ceph-client.3.39360.asok 2023-10-06T03:30:08.582 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.3.39360.asok 2023-10-06T03:30:08.583 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-06T03:30:08.583 DEBUG:teuthology.orchestra.run.smithi192:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.3.39360.asok status 2023-10-06T03:30:08.677 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2023-10-06T03:30:08.677 INFO:teuthology.orchestra.run.smithi192.stdout: "metadata": { 2023-10-06T03:30:08.677 INFO:teuthology.orchestra.run.smithi192.stdout: "ceph_sha1": "736ac5ba72cb47af7d6f6666b1e100f7e20a20e2", 2023-10-06T03:30:08.677 INFO:teuthology.orchestra.run.smithi192.stdout: "ceph_version": "ceph version 17.2.6-1296-g736ac5ba (736ac5ba72cb47af7d6f6666b1e100f7e20a20e2) quincy (stable)", 2023-10-06T03:30:08.677 INFO:teuthology.orchestra.run.smithi192.stdout: "entity_id": "3", 2023-10-06T03:30:08.678 INFO:teuthology.orchestra.run.smithi192.stdout: "hostname": "smithi192", 2023-10-06T03:30:08.678 INFO:teuthology.orchestra.run.smithi192.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.3", 2023-10-06T03:30:08.678 INFO:teuthology.orchestra.run.smithi192.stdout: "pid": "39360", 2023-10-06T03:30:08.678 INFO:teuthology.orchestra.run.smithi192.stdout: "root": "/" 2023-10-06T03:30:08.678 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2023-10-06T03:30:08.678 INFO:teuthology.orchestra.run.smithi192.stdout: "dentry_count": 0, 2023-10-06T03:30:08.678 INFO:teuthology.orchestra.run.smithi192.stdout: "dentry_pinned_count": 0, 2023-10-06T03:30:08.678 INFO:teuthology.orchestra.run.smithi192.stdout: "id": 4828, 2023-10-06T03:30:08.678 INFO:teuthology.orchestra.run.smithi192.stdout: "inst": { 2023-10-06T03:30:08.678 INFO:teuthology.orchestra.run.smithi192.stdout: "name": { 2023-10-06T03:30:08.678 INFO:teuthology.orchestra.run.smithi192.stdout: "type": "client", 2023-10-06T03:30:08.679 INFO:teuthology.orchestra.run.smithi192.stdout: "num": 4828 2023-10-06T03:30:08.679 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2023-10-06T03:30:08.679 INFO:teuthology.orchestra.run.smithi192.stdout: "addr": { 2023-10-06T03:30:08.679 INFO:teuthology.orchestra.run.smithi192.stdout: "type": "v1", 2023-10-06T03:30:08.679 INFO:teuthology.orchestra.run.smithi192.stdout: "addr": "192.168.0.2:0", 2023-10-06T03:30:08.679 INFO:teuthology.orchestra.run.smithi192.stdout: "nonce": 1075616156 2023-10-06T03:30:08.679 INFO:teuthology.orchestra.run.smithi192.stdout: } 2023-10-06T03:30:08.679 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2023-10-06T03:30:08.679 INFO:teuthology.orchestra.run.smithi192.stdout: "addr": { 2023-10-06T03:30:08.679 INFO:teuthology.orchestra.run.smithi192.stdout: "type": "v1", 2023-10-06T03:30:08.679 INFO:teuthology.orchestra.run.smithi192.stdout: "addr": "192.168.0.2:0", 2023-10-06T03:30:08.679 INFO:teuthology.orchestra.run.smithi192.stdout: "nonce": 1075616156 2023-10-06T03:30:08.680 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2023-10-06T03:30:08.680 INFO:teuthology.orchestra.run.smithi192.stdout: "inst_str": "client.4828 192.168.0.2:0/1075616156", 2023-10-06T03:30:08.680 INFO:teuthology.orchestra.run.smithi192.stdout: "addr_str": "192.168.0.2:0/1075616156", 2023-10-06T03:30:08.680 INFO:teuthology.orchestra.run.smithi192.stdout: "inode_count": 1, 2023-10-06T03:30:08.680 INFO:teuthology.orchestra.run.smithi192.stdout: "mds_epoch": 8, 2023-10-06T03:30:08.680 INFO:teuthology.orchestra.run.smithi192.stdout: "osd_epoch": 28, 2023-10-06T03:30:08.680 INFO:teuthology.orchestra.run.smithi192.stdout: "osd_epoch_barrier": 0, 2023-10-06T03:30:08.680 INFO:teuthology.orchestra.run.smithi192.stdout: "blocklisted": false, 2023-10-06T03:30:08.680 INFO:teuthology.orchestra.run.smithi192.stdout: "fs_name": "cephfs" 2023-10-06T03:30:08.680 INFO:teuthology.orchestra.run.smithi192.stdout:} 2023-10-06T03:30:08.686 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-06T03:30:08.686 DEBUG:teuthology.orchestra.run.smithi110:> stat --file-system '--printf=%T 2023-10-06T03:30:08.686 DEBUG:teuthology.orchestra.run.smithi110:> ' -- /home/ubuntu/cephtest/mnt.0 2023-10-06T03:30:08.695 INFO:teuthology.orchestra.run.smithi110.stdout:fuseblk 2023-10-06T03:30:08.695 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2023-10-06T03:30:08.695 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-06T03:30:08.695 DEBUG:teuthology.orchestra.run.smithi110:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2023-10-06T03:30:08.710 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-06T03:30:08.710 DEBUG:teuthology.orchestra.run.smithi110:> stat --file-system '--printf=%T 2023-10-06T03:30:08.711 DEBUG:teuthology.orchestra.run.smithi110:> ' -- /home/ubuntu/cephtest/mnt.2 2023-10-06T03:30:08.759 INFO:teuthology.orchestra.run.smithi110.stdout:fuseblk 2023-10-06T03:30:08.759 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.2 2023-10-06T03:30:08.759 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-06T03:30:08.759 DEBUG:teuthology.orchestra.run.smithi110:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.2 2023-10-06T03:30:08.771 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-06T03:30:08.771 DEBUG:teuthology.orchestra.run.smithi192:> stat --file-system '--printf=%T 2023-10-06T03:30:08.771 DEBUG:teuthology.orchestra.run.smithi192:> ' -- /home/ubuntu/cephtest/mnt.1 2023-10-06T03:30:08.780 INFO:teuthology.orchestra.run.smithi192.stdout:fuseblk 2023-10-06T03:30:08.780 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2023-10-06T03:30:08.780 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-06T03:30:08.780 DEBUG:teuthology.orchestra.run.smithi192:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2023-10-06T03:30:08.836 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-06T03:30:08.836 DEBUG:teuthology.orchestra.run.smithi192:> stat --file-system '--printf=%T 2023-10-06T03:30:08.836 DEBUG:teuthology.orchestra.run.smithi192:> ' -- /home/ubuntu/cephtest/mnt.3 2023-10-06T03:30:08.844 INFO:teuthology.orchestra.run.smithi192.stdout:fuseblk 2023-10-06T03:30:08.844 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.3 2023-10-06T03:30:08.844 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-06T03:30:08.844 DEBUG:teuthology.orchestra.run.smithi192:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.3 2023-10-06T03:30:08.857 INFO:teuthology.run_tasks:Running task cephfs_test_runner... 2023-10-06T03:30:08.873 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-06T03:30:08.989 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:08.985+0000 7f2ef243b700 1 Processor -- start 2023-10-06T03:30:08.989 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:08.985+0000 7f2ef243b700 1 -- start start 2023-10-06T03:30:08.990 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:08.985+0000 7f2ef243b700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f2eec15d420 0x7f2eec0c7b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:08.990 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:08.985+0000 7f2ef243b700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2eec0ca980 0x7f2eec06b7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:08.990 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:08.985+0000 7f2ef243b700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2eec06aef0 con 0x7f2eec069d70 2023-10-06T03:30:08.991 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:08.985+0000 7f2ef243b700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2eec06a2e0 con 0x7f2eec15d420 2023-10-06T03:30:08.991 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:08.985+0000 7f2ef243b700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2eec06a5c0 con 0x7f2eec0ca980 2023-10-06T03:30:08.991 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:08.985+0000 7f2eebfff700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f2eec15d420 0x7f2eec0c7b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:08.991 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:08.985+0000 7f2eeb7fe700 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f2eec069d70 0x7f2eec0ca270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:59428/0 (socket says 172.21.15.110:59428) 2023-10-06T03:30:08.991 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:08.985+0000 7f2eeb7fe700 1 -- 172.21.15.110:0/1412492615 learned_addr learned my addr 172.21.15.110:0/1412492615 (peer_addr_for_me v1:172.21.15.110:0/0) 2023-10-06T03:30:08.991 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:08.985+0000 7f2ef09d8700 1 --2- 172.21.15.110:0/1412492615 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2eec0ca980 0x7f2eec06b7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:08.992 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:08.985+0000 7f2eeaffd700 1 -- 172.21.15.110:0/1412492615 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1484954030 0 0) 0x7f2eec06aef0 con 0x7f2eec069d70 2023-10-06T03:30:08.992 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:08.985+0000 7f2eeaffd700 1 -- 172.21.15.110:0/1412492615 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2ecc003570 con 0x7f2eec069d70 2023-10-06T03:30:08.993 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:08.985+0000 7f2eebfff700 1 -- 172.21.15.110:0/1412492615 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2eec0ca980 msgr2=0x7f2eec06b7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:30:08.993 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:08.985+0000 7f2eebfff700 1 --2- 172.21.15.110:0/1412492615 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2eec0ca980 0x7f2eec06b7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:08.993 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:08.985+0000 7f2eebfff700 1 -- 172.21.15.110:0/1412492615 >> v1:172.21.15.110:6789/0 conn(0x7f2eec069d70 legacy=0x7f2eec0ca270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:08.994 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:08.985+0000 7f2eebfff700 1 -- 172.21.15.110:0/1412492615 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2eec06a8a0 con 0x7f2eec15d420 2023-10-06T03:30:08.994 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:08.989+0000 7f2eebfff700 1 --2- 172.21.15.110:0/1412492615 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2eec15d420 0x7f2eec0c7b90 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f2edc0098e0 tx=0x7f2edc0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c6584588811d973 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:30:08.994 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:08.989+0000 7f2eeaffd700 1 -- 172.21.15.110:0/1412492615 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2edc011630 con 0x7f2eec15d420 2023-10-06T03:30:08.994 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:08.989+0000 7f2eeaffd700 1 -- 172.21.15.110:0/1412492615 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2edc011790 con 0x7f2eec15d420 2023-10-06T03:30:08.994 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:08.989+0000 7f2eeaffd700 1 -- 172.21.15.110:0/1412492615 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2edc011b10 con 0x7f2eec15d420 2023-10-06T03:30:08.994 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:08.989+0000 7f2ef243b700 1 -- 172.21.15.110:0/1412492615 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2eec15d420 msgr2=0x7f2eec0c7b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:08.994 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:08.989+0000 7f2ef243b700 1 --2- 172.21.15.110:0/1412492615 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2eec15d420 0x7f2eec0c7b90 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f2edc0098e0 tx=0x7f2edc0091a0 comp rx=0 tx=0).stop 2023-10-06T03:30:08.994 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:08.989+0000 7f2ef243b700 1 -- 172.21.15.110:0/1412492615 shutdown_connections 2023-10-06T03:30:08.994 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:08.989+0000 7f2ef243b700 1 --2- 172.21.15.110:0/1412492615 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2eec0ca980 0x7f2eec06b7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:08.994 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:08.989+0000 7f2ef243b700 1 --2- 172.21.15.110:0/1412492615 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2eec15d420 0x7f2eec0c7b90 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:08.994 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:08.989+0000 7f2ef243b700 1 -- 172.21.15.110:0/1412492615 >> 172.21.15.110:0/1412492615 conn(0x7f2eec0cbd00 msgr2=0x7f2eec0cc100 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:30:08.995 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:08.989+0000 7f2ef243b700 1 -- 172.21.15.110:0/1412492615 shutdown_connections 2023-10-06T03:30:08.995 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:08.989+0000 7f2ef243b700 1 -- 172.21.15.110:0/1412492615 wait complete. 2023-10-06T03:30:08.995 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:08.993+0000 7f2ef243b700 1 Processor -- start 2023-10-06T03:30:08.995 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:08.993+0000 7f2ef243b700 1 -- start start 2023-10-06T03:30:08.997 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:08.993+0000 7f2ef243b700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2eec069d70 0x7f2eec148ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:08.998 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:08.993+0000 7f2ef243b700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2eec0ca980 0x7f2eec0c8630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:08.998 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:08.993+0000 7f2ef243b700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2eec15d420 0x7f2eec0c8b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:08.998 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:08.993+0000 7f2ef243b700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2eec15a990 con 0x7f2eec15d420 2023-10-06T03:30:08.998 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:08.993+0000 7f2ef243b700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2eec15fea0 con 0x7f2eec0ca980 2023-10-06T03:30:08.998 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:08.993+0000 7f2ef243b700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2eec15f290 con 0x7f2eec069d70 2023-10-06T03:30:08.998 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:08.993+0000 7f2ef09d8700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2eec15d420 0x7f2eec0c8b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:08.998 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:08.993+0000 7f2eebfff700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2eec069d70 0x7f2eec148ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:08.998 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:08.993+0000 7f2ef09d8700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2eec15d420 0x7f2eec0c8b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:40464/0 (socket says 172.21.15.110:40464) 2023-10-06T03:30:08.998 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:08.993+0000 7f2ef09d8700 1 -- 172.21.15.110:0/629440915 learned_addr learned my addr 172.21.15.110:0/629440915 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:30:08.999 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:08.993+0000 7f2eeb7fe700 1 --2- 172.21.15.110:0/629440915 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2eec0ca980 0x7f2eec0c8630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:09.000 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:08.993+0000 7f2ef09d8700 1 -- 172.21.15.110:0/629440915 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2eec069d70 msgr2=0x7f2eec148ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:09.000 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:08.993+0000 7f2ef09d8700 1 --2- 172.21.15.110:0/629440915 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2eec069d70 0x7f2eec148ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:09.000 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:08.993+0000 7f2ef09d8700 1 -- 172.21.15.110:0/629440915 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2eec0ca980 msgr2=0x7f2eec0c8630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:09.000 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:08.993+0000 7f2ef09d8700 1 --2- 172.21.15.110:0/629440915 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2eec0ca980 0x7f2eec0c8630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:09.001 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:08.993+0000 7f2ef09d8700 1 -- 172.21.15.110:0/629440915 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2edc009420 con 0x7f2eec15d420 2023-10-06T03:30:09.001 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:08.993+0000 7f2eebfff700 1 --2- 172.21.15.110:0/629440915 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2eec069d70 0x7f2eec148ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-06T03:30:09.001 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:08.993+0000 7f2ef09d8700 1 --2- 172.21.15.110:0/629440915 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2eec15d420 0x7f2eec0c8b70 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f2ee4009ec0 tx=0x7f2ee400b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:30:09.001 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:08.993+0000 7f2ee3fff700 1 -- 172.21.15.110:0/629440915 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ee400cb70 con 0x7f2eec15d420 2023-10-06T03:30:09.001 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:08.993+0000 7f2ee3fff700 1 -- 172.21.15.110:0/629440915 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ee400f040 con 0x7f2eec15d420 2023-10-06T03:30:09.001 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:08.993+0000 7f2ef243b700 1 -- 172.21.15.110:0/629440915 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2eec15f570 con 0x7f2eec15d420 2023-10-06T03:30:09.001 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:08.993+0000 7f2ee3fff700 1 -- 172.21.15.110:0/629440915 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ee4014480 con 0x7f2eec15d420 2023-10-06T03:30:09.001 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:08.993+0000 7f2ef243b700 1 -- 172.21.15.110:0/629440915 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2eec15fb30 con 0x7f2eec15d420 2023-10-06T03:30:09.002 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:08.997+0000 7f2ef243b700 1 -- 172.21.15.110:0/629440915 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2eec160050 con 0x7f2eec15d420 2023-10-06T03:30:09.002 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:08.997+0000 7f2ee3fff700 1 -- 172.21.15.110:0/629440915 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7f2ee40147f0 con 0x7f2eec15d420 2023-10-06T03:30:09.003 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:08.997+0000 7f2ee3fff700 1 --2- 172.21.15.110:0/629440915 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f2ed0060550 0x7f2ed0062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:09.003 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:08.997+0000 7f2ee3fff700 1 -- 172.21.15.110:0/629440915 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6809+0+0 (secure 0 0 0) 0x7f2ee407d860 con 0x7f2eec15d420 2023-10-06T03:30:09.003 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:08.997+0000 7f2eebfff700 1 --2- 172.21.15.110:0/629440915 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f2ed0060550 0x7f2ed0062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:09.003 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:08.997+0000 7f2eebfff700 1 --2- 172.21.15.110:0/629440915 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f2ed0060550 0x7f2ed0062a00 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f2edc0091e0 tx=0x7f2edc017000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:30:09.011 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:09.005+0000 7f2ee3fff700 1 -- 172.21.15.110:0/629440915 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f2eec160050 con 0x7f2eec15d420 2023-10-06T03:30:09.175 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:09.173+0000 7f2ef243b700 1 -- 172.21.15.110:0/629440915 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2eec160310 con 0x7f2eec15d420 2023-10-06T03:30:09.176 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:09.173+0000 7f2ee3fff700 1 -- 172.21.15.110:0/629440915 <== mon.0 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v28) v1 ==== 74+0+18755 (secure 0 0 0) 0x7f2ee4018090 con 0x7f2eec15d420 2023-10-06T03:30:09.177 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-10-06T03:30:09.177 INFO:teuthology.orchestra.run.smithi110.stdout:{"epoch":28,"fsid":"93b8f6ac-4c92-4b45-b8e4-b59617024aa2","created":"2023-10-06T03:29:10.519583+0000","modified":"2023-10-06T03:29:38.951740+0000","last_up_change":"2023-10-06T03:29:21.960511+0000","last_in_change":"2023-10-06T03:29:12.560436+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-06T03:29:20.703743+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2023-10-06T03:29:28.307281+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2023-10-06T03:29:32.247251+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data","create_time":"2023-10-06T03:29:32.880086+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}}},{"pool":5,"pool_name":"cephfs_data_ec","create_time":"2023-10-06T03:29:36.613779+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"28","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"5866eb6c-2adc-46e4-be7a-e5bcda0f4065","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6824","nonce":33394},{"type":"v1","addr":"172.21.15.110:6825","nonce":33394}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6826","nonce":33394},{"type":"v1","addr":"172.21.15.110:6827","nonce":33394}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6830","nonce":33394},{"type":"v1","addr":"172.21.15.110:6831","nonce":33394}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6828","nonce":33394},{"type":"v1","addr":"172.21.15.110:6829","nonce":33394}]},"public_addr":"172.21.15.110:6825/33394","cluster_addr":"172.21.15.110:6827/33394","heartbeat_back_addr":"172.21.15.110:6831/33394","heartbeat_front_addr":"172.21.15.110:6829/33394","state":["exists","up"]},{"osd":1,"uuid":"a38de118-062b-4a05-9782-2ceddf6ff4df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6800","nonce":33396},{"type":"v1","addr":"172.21.15.110:6801","nonce":33396}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6802","nonce":33396},{"type":"v1","addr":"172.21.15.110:6803","nonce":33396}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6806","nonce":33396},{"type":"v1","addr":"172.21.15.110:6807","nonce":33396}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6804","nonce":33396},{"type":"v1","addr":"172.21.15.110:6805","nonce":33396}]},"public_addr":"172.21.15.110:6801/33396","cluster_addr":"172.21.15.110:6803/33396","heartbeat_back_addr":"172.21.15.110:6807/33396","heartbeat_front_addr":"172.21.15.110:6805/33396","state":["exists","up"]},{"osd":2,"uuid":"56e39be2-3407-4b69-877e-cf16617b06d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6816","nonce":33397},{"type":"v1","addr":"172.21.15.110:6817","nonce":33397}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6818","nonce":33397},{"type":"v1","addr":"172.21.15.110:6819","nonce":33397}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6822","nonce":33397},{"type":"v1","addr":"172.21.15.110:6823","nonce":33397}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6820","nonce":33397},{"type":"v1","addr":"172.21.15.110:6821","nonce":33397}]},"public_addr":"172.21.15.110:6817/33397","cluster_addr":"172.21.15.110:6819/33397","heartbeat_back_addr":"172.21.15.110:6823/33397","heartbeat_front_addr":"172.21.15.110:6821/33397","state":["exists","up"]},{"osd":3,"uuid":"01c82d13-c1d4-4bde-b55e-3e0089b40115","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6808","nonce":33395},{"type":"v1","addr":"172.21.15.110:6809","nonce":33395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6810","nonce":33395},{"type":"v1","addr":"172.21.15.110:6811","nonce":33395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6814","nonce":33395},{"type":"v1","addr":"172.21.15.110:6815","nonce":33395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6812","nonce":33395},{"type":"v1","addr":"172.21.15.110:6813","nonce":33395}]},"public_addr":"172.21.15.110:6809/33395","cluster_addr":"172.21.15.110:6811/33395","heartbeat_back_addr":"172.21.15.110:6815/33395","heartbeat_front_addr":"172.21.15.110:6813/33395","state":["exists","up"]},{"osd":4,"uuid":"86983430-5fa3-4d17-998f-03e2cd843237","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":33683},{"type":"v1","addr":"172.21.15.192:6817","nonce":33683}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":33683},{"type":"v1","addr":"172.21.15.192:6819","nonce":33683}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":33683},{"type":"v1","addr":"172.21.15.192:6823","nonce":33683}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6820","nonce":33683},{"type":"v1","addr":"172.21.15.192:6821","nonce":33683}]},"public_addr":"172.21.15.192:6817/33683","cluster_addr":"172.21.15.192:6819/33683","heartbeat_back_addr":"172.21.15.192:6823/33683","heartbeat_front_addr":"172.21.15.192:6821/33683","state":["exists","up"]},{"osd":5,"uuid":"6945257f-821d-4619-a63f-ebe857eba6f4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":33684},{"type":"v1","addr":"172.21.15.192:6801","nonce":33684}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":33684},{"type":"v1","addr":"172.21.15.192:6803","nonce":33684}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":33684},{"type":"v1","addr":"172.21.15.192:6807","nonce":33684}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6804","nonce":33684},{"type":"v1","addr":"172.21.15.192:6805","nonce":33684}]},"public_addr":"172.21.15.192:6801/33684","cluster_addr":"172.21.15.192:6803/33684","heartbeat_back_addr":"172.21.15.192:6807/33684","heartbeat_front_addr":"172.21.15.192:6805/33684","state":["exists","up"]},{"osd":6,"uuid":"7854d1ae-7060-4780-8f97-73cea5e7a12f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6824","nonce":33686},{"type":"v1","addr":"172.21.15.192:6825","nonce":33686}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":33686},{"type":"v1","addr":"172.21.15.192:6827","nonce":33686}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":33686},{"type":"v1","addr":"172.21.15.192:6831","nonce":33686}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6828","nonce":33686},{"type":"v1","addr":"172.21.15.192:6829","nonce":33686}]},"public_addr":"172.21.15.192:6825/33686","cluster_addr":"172.21.15.192:6827/33686","heartbeat_back_addr":"172.21.15.192:6831/33686","heartbeat_front_addr":"172.21.15.192:6829/33686","state":["exists","up"]},{"osd":7,"uuid":"f1ebdfd5-b859-482b-9f97-c18c4a26f445","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6808","nonce":33685},{"type":"v1","addr":"172.21.15.192:6809","nonce":33685}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":33685},{"type":"v1","addr":"172.21.15.192:6811","nonce":33685}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":33685},{"type":"v1","addr":"172.21.15.192:6815","nonce":33685}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6812","nonce":33685},{"type":"v1","addr":"172.21.15.192:6813","nonce":33685}]},"public_addr":"172.21.15.192:6809/33685","cluster_addr":"172.21.15.192:6811/33685","heartbeat_back_addr":"172.21.15.192:6815/33685","heartbeat_front_addr":"172.21.15.192:6813/33685","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T03:29:18.002048+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T03:29:17.478576+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T03:29:18.034944+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T03:29:17.628513+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T03:29:18.047338+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T03:29:17.772349+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T03:29:17.883958+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T03:29:18.151838+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-06T03:30:09.179 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:09.173+0000 7f2ef243b700 1 -- 172.21.15.110:0/629440915 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f2ed0060550 msgr2=0x7f2ed0062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:09.179 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:09.173+0000 7f2ef243b700 1 --2- 172.21.15.110:0/629440915 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f2ed0060550 0x7f2ed0062a00 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f2edc0091e0 tx=0x7f2edc017000 comp rx=0 tx=0).stop 2023-10-06T03:30:09.179 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:09.173+0000 7f2ef243b700 1 -- 172.21.15.110:0/629440915 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2eec15d420 msgr2=0x7f2eec0c8b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:09.179 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:09.173+0000 7f2ef243b700 1 --2- 172.21.15.110:0/629440915 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2eec15d420 0x7f2eec0c8b70 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f2ee4009ec0 tx=0x7f2ee400b040 comp rx=0 tx=0).stop 2023-10-06T03:30:09.179 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:09.173+0000 7f2ef243b700 1 -- 172.21.15.110:0/629440915 shutdown_connections 2023-10-06T03:30:09.180 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:09.173+0000 7f2ef243b700 1 --2- 172.21.15.110:0/629440915 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f2ed0060550 0x7f2ed0062a00 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:09.180 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:09.173+0000 7f2ef243b700 1 --2- 172.21.15.110:0/629440915 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2eec15d420 0x7f2eec0c8b70 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:09.180 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:09.173+0000 7f2ef243b700 1 --2- 172.21.15.110:0/629440915 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2eec0ca980 0x7f2eec0c8630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:09.180 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:09.173+0000 7f2ef243b700 1 --2- 172.21.15.110:0/629440915 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2eec069d70 0x7f2eec148ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:09.180 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:09.173+0000 7f2ef243b700 1 -- 172.21.15.110:0/629440915 >> 172.21.15.110:0/629440915 conn(0x7f2eec0cbd00 msgr2=0x7f2eec15b270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:30:09.180 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:09.177+0000 7f2ef243b700 1 -- 172.21.15.110:0/629440915 shutdown_connections 2023-10-06T03:30:09.180 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:09.177+0000 7f2ef243b700 1 -- 172.21.15.110:0/629440915 wait complete. 2023-10-06T03:30:09.191 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-06T03:29:20.703743+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-10-06T03:29:28.307281+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 3, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-06T03:29:32.247251+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 4, 'pool_name': 'cephfs_data', 'create_time': '2023-10-06T03:29:32.880086+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}, {'pool': 5, 'pool_name': 'cephfs_data_ec', 'create_time': '2023-10-06T03:29:36.613779+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '28', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-10-06T03:30:09.192 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-06T03:30:09.309 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:09.305+0000 7fa5bc88c700 1 Processor -- start 2023-10-06T03:30:09.309 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:09.305+0000 7fa5bc88c700 1 -- start start 2023-10-06T03:30:09.311 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:09.305+0000 7fa5bc88c700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fa5b806b7c0 0x7fa5b80ce170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:09.311 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:09.305+0000 7fa5bc88c700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa5b806c2a0 0x7fa5b80b0a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:09.311 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:09.305+0000 7fa5bc88c700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5b80cfa00 con 0x7fa5b80ce6b0 2023-10-06T03:30:09.311 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:09.305+0000 7fa5bc88c700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa5b80cb1c0 con 0x7fa5b806b7c0 2023-10-06T03:30:09.311 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:09.305+0000 7fa5bc88c700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5b80cedf0 con 0x7fa5b806c2a0 2023-10-06T03:30:09.312 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:09.309+0000 7fa5b77fe700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fa5b806b7c0 0x7fa5b80ce170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:09.312 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:09.309+0000 7fa5b77fe700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fa5b806b7c0 0x7fa5b80ce170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.110:40672/0 (socket says 172.21.15.110:40672) 2023-10-06T03:30:09.312 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:09.309+0000 7fa5b77fe700 1 -- 172.21.15.110:0/1015328341 learned_addr learned my addr 172.21.15.110:0/1015328341 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:30:09.312 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:09.309+0000 7fa5b7fff700 1 --2- 172.21.15.110:0/1015328341 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa5b806c2a0 0x7fa5b80b0a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:09.312 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:09.309+0000 7fa5b77fe700 1 -- 172.21.15.110:0/1015328341 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa5b806c2a0 msgr2=0x7fa5b80b0a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:30:09.313 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:09.309+0000 7fa5b77fe700 1 --2- 172.21.15.110:0/1015328341 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa5b806c2a0 0x7fa5b80b0a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:09.313 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:09.309+0000 7fa5b77fe700 1 -- 172.21.15.110:0/1015328341 >> v1:172.21.15.192:6789/0 conn(0x7fa5b80ce6b0 legacy=0x7fa5b806bb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:09.313 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:09.309+0000 7fa5b77fe700 1 -- 172.21.15.110:0/1015328341 --> v2:172.21.15.192:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5b80cf0d0 con 0x7fa5b806b7c0 2023-10-06T03:30:09.313 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:09.309+0000 7fa5b77fe700 1 --2- 172.21.15.110:0/1015328341 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa5b806b7c0 0x7fa5b80ce170 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fa5a8009f20 tx=0x7fa5a8009840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5951798682be5698 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:30:09.314 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:09.309+0000 7fa5b67fc700 1 -- 172.21.15.110:0/1015328341 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5a800e070 con 0x7fa5b806b7c0 2023-10-06T03:30:09.314 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:09.309+0000 7fa5b67fc700 1 -- 172.21.15.110:0/1015328341 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5a800b9d0 con 0x7fa5b806b7c0 2023-10-06T03:30:09.314 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:09.309+0000 7fa5b67fc700 1 -- 172.21.15.110:0/1015328341 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5a80092e0 con 0x7fa5b806b7c0 2023-10-06T03:30:09.314 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:09.309+0000 7fa5bc88c700 1 -- 172.21.15.110:0/1015328341 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa5b806b7c0 msgr2=0x7fa5b80ce170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:09.314 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:09.309+0000 7fa5bc88c700 1 --2- 172.21.15.110:0/1015328341 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa5b806b7c0 0x7fa5b80ce170 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fa5a8009f20 tx=0x7fa5a8009840 comp rx=0 tx=0).stop 2023-10-06T03:30:09.314 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:09.309+0000 7fa5bc88c700 1 -- 172.21.15.110:0/1015328341 shutdown_connections 2023-10-06T03:30:09.314 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:09.309+0000 7fa5bc88c700 1 --2- 172.21.15.110:0/1015328341 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa5b806c2a0 0x7fa5b80b0a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:09.314 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:09.309+0000 7fa5bc88c700 1 --2- 172.21.15.110:0/1015328341 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa5b806b7c0 0x7fa5b80ce170 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:09.314 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:09.309+0000 7fa5bc88c700 1 -- 172.21.15.110:0/1015328341 >> 172.21.15.110:0/1015328341 conn(0x7fa5b80cfea0 msgr2=0x7fa5b80d02a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:30:09.315 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:09.309+0000 7fa5bc88c700 1 -- 172.21.15.110:0/1015328341 shutdown_connections 2023-10-06T03:30:09.315 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:09.309+0000 7fa5bc88c700 1 -- 172.21.15.110:0/1015328341 wait complete. 2023-10-06T03:30:09.318 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:09.313+0000 7fa5bc88c700 1 Processor -- start 2023-10-06T03:30:09.318 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:09.313+0000 7fa5bc88c700 1 -- start start 2023-10-06T03:30:09.318 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:09.313+0000 7fa5bc88c700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa5b806b7c0 0x7fa5b81457d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:09.318 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:09.313+0000 7fa5bc88c700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa5b806c2a0 0x7fa5b81560b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:09.319 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:09.313+0000 7fa5bc88c700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa5b80ce6b0 0x7fa5b8159d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:09.319 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:09.313+0000 7fa5bc88c700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5b80b3850 con 0x7fa5b80ce6b0 2023-10-06T03:30:09.319 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:09.313+0000 7fa5bc88c700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5b80b2c40 con 0x7fa5b806c2a0 2023-10-06T03:30:09.319 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:09.313+0000 7fa5bc88c700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5b80b2f20 con 0x7fa5b806b7c0 2023-10-06T03:30:09.319 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:09.313+0000 7fa5b7fff700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa5b80ce6b0 0x7fa5b8159d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:09.319 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:09.313+0000 7fa5b7fff700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa5b80ce6b0 0x7fa5b8159d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:40480/0 (socket says 172.21.15.110:40480) 2023-10-06T03:30:09.320 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:09.313+0000 7fa5b7fff700 1 -- 172.21.15.110:0/1934826371 learned_addr learned my addr 172.21.15.110:0/1934826371 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:30:09.320 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:09.313+0000 7fa5b77fe700 1 --2- 172.21.15.110:0/1934826371 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa5b806b7c0 0x7fa5b81457d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:09.320 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:09.313+0000 7fa5b6ffd700 1 --2- 172.21.15.110:0/1934826371 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa5b806c2a0 0x7fa5b81560b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:09.320 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:09.313+0000 7fa5b7fff700 1 -- 172.21.15.110:0/1934826371 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa5b806b7c0 msgr2=0x7fa5b81457d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:09.320 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:09.313+0000 7fa5b7fff700 1 --2- 172.21.15.110:0/1934826371 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa5b806b7c0 0x7fa5b81457d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:09.320 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:09.313+0000 7fa5b7fff700 1 -- 172.21.15.110:0/1934826371 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa5b806c2a0 msgr2=0x7fa5b81560b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:09.320 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:09.313+0000 7fa5b7fff700 1 --2- 172.21.15.110:0/1934826371 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa5b806c2a0 0x7fa5b81560b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:09.320 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:09.313+0000 7fa5b7fff700 1 -- 172.21.15.110:0/1934826371 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5a8009ae0 con 0x7fa5b80ce6b0 2023-10-06T03:30:09.320 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:09.313+0000 7fa5b77fe700 1 --2- 172.21.15.110:0/1934826371 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa5b806b7c0 0x7fa5b81457d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-06T03:30:09.321 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:09.313+0000 7fa5b7fff700 1 --2- 172.21.15.110:0/1934826371 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa5b80ce6b0 0x7fa5b8159d80 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fa5b000ea50 tx=0x7fa5b00091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:30:09.322 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:09.313+0000 7fa5af7fe700 1 -- 172.21.15.110:0/1934826371 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5b000c920 con 0x7fa5b80ce6b0 2023-10-06T03:30:09.322 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:09.313+0000 7fa5af7fe700 1 -- 172.21.15.110:0/1934826371 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5b000caa0 con 0x7fa5b80ce6b0 2023-10-06T03:30:09.322 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:09.313+0000 7fa5af7fe700 1 -- 172.21.15.110:0/1934826371 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5b000ce60 con 0x7fa5b80ce6b0 2023-10-06T03:30:09.322 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:09.313+0000 7fa5bc88c700 1 -- 172.21.15.110:0/1934826371 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5b80b3200 con 0x7fa5b80ce6b0 2023-10-06T03:30:09.322 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:09.313+0000 7fa5bc88c700 1 -- 172.21.15.110:0/1934826371 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5b8145d10 con 0x7fa5b80ce6b0 2023-10-06T03:30:09.322 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:09.317+0000 7fa5bc88c700 1 -- 172.21.15.110:0/1934826371 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa5b8070ff0 con 0x7fa5b80ce6b0 2023-10-06T03:30:09.322 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:09.317+0000 7fa5af7fe700 1 -- 172.21.15.110:0/1934826371 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7fa5b0013680 con 0x7fa5b80ce6b0 2023-10-06T03:30:09.323 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:09.317+0000 7fa5af7fe700 1 --2- 172.21.15.110:0/1934826371 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fa598060550 0x7fa598062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:09.323 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:09.317+0000 7fa5b77fe700 1 --2- 172.21.15.110:0/1934826371 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fa598060550 0x7fa598062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:09.323 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:09.317+0000 7fa5af7fe700 1 -- 172.21.15.110:0/1934826371 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6809+0+0 (secure 0 0 0) 0x7fa5b007d050 con 0x7fa5b80ce6b0 2023-10-06T03:30:09.323 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:09.317+0000 7fa5b77fe700 1 --2- 172.21.15.110:0/1934826371 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fa598060550 0x7fa598062a00 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fa5a8009880 tx=0x7fa5a8018000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:30:09.329 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:09.325+0000 7fa5af7fe700 1 -- 172.21.15.110:0/1934826371 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fa5b00acbe0 con 0x7fa5b80ce6b0 2023-10-06T03:30:09.490 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:09.485+0000 7fa5bc88c700 1 -- 172.21.15.110:0/1934826371 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fa5b80cf3b0 con 0x7fa5b80ce6b0 2023-10-06T03:30:09.491 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:09.489+0000 7fa5af7fe700 1 -- 172.21.15.110:0/1934826371 <== mon.0 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v28) v1 ==== 93+0+10 (secure 0 0 0) 0x7fa5b004d150 con 0x7fa5b80ce6b0 2023-10-06T03:30:09.491 INFO:teuthology.orchestra.run.smithi110.stdout:pg_num: 1 2023-10-06T03:30:09.494 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:09.489+0000 7fa5bc88c700 1 -- 172.21.15.110:0/1934826371 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fa598060550 msgr2=0x7fa598062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:09.494 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:09.489+0000 7fa5bc88c700 1 --2- 172.21.15.110:0/1934826371 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fa598060550 0x7fa598062a00 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fa5a8009880 tx=0x7fa5a8018000 comp rx=0 tx=0).stop 2023-10-06T03:30:09.494 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:09.489+0000 7fa5bc88c700 1 -- 172.21.15.110:0/1934826371 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa5b80ce6b0 msgr2=0x7fa5b8159d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:09.494 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:09.489+0000 7fa5bc88c700 1 --2- 172.21.15.110:0/1934826371 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa5b80ce6b0 0x7fa5b8159d80 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fa5b000ea50 tx=0x7fa5b00091a0 comp rx=0 tx=0).stop 2023-10-06T03:30:09.494 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:09.489+0000 7fa5bc88c700 1 -- 172.21.15.110:0/1934826371 shutdown_connections 2023-10-06T03:30:09.494 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:09.489+0000 7fa5bc88c700 1 --2- 172.21.15.110:0/1934826371 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fa598060550 0x7fa598062a00 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:09.494 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:09.489+0000 7fa5bc88c700 1 --2- 172.21.15.110:0/1934826371 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa5b80ce6b0 0x7fa5b8159d80 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:09.494 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:09.489+0000 7fa5bc88c700 1 --2- 172.21.15.110:0/1934826371 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa5b806c2a0 0x7fa5b81560b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:09.494 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:09.489+0000 7fa5bc88c700 1 --2- 172.21.15.110:0/1934826371 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa5b806b7c0 0x7fa5b81457d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:09.495 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:09.489+0000 7fa5bc88c700 1 -- 172.21.15.110:0/1934826371 >> 172.21.15.110:0/1934826371 conn(0x7fa5b80cfea0 msgr2=0x7fa5b8153b50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:30:09.495 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:09.489+0000 7fa5bc88c700 1 -- 172.21.15.110:0/1934826371 shutdown_connections 2023-10-06T03:30:09.495 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:09.489+0000 7fa5bc88c700 1 -- 172.21.15.110:0/1934826371 wait complete. 2023-10-06T03:30:09.505 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-10-06T03:30:09.612 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:09.609+0000 7fc8b4e4c700 1 Processor -- start 2023-10-06T03:30:09.612 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:09.609+0000 7fc8b4e4c700 1 -- start start 2023-10-06T03:30:09.613 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:09.609+0000 7fc8b4e4c700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc8b00beb50 0x7fc8b006acb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:09.614 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:09.609+0000 7fc8b4e4c700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8b006a0e0 con 0x7fc8b006b7c0 2023-10-06T03:30:09.614 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:09.609+0000 7fc8b4e4c700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8b006bdf0 con 0x7fc8b006ce70 2023-10-06T03:30:09.614 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:09.609+0000 7fc8b4e4c700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8b006bc10 con 0x7fc8b00beb50 2023-10-06T03:30:09.614 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:09.609+0000 7fc8aed9d700 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7fc8b006b7c0 0x7fc8b00c6810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:59430/0 (socket says 172.21.15.110:59430) 2023-10-06T03:30:09.614 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:09.609+0000 7fc8aed9d700 1 -- 172.21.15.110:0/1006816832 learned_addr learned my addr 172.21.15.110:0/1006816832 (peer_addr_for_me v1:172.21.15.110:0/0) 2023-10-06T03:30:09.614 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:09.609+0000 7fc8ae59c700 1 --2- 172.21.15.110:0/1006816832 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc8b00beb50 0x7fc8b006acb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:09.614 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:09.609+0000 7fc8ad59a700 1 -- 172.21.15.110:0/1006816832 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1497294096 0 0) 0x7fc8b006a0e0 con 0x7fc8b006b7c0 2023-10-06T03:30:09.614 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:09.609+0000 7fc8ad59a700 1 -- 172.21.15.110:0/1006816832 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc89c003570 con 0x7fc8b006b7c0 2023-10-06T03:30:09.616 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:09.609+0000 7fc8ad59a700 1 -- 172.21.15.110:0/1006816832 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2323191885 0 0) 0x7fc8b006bdf0 con 0x7fc8b006ce70 2023-10-06T03:30:09.616 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:09.609+0000 7fc8ad59a700 1 -- 172.21.15.110:0/1006816832 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc8b006a0e0 con 0x7fc8b006ce70 2023-10-06T03:30:09.616 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:09.609+0000 7fc8ae59c700 1 -- 172.21.15.110:0/1006816832 >> v1:172.21.15.192:6789/0 conn(0x7fc8b006ce70 legacy=0x7fc8b00bef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:09.617 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:09.609+0000 7fc8ae59c700 1 -- 172.21.15.110:0/1006816832 >> v1:172.21.15.110:6789/0 conn(0x7fc8b006b7c0 legacy=0x7fc8b00c6810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:09.617 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:09.609+0000 7fc8ae59c700 1 -- 172.21.15.110:0/1006816832 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8b006c0b0 con 0x7fc8b00beb50 2023-10-06T03:30:09.617 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:09.609+0000 7fc8ae59c700 1 --2- 172.21.15.110:0/1006816832 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc8b00beb50 0x7fc8b006acb0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fc8a8009e90 tx=0x7fc8a8009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=21842a3951d00527 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:30:09.617 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:09.609+0000 7fc8ad59a700 1 -- 172.21.15.110:0/1006816832 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8a800e070 con 0x7fc8b00beb50 2023-10-06T03:30:09.617 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:09.609+0000 7fc8ad59a700 1 -- 172.21.15.110:0/1006816832 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8a800bb80 con 0x7fc8b00beb50 2023-10-06T03:30:09.617 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:09.609+0000 7fc8ad59a700 1 -- 172.21.15.110:0/1006816832 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8a8012430 con 0x7fc8b00beb50 2023-10-06T03:30:09.617 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:09.613+0000 7fc8b4e4c700 1 -- 172.21.15.110:0/1006816832 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc8b00beb50 msgr2=0x7fc8b006acb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:09.617 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:09.613+0000 7fc8b4e4c700 1 --2- 172.21.15.110:0/1006816832 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc8b00beb50 0x7fc8b006acb0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fc8a8009e90 tx=0x7fc8a8009780 comp rx=0 tx=0).stop 2023-10-06T03:30:09.617 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:09.613+0000 7fc8b4e4c700 1 -- 172.21.15.110:0/1006816832 shutdown_connections 2023-10-06T03:30:09.618 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:09.613+0000 7fc8b4e4c700 1 --2- 172.21.15.110:0/1006816832 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc8b00beb50 0x7fc8b006acb0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:09.618 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:09.613+0000 7fc8b4e4c700 1 -- 172.21.15.110:0/1006816832 >> 172.21.15.110:0/1006816832 conn(0x7fc8b00cc6d0 msgr2=0x7fc8b00c8670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:30:09.618 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:09.613+0000 7fc8b4e4c700 1 -- 172.21.15.110:0/1006816832 shutdown_connections 2023-10-06T03:30:09.618 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:09.613+0000 7fc8b4e4c700 1 -- 172.21.15.110:0/1006816832 wait complete. 2023-10-06T03:30:09.619 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:09.613+0000 7fc8b4e4c700 1 Processor -- start 2023-10-06T03:30:09.619 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:09.613+0000 7fc8b4e4c700 1 -- start start 2023-10-06T03:30:09.620 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:09.617+0000 7fc8b4e4c700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc8b006b7c0 0x7fc8b015dee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:09.621 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:09.617+0000 7fc8b4e4c700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc8b006ce70 0x7fc8b01501b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:09.621 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:09.617+0000 7fc8b4e4c700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc8b00beb50 0x7fc8b00c1680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:09.621 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:09.617+0000 7fc8b4e4c700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8b016dda0 con 0x7fc8b006b7c0 2023-10-06T03:30:09.621 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:09.617+0000 7fc8b4e4c700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8b016ec00 con 0x7fc8b00beb50 2023-10-06T03:30:09.621 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:09.617+0000 7fc8b4e4c700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8b016dff0 con 0x7fc8b006ce70 2023-10-06T03:30:09.621 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:09.617+0000 7fc8aed9d700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc8b00beb50 0x7fc8b00c1680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:09.621 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:09.617+0000 7fc8add9b700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc8b006ce70 0x7fc8b01501b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:09.622 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:09.617+0000 7fc8ae59c700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc8b006b7c0 0x7fc8b015dee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:09.622 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:09.617+0000 7fc8add9b700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc8b006ce70 0x7fc8b01501b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3301/0 says I am v2:172.21.15.110:33190/0 (socket says 172.21.15.110:33190) 2023-10-06T03:30:09.622 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:09.617+0000 7fc8aed9d700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc8b00beb50 0x7fc8b00c1680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.110:40694/0 (socket says 172.21.15.110:40694) 2023-10-06T03:30:09.622 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:09.617+0000 7fc8add9b700 1 -- 172.21.15.110:0/908566188 learned_addr learned my addr 172.21.15.110:0/908566188 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:30:09.622 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:09.617+0000 7fc8add9b700 1 -- 172.21.15.110:0/908566188 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc8b00beb50 msgr2=0x7fc8b00c1680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:09.622 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:09.617+0000 7fc8add9b700 1 --2- 172.21.15.110:0/908566188 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc8b00beb50 0x7fc8b00c1680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:09.623 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:09.617+0000 7fc8add9b700 1 -- 172.21.15.110:0/908566188 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc8b006b7c0 msgr2=0x7fc8b015dee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:09.623 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:09.617+0000 7fc8add9b700 1 --2- 172.21.15.110:0/908566188 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc8b006b7c0 0x7fc8b015dee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:09.623 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:09.617+0000 7fc8add9b700 1 -- 172.21.15.110:0/908566188 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8a8009a20 con 0x7fc8b006ce70 2023-10-06T03:30:09.623 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:09.617+0000 7fc8aed9d700 1 --2- 172.21.15.110:0/908566188 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc8b00beb50 0x7fc8b00c1680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-06T03:30:09.623 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:09.617+0000 7fc8ae59c700 1 --2- 172.21.15.110:0/908566188 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc8b006b7c0 0x7fc8b015dee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-06T03:30:09.623 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:09.617+0000 7fc8add9b700 1 --2- 172.21.15.110:0/908566188 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc8b006ce70 0x7fc8b01501b0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fc89800c5e0 tx=0x7fc898007d90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:30:09.624 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:09.617+0000 7fc8a67fc700 1 -- 172.21.15.110:0/908566188 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8980105c0 con 0x7fc8b006ce70 2023-10-06T03:30:09.624 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:09.617+0000 7fc8b4e4c700 1 -- 172.21.15.110:0/908566188 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8b016e2d0 con 0x7fc8b006ce70 2023-10-06T03:30:09.625 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:09.617+0000 7fc8a67fc700 1 -- 172.21.15.110:0/908566188 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc898010720 con 0x7fc8b006ce70 2023-10-06T03:30:09.625 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:09.617+0000 7fc8a67fc700 1 -- 172.21.15.110:0/908566188 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc898010aa0 con 0x7fc8b006ce70 2023-10-06T03:30:09.625 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:09.617+0000 7fc8b4e4c700 1 -- 172.21.15.110:0/908566188 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8b016e890 con 0x7fc8b006ce70 2023-10-06T03:30:09.625 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:09.617+0000 7fc8b4e4c700 1 -- 172.21.15.110:0/908566188 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc8b006ff60 con 0x7fc8b006ce70 2023-10-06T03:30:09.625 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:09.621+0000 7fc8a67fc700 1 -- 172.21.15.110:0/908566188 <== mon.2 v2:172.21.15.192:3301/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7fc8980123e0 con 0x7fc8b006ce70 2023-10-06T03:30:09.625 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:09.621+0000 7fc8a67fc700 1 --2- 172.21.15.110:0/908566188 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fc890060550 0x7fc890062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:09.625 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:09.621+0000 7fc8a67fc700 1 -- 172.21.15.110:0/908566188 <== mon.2 v2:172.21.15.192:3301/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6809+0+0 (secure 0 0 0) 0x7fc89807c650 con 0x7fc8b006ce70 2023-10-06T03:30:09.625 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:09.621+0000 7fc8ae59c700 1 --2- 172.21.15.110:0/908566188 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fc890060550 0x7fc890062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:09.626 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:09.621+0000 7fc8ae59c700 1 --2- 172.21.15.110:0/908566188 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fc890060550 0x7fc890062a00 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fc8a80099f0 tx=0x7fc8a8019000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:30:09.635 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:09.633+0000 7fc8a67fc700 1 -- 172.21.15.110:0/908566188 <== mon.2 v2:172.21.15.192:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fc89804c8a0 con 0x7fc8b006ce70 2023-10-06T03:30:09.798 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:09.793+0000 7fc8b4e4c700 1 -- 172.21.15.110:0/908566188 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7fc8b006afa0 con 0x7fc8b006ce70 2023-10-06T03:30:09.799 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:09.793+0000 7fc8a67fc700 1 -- 172.21.15.110:0/908566188 <== mon.2 v2:172.21.15.192:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v28) v1 ==== 92+0+10 (secure 0 0 0) 0x7fc8980180a0 con 0x7fc8b006ce70 2023-10-06T03:30:09.799 INFO:teuthology.orchestra.run.smithi110.stdout:pg_num: 8 2023-10-06T03:30:09.802 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:09.797+0000 7fc8b4e4c700 1 -- 172.21.15.110:0/908566188 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fc890060550 msgr2=0x7fc890062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:09.803 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:09.797+0000 7fc8b4e4c700 1 --2- 172.21.15.110:0/908566188 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fc890060550 0x7fc890062a00 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fc8a80099f0 tx=0x7fc8a8019000 comp rx=0 tx=0).stop 2023-10-06T03:30:09.803 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:09.797+0000 7fc8b4e4c700 1 -- 172.21.15.110:0/908566188 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc8b006ce70 msgr2=0x7fc8b01501b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:09.803 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:09.797+0000 7fc8b4e4c700 1 --2- 172.21.15.110:0/908566188 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc8b006ce70 0x7fc8b01501b0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fc89800c5e0 tx=0x7fc898007d90 comp rx=0 tx=0).stop 2023-10-06T03:30:09.803 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:09.797+0000 7fc8b4e4c700 1 -- 172.21.15.110:0/908566188 shutdown_connections 2023-10-06T03:30:09.804 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:09.797+0000 7fc8b4e4c700 1 --2- 172.21.15.110:0/908566188 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fc890060550 0x7fc890062a00 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:09.804 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:09.797+0000 7fc8b4e4c700 1 --2- 172.21.15.110:0/908566188 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc8b00beb50 0x7fc8b00c1680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:09.804 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:09.797+0000 7fc8b4e4c700 1 --2- 172.21.15.110:0/908566188 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc8b006ce70 0x7fc8b01501b0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:09.804 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:09.797+0000 7fc8b4e4c700 1 --2- 172.21.15.110:0/908566188 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc8b006b7c0 0x7fc8b015dee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:09.804 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:09.797+0000 7fc8b4e4c700 1 -- 172.21.15.110:0/908566188 >> 172.21.15.110:0/908566188 conn(0x7fc8b00cc6d0 msgr2=0x7fc8b00c2160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:30:09.805 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:09.797+0000 7fc8b4e4c700 1 -- 172.21.15.110:0/908566188 shutdown_connections 2023-10-06T03:30:09.805 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:09.797+0000 7fc8b4e4c700 1 -- 172.21.15.110:0/908566188 wait complete. 2023-10-06T03:30:09.814 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-06T03:30:09.923 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:09.917+0000 7fdffd310700 1 Processor -- start 2023-10-06T03:30:09.923 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:09.921+0000 7fdffd310700 1 -- start start 2023-10-06T03:30:09.925 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:09.921+0000 7fdffd310700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fdff8155b70 0x7fdff8155f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:09.925 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:09.921+0000 7fdffd310700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdff8156480 con 0x7fdff806b7c0 2023-10-06T03:30:09.925 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:09.921+0000 7fdffd310700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdff8156750 con 0x7fdff80cac10 2023-10-06T03:30:09.925 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:09.921+0000 7fdffd310700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdff815a1c0 con 0x7fdff8155b70 2023-10-06T03:30:09.925 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:09.921+0000 7fdff77fe700 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7fdff806b7c0 0x7fdff80ca060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:59444/0 (socket says 172.21.15.110:59444) 2023-10-06T03:30:09.925 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:09.921+0000 7fdff6ffd700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fdff8155b70 0x7fdff8155f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:09.925 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:09.921+0000 7fdff77fe700 1 -- 172.21.15.110:0/2700906075 learned_addr learned my addr 172.21.15.110:0/2700906075 (peer_addr_for_me v1:172.21.15.110:0/0) 2023-10-06T03:30:09.927 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:09.921+0000 7fdff6ffd700 1 -- 172.21.15.110:0/2700906075 >> v1:172.21.15.192:6789/0 conn(0x7fdff80cac10 legacy=0x7fdff8159970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:09.929 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:09.921+0000 7fdff6ffd700 1 -- 172.21.15.110:0/2700906075 >> v1:172.21.15.110:6789/0 conn(0x7fdff806b7c0 legacy=0x7fdff80ca060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:09.929 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:09.921+0000 7fdff6ffd700 1 -- 172.21.15.110:0/2700906075 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdff815a4a0 con 0x7fdff8155b70 2023-10-06T03:30:09.929 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:09.921+0000 7fdff6ffd700 1 --2- 172.21.15.110:0/2700906075 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdff8155b70 0x7fdff8155f40 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fdfe8009e60 tx=0x7fdfe8009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c0902174eacb9f2f server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:30:09.929 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:09.921+0000 7fdff5ffb700 1 -- 172.21.15.110:0/2700906075 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfe800e070 con 0x7fdff8155b70 2023-10-06T03:30:09.929 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:09.921+0000 7fdff5ffb700 1 -- 172.21.15.110:0/2700906075 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdfe800bc30 con 0x7fdff8155b70 2023-10-06T03:30:09.929 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:09.921+0000 7fdff5ffb700 1 -- 172.21.15.110:0/2700906075 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfe8012430 con 0x7fdff8155b70 2023-10-06T03:30:09.929 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:09.921+0000 7fdffd310700 1 -- 172.21.15.110:0/2700906075 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdff8155b70 msgr2=0x7fdff8155f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:09.929 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:09.921+0000 7fdffd310700 1 --2- 172.21.15.110:0/2700906075 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdff8155b70 0x7fdff8155f40 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fdfe8009e60 tx=0x7fdfe8009780 comp rx=0 tx=0).stop 2023-10-06T03:30:09.930 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:09.921+0000 7fdffd310700 1 -- 172.21.15.110:0/2700906075 shutdown_connections 2023-10-06T03:30:09.930 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:09.921+0000 7fdffd310700 1 --2- 172.21.15.110:0/2700906075 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdff8155b70 0x7fdff8155f40 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:09.930 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:09.921+0000 7fdffd310700 1 -- 172.21.15.110:0/2700906075 >> 172.21.15.110:0/2700906075 conn(0x7fdff80c8f00 msgr2=0x7fdff80c9300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:30:09.930 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:09.921+0000 7fdffd310700 1 -- 172.21.15.110:0/2700906075 shutdown_connections 2023-10-06T03:30:09.930 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:09.921+0000 7fdffd310700 1 -- 172.21.15.110:0/2700906075 wait complete. 2023-10-06T03:30:09.932 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:09.925+0000 7fdffd310700 1 Processor -- start 2023-10-06T03:30:09.932 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:09.925+0000 7fdffd310700 1 -- start start 2023-10-06T03:30:09.932 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:09.925+0000 7fdffd310700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fdff806b7c0 0x7fdff814f3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:09.932 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:09.925+0000 7fdffd310700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdff80cac10 0x7fdff8154210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:09.932 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:09.925+0000 7fdffd310700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdff8155b70 0x7fdff8154750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:09.932 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:09.925+0000 7fdffd310700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdff80a90a0 con 0x7fdff80cac10 2023-10-06T03:30:09.933 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:09.925+0000 7fdffd310700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdff80ae240 con 0x7fdff8155b70 2023-10-06T03:30:09.933 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:09.925+0000 7fdffd310700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdff80a8a50 con 0x7fdff806b7c0 2023-10-06T03:30:09.933 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:09.925+0000 7fdff67fc700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdff80cac10 0x7fdff8154210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:09.933 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:09.925+0000 7fdff67fc700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdff80cac10 0x7fdff8154210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:40498/0 (socket says 172.21.15.110:40498) 2023-10-06T03:30:09.933 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:09.925+0000 7fdff6ffd700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fdff806b7c0 0x7fdff814f3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:09.933 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:09.925+0000 7fdff67fc700 1 -- 172.21.15.110:0/3618118559 learned_addr learned my addr 172.21.15.110:0/3618118559 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:30:09.933 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:09.925+0000 7fdff77fe700 1 --2- 172.21.15.110:0/3618118559 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdff8155b70 0x7fdff8154750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:09.933 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:09.925+0000 7fdff67fc700 1 -- 172.21.15.110:0/3618118559 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fdff806b7c0 msgr2=0x7fdff814f3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:09.933 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:09.925+0000 7fdff67fc700 1 --2- 172.21.15.110:0/3618118559 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fdff806b7c0 0x7fdff814f3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:09.933 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:09.925+0000 7fdff67fc700 1 -- 172.21.15.110:0/3618118559 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdff8155b70 msgr2=0x7fdff8154750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:09.933 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:09.925+0000 7fdff67fc700 1 --2- 172.21.15.110:0/3618118559 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdff8155b70 0x7fdff8154750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:09.934 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:09.925+0000 7fdff67fc700 1 -- 172.21.15.110:0/3618118559 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfe8009a20 con 0x7fdff80cac10 2023-10-06T03:30:09.934 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:09.925+0000 7fdff67fc700 1 --2- 172.21.15.110:0/3618118559 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdff80cac10 0x7fdff8154210 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fdfe000c590 tx=0x7fdfe0007f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:30:09.934 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:09.925+0000 7fdfeeffd700 1 -- 172.21.15.110:0/3618118559 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfe0011590 con 0x7fdff80cac10 2023-10-06T03:30:09.934 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:09.925+0000 7fdfeeffd700 1 -- 172.21.15.110:0/3618118559 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdfe0011710 con 0x7fdff80cac10 2023-10-06T03:30:09.934 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:09.925+0000 7fdffd310700 1 -- 172.21.15.110:0/3618118559 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdff80a8d30 con 0x7fdff80cac10 2023-10-06T03:30:09.934 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:09.925+0000 7fdfeeffd700 1 -- 172.21.15.110:0/3618118559 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfe0011ad0 con 0x7fdff80cac10 2023-10-06T03:30:09.934 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:09.925+0000 7fdffd310700 1 -- 172.21.15.110:0/3618118559 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdff80ad6e0 con 0x7fdff80cac10 2023-10-06T03:30:09.934 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:09.929+0000 7fdffd310700 1 -- 172.21.15.110:0/3618118559 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdff80a9210 con 0x7fdff80cac10 2023-10-06T03:30:09.934 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:09.929+0000 7fdfeeffd700 1 -- 172.21.15.110:0/3618118559 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7fdfe001b3b0 con 0x7fdff80cac10 2023-10-06T03:30:09.935 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:09.929+0000 7fdfeeffd700 1 --2- 172.21.15.110:0/3618118559 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fdfd8060550 0x7fdfd8062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:09.935 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:09.929+0000 7fdff6ffd700 1 --2- 172.21.15.110:0/3618118559 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fdfd8060550 0x7fdfd8062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:09.935 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:09.929+0000 7fdfeeffd700 1 -- 172.21.15.110:0/3618118559 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6809+0+0 (secure 0 0 0) 0x7fdfe007c050 con 0x7fdff80cac10 2023-10-06T03:30:09.935 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:09.929+0000 7fdff6ffd700 1 --2- 172.21.15.110:0/3618118559 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fdfd8060550 0x7fdfd8062a00 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fdfe8009e60 tx=0x7fdfe8019000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:30:09.940 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:09.937+0000 7fdfeeffd700 1 -- 172.21.15.110:0/3618118559 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fdfe00abbe0 con 0x7fdff80cac10 2023-10-06T03:30:10.107 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:10.101+0000 7fdffd310700 1 -- 172.21.15.110:0/3618118559 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fdff806ff60 con 0x7fdff80cac10 2023-10-06T03:30:10.107 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:10.101+0000 7fdfeeffd700 1 -- 172.21.15.110:0/3618118559 <== mon.0 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v28) v1 ==== 104+0+11 (secure 0 0 0) 0x7fdfe00170a0 con 0x7fdff80cac10 2023-10-06T03:30:10.107 INFO:teuthology.orchestra.run.smithi110.stdout:pg_num: 32 2023-10-06T03:30:10.110 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:10.105+0000 7fdffd310700 1 -- 172.21.15.110:0/3618118559 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fdfd8060550 msgr2=0x7fdfd8062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:10.110 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:10.105+0000 7fdffd310700 1 --2- 172.21.15.110:0/3618118559 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fdfd8060550 0x7fdfd8062a00 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fdfe8009e60 tx=0x7fdfe8019000 comp rx=0 tx=0).stop 2023-10-06T03:30:10.110 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:10.105+0000 7fdffd310700 1 -- 172.21.15.110:0/3618118559 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdff80cac10 msgr2=0x7fdff8154210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:10.110 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:10.105+0000 7fdffd310700 1 --2- 172.21.15.110:0/3618118559 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdff80cac10 0x7fdff8154210 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fdfe000c590 tx=0x7fdfe0007f10 comp rx=0 tx=0).stop 2023-10-06T03:30:10.110 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:10.105+0000 7fdffd310700 1 -- 172.21.15.110:0/3618118559 shutdown_connections 2023-10-06T03:30:10.110 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:10.105+0000 7fdffd310700 1 --2- 172.21.15.110:0/3618118559 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fdfd8060550 0x7fdfd8062a00 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:10.110 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:10.105+0000 7fdffd310700 1 --2- 172.21.15.110:0/3618118559 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdff8155b70 0x7fdff8154750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:10.110 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:10.105+0000 7fdffd310700 1 --2- 172.21.15.110:0/3618118559 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdff80cac10 0x7fdff8154210 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:10.110 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:10.105+0000 7fdffd310700 1 --2- 172.21.15.110:0/3618118559 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fdff806b7c0 0x7fdff814f3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:10.110 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:10.105+0000 7fdffd310700 1 -- 172.21.15.110:0/3618118559 >> 172.21.15.110:0/3618118559 conn(0x7fdff80c8f00 msgr2=0x7fdff806a100 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:30:10.111 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:10.105+0000 7fdffd310700 1 -- 172.21.15.110:0/3618118559 shutdown_connections 2023-10-06T03:30:10.111 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:10.105+0000 7fdffd310700 1 -- 172.21.15.110:0/3618118559 wait complete. 2023-10-06T03:30:10.121 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-06T03:30:10.231 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:10.225+0000 7f6665603700 1 Processor -- start 2023-10-06T03:30:10.231 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:10.225+0000 7f6665603700 1 -- start start 2023-10-06T03:30:10.233 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:10.229+0000 7f6665603700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f666015d360 0x7f66600af440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:10.233 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:10.229+0000 7f6665603700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66600cf5b0 con 0x7f66600af070 2023-10-06T03:30:10.233 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:10.229+0000 7f6665603700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66600ce9a0 con 0x7f66600cfa40 2023-10-06T03:30:10.233 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:10.229+0000 7f6665603700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f66600cec80 con 0x7f666015d360 2023-10-06T03:30:10.233 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:10.229+0000 7f66567fc700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f666015d360 0x7f66600af440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:10.233 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:10.229+0000 7f665effd700 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f66600af070 0x7f666015cc50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:59460/0 (socket says 172.21.15.110:59460) 2023-10-06T03:30:10.234 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:10.229+0000 7f665effd700 1 -- 172.21.15.110:0/2694252977 learned_addr learned my addr 172.21.15.110:0/2694252977 (peer_addr_for_me v1:172.21.15.110:0/0) 2023-10-06T03:30:10.235 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:10.229+0000 7f66567fc700 1 -- 172.21.15.110:0/2694252977 >> v1:172.21.15.192:6789/0 conn(0x7f66600cfa40 legacy=0x7f66600afbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:10.235 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:10.229+0000 7f66567fc700 1 -- 172.21.15.110:0/2694252977 >> v1:172.21.15.110:6789/0 conn(0x7f66600af070 legacy=0x7f666015cc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:10.235 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:10.229+0000 7f66567fc700 1 -- 172.21.15.110:0/2694252977 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66600cef60 con 0x7f666015d360 2023-10-06T03:30:10.235 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:10.229+0000 7f66567fc700 1 --2- 172.21.15.110:0/2694252977 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f666015d360 0x7f66600af440 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f66480098e0 tx=0x7f66480091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2caac01d49ab107f server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:30:10.235 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:10.229+0000 7f665e7fc700 1 -- 172.21.15.110:0/2694252977 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6648009e80 con 0x7f666015d360 2023-10-06T03:30:10.235 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:10.229+0000 7f665e7fc700 1 -- 172.21.15.110:0/2694252977 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f664800b860 con 0x7f666015d360 2023-10-06T03:30:10.235 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:10.229+0000 7f665e7fc700 1 -- 172.21.15.110:0/2694252977 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f664800bbe0 con 0x7f666015d360 2023-10-06T03:30:10.236 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:10.229+0000 7f6665603700 1 -- 172.21.15.110:0/2694252977 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f666015d360 msgr2=0x7f66600af440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:10.236 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:10.229+0000 7f6665603700 1 --2- 172.21.15.110:0/2694252977 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f666015d360 0x7f66600af440 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f66480098e0 tx=0x7f66480091a0 comp rx=0 tx=0).stop 2023-10-06T03:30:10.236 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:10.229+0000 7f6665603700 1 -- 172.21.15.110:0/2694252977 shutdown_connections 2023-10-06T03:30:10.236 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:10.229+0000 7f6665603700 1 --2- 172.21.15.110:0/2694252977 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f666015d360 0x7f66600af440 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:10.236 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:10.229+0000 7f6665603700 1 -- 172.21.15.110:0/2694252977 >> 172.21.15.110:0/2694252977 conn(0x7f666006e840 msgr2=0x7f66600d0010 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:30:10.236 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:10.229+0000 7f6665603700 1 -- 172.21.15.110:0/2694252977 shutdown_connections 2023-10-06T03:30:10.236 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:10.229+0000 7f6665603700 1 -- 172.21.15.110:0/2694252977 wait complete. 2023-10-06T03:30:10.239 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:10.233+0000 7f6665603700 1 Processor -- start 2023-10-06T03:30:10.239 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:10.233+0000 7f6665603700 1 -- start start 2023-10-06T03:30:10.239 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:10.233+0000 7f6665603700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f66600af070 0x7f666014fd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:10.239 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:10.233+0000 7f6665603700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f66600cfa40 0x7f66600b12f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:10.239 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:10.233+0000 7f6665603700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f666015d360 0x7f66600b1830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:10.240 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:10.233+0000 7f6665603700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f666015a850 con 0x7f66600cfa40 2023-10-06T03:30:10.240 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:10.233+0000 7f6665603700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6660150280 con 0x7f666015d360 2023-10-06T03:30:10.240 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:10.233+0000 7f6665603700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6660159c40 con 0x7f66600af070 2023-10-06T03:30:10.240 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:10.233+0000 7f66567fc700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f66600cfa40 0x7f66600b12f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:10.240 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:10.233+0000 7f66567fc700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f66600cfa40 0x7f66600b12f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:40520/0 (socket says 172.21.15.110:40520) 2023-10-06T03:30:10.240 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:10.233+0000 7f66567fc700 1 -- 172.21.15.110:0/1013129511 learned_addr learned my addr 172.21.15.110:0/1013129511 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:30:10.240 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:10.233+0000 7f665effd700 1 --2- 172.21.15.110:0/1013129511 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f66600af070 0x7f666014fd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:10.240 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:10.233+0000 7f665f7fe700 1 --2- 172.21.15.110:0/1013129511 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f666015d360 0x7f66600b1830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:10.241 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:10.233+0000 7f66567fc700 1 -- 172.21.15.110:0/1013129511 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f66600af070 msgr2=0x7f666014fd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:10.241 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:10.233+0000 7f66567fc700 1 --2- 172.21.15.110:0/1013129511 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f66600af070 0x7f666014fd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:10.241 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:10.233+0000 7f66567fc700 1 -- 172.21.15.110:0/1013129511 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f666015d360 msgr2=0x7f66600b1830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:10.241 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:10.233+0000 7f66567fc700 1 --2- 172.21.15.110:0/1013129511 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f666015d360 0x7f66600b1830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:10.241 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:10.233+0000 7f66567fc700 1 -- 172.21.15.110:0/1013129511 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6648009420 con 0x7f66600cfa40 2023-10-06T03:30:10.241 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:10.233+0000 7f665f7fe700 1 --2- 172.21.15.110:0/1013129511 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f666015d360 0x7f66600b1830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-06T03:30:10.241 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:10.233+0000 7f66567fc700 1 --2- 172.21.15.110:0/1013129511 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f66600cfa40 0x7f66600b12f0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f6648013000 tx=0x7f6648017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:30:10.241 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:10.233+0000 7f66577fe700 1 -- 172.21.15.110:0/1013129511 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f664800bb20 con 0x7f66600cfa40 2023-10-06T03:30:10.241 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:10.233+0000 7f66577fe700 1 -- 172.21.15.110:0/1013129511 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6648025420 con 0x7f66600cfa40 2023-10-06T03:30:10.242 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:10.233+0000 7f6665603700 1 -- 172.21.15.110:0/1013129511 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6660159f20 con 0x7f66600cfa40 2023-10-06T03:30:10.242 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:10.233+0000 7f66577fe700 1 -- 172.21.15.110:0/1013129511 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66480257c0 con 0x7f66600cfa40 2023-10-06T03:30:10.242 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:10.233+0000 7f6665603700 1 -- 172.21.15.110:0/1013129511 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f666015a4e0 con 0x7f66600cfa40 2023-10-06T03:30:10.243 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:10.237+0000 7f6665603700 1 -- 172.21.15.110:0/1013129511 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f66600b5e70 con 0x7f66600cfa40 2023-10-06T03:30:10.243 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:10.237+0000 7f66577fe700 1 -- 172.21.15.110:0/1013129511 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7f6648023070 con 0x7f66600cfa40 2023-10-06T03:30:10.243 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:10.237+0000 7f66577fe700 1 --2- 172.21.15.110:0/1013129511 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f6644060550 0x7f6644062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:10.243 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:10.237+0000 7f66577fe700 1 -- 172.21.15.110:0/1013129511 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6809+0+0 (secure 0 0 0) 0x7f664807e3d0 con 0x7f66600cfa40 2023-10-06T03:30:10.243 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:10.237+0000 7f665effd700 1 --2- 172.21.15.110:0/1013129511 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f6644060550 0x7f6644062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:10.244 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:10.237+0000 7f665effd700 1 --2- 172.21.15.110:0/1013129511 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f6644060550 0x7f6644062a00 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f6650007fd0 tx=0x7f6650009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:30:10.250 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:10.245+0000 7f66577fe700 1 -- 172.21.15.110:0/1013129511 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f66600b5e70 con 0x7f66600cfa40 2023-10-06T03:30:10.406 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:10.401+0000 7f6665603700 1 -- 172.21.15.110:0/1013129511 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f66600b5e70 con 0x7f66600cfa40 2023-10-06T03:30:10.406 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:10.401+0000 7f66577fe700 1 -- 172.21.15.110:0/1013129511 <== mon.0 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v28) v1 ==== 100+0+11 (secure 0 0 0) 0x7f66600b5e70 con 0x7f66600cfa40 2023-10-06T03:30:10.406 INFO:teuthology.orchestra.run.smithi110.stdout:pg_num: 64 2023-10-06T03:30:10.409 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:10.405+0000 7f6665603700 1 -- 172.21.15.110:0/1013129511 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f6644060550 msgr2=0x7f6644062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:10.409 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:10.405+0000 7f6665603700 1 --2- 172.21.15.110:0/1013129511 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f6644060550 0x7f6644062a00 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f6650007fd0 tx=0x7f6650009040 comp rx=0 tx=0).stop 2023-10-06T03:30:10.409 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:10.405+0000 7f6665603700 1 -- 172.21.15.110:0/1013129511 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f66600cfa40 msgr2=0x7f66600b12f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:10.409 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:10.405+0000 7f6665603700 1 --2- 172.21.15.110:0/1013129511 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f66600cfa40 0x7f66600b12f0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f6648013000 tx=0x7f6648017000 comp rx=0 tx=0).stop 2023-10-06T03:30:10.409 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:10.405+0000 7f6665603700 1 -- 172.21.15.110:0/1013129511 shutdown_connections 2023-10-06T03:30:10.409 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:10.405+0000 7f6665603700 1 --2- 172.21.15.110:0/1013129511 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f6644060550 0x7f6644062a00 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:10.409 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:10.405+0000 7f6665603700 1 --2- 172.21.15.110:0/1013129511 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f666015d360 0x7f66600b1830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:10.409 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:10.405+0000 7f6665603700 1 --2- 172.21.15.110:0/1013129511 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f66600cfa40 0x7f66600b12f0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:10.410 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:10.405+0000 7f6665603700 1 --2- 172.21.15.110:0/1013129511 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f66600af070 0x7f666014fd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:10.410 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:10.405+0000 7f6665603700 1 -- 172.21.15.110:0/1013129511 >> 172.21.15.110:0/1013129511 conn(0x7f666006e840 msgr2=0x7f66600c6670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:30:10.410 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:10.405+0000 7f6665603700 1 -- 172.21.15.110:0/1013129511 shutdown_connections 2023-10-06T03:30:10.410 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:10.405+0000 7f6665603700 1 -- 172.21.15.110:0/1013129511 wait complete. 2023-10-06T03:30:10.421 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2023-10-06T03:30:10.534 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:10.529+0000 7f938b59e700 1 Processor -- start 2023-10-06T03:30:10.534 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:10.529+0000 7f938b59e700 1 -- start start 2023-10-06T03:30:10.536 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:10.533+0000 7f938b59e700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f9384037300 0x7f93840e00c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:10.537 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:10.533+0000 7f938b59e700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9384007380 con 0x7f93840dfcf0 2023-10-06T03:30:10.537 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:10.533+0000 7f938b59e700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9384006770 con 0x7f9384007810 2023-10-06T03:30:10.537 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:10.533+0000 7f938b59e700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9384006a50 con 0x7f9384037300 2023-10-06T03:30:10.537 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:10.533+0000 7f938a59c700 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f93840dfcf0 0x7f9384036bf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:59474/0 (socket says 172.21.15.110:59474) 2023-10-06T03:30:10.537 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:10.533+0000 7f938a59c700 1 -- 172.21.15.110:0/810491563 learned_addr learned my addr 172.21.15.110:0/810491563 (peer_addr_for_me v1:172.21.15.110:0/0) 2023-10-06T03:30:10.537 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:10.533+0000 7f938ad9d700 1 --2- 172.21.15.110:0/810491563 >> v2:172.21.15.110:3300/0 conn(0x7f9384037300 0x7f93840e00c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:10.537 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:10.533+0000 7f938959a700 1 -- 172.21.15.110:0/810491563 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3488611659 0 0) 0x7f9384007380 con 0x7f93840dfcf0 2023-10-06T03:30:10.537 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:10.533+0000 7f938959a700 1 -- 172.21.15.110:0/810491563 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9374003570 con 0x7f93840dfcf0 2023-10-06T03:30:10.537 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:10.533+0000 7f938ad9d700 1 -- 172.21.15.110:0/810491563 >> v1:172.21.15.192:6789/0 conn(0x7f9384007810 legacy=0x7f938404fa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:10.538 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:10.533+0000 7f938ad9d700 1 -- 172.21.15.110:0/810491563 >> v1:172.21.15.110:6789/0 conn(0x7f93840dfcf0 legacy=0x7f9384036bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:10.538 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:10.533+0000 7f938ad9d700 1 -- 172.21.15.110:0/810491563 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9384006d30 con 0x7f9384037300 2023-10-06T03:30:10.538 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:10.533+0000 7f938ad9d700 1 --2- 172.21.15.110:0/810491563 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9384037300 0x7f93840e00c0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f938c066ee0 tx=0x7f938c066cd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c223d79075e539 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:30:10.539 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:10.533+0000 7f938959a700 1 -- 172.21.15.110:0/810491563 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f938c06b040 con 0x7f9384037300 2023-10-06T03:30:10.539 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:10.533+0000 7f938959a700 1 -- 172.21.15.110:0/810491563 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f938c04bf10 con 0x7f9384037300 2023-10-06T03:30:10.539 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:10.533+0000 7f938959a700 1 -- 172.21.15.110:0/810491563 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f938c06b3d0 con 0x7f9384037300 2023-10-06T03:30:10.542 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:10.533+0000 7f938b59e700 1 -- 172.21.15.110:0/810491563 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9384037300 msgr2=0x7f93840e00c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:10.542 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:10.533+0000 7f938b59e700 1 --2- 172.21.15.110:0/810491563 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9384037300 0x7f93840e00c0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f938c066ee0 tx=0x7f938c066cd0 comp rx=0 tx=0).stop 2023-10-06T03:30:10.542 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:10.533+0000 7f938b59e700 1 -- 172.21.15.110:0/810491563 shutdown_connections 2023-10-06T03:30:10.542 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:10.533+0000 7f938b59e700 1 --2- 172.21.15.110:0/810491563 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9384037300 0x7f93840e00c0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:10.543 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:10.533+0000 7f938b59e700 1 -- 172.21.15.110:0/810491563 >> 172.21.15.110:0/810491563 conn(0x7f9384054da0 msgr2=0x7f93840551a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:30:10.543 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:10.533+0000 7f938b59e700 1 -- 172.21.15.110:0/810491563 shutdown_connections 2023-10-06T03:30:10.543 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:10.533+0000 7f938b59e700 1 -- 172.21.15.110:0/810491563 wait complete. 2023-10-06T03:30:10.543 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:10.533+0000 7f938b59e700 1 Processor -- start 2023-10-06T03:30:10.543 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:10.537+0000 7f938b59e700 1 -- start start 2023-10-06T03:30:10.543 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:10.537+0000 7f938b59e700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9384007810 0x7f93840508d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:10.543 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:10.537+0000 7f938b59e700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9384037300 0x7f93840dc170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:10.543 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:10.537+0000 7f938b59e700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f93840dfcf0 0x7f93840ddbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:10.543 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:10.537+0000 7f938b59e700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93840dd760 con 0x7f9384007810 2023-10-06T03:30:10.543 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:10.537+0000 7f938b59e700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93840dcb50 con 0x7f93840dfcf0 2023-10-06T03:30:10.543 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:10.537+0000 7f938b59e700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93840dce30 con 0x7f9384037300 2023-10-06T03:30:10.543 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:10.537+0000 7f938a59c700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9384007810 0x7f93840508d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:10.544 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:10.537+0000 7f938a59c700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9384007810 0x7f93840508d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:40544/0 (socket says 172.21.15.110:40544) 2023-10-06T03:30:10.544 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:10.537+0000 7f938a59c700 1 -- 172.21.15.110:0/2235354066 learned_addr learned my addr 172.21.15.110:0/2235354066 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:30:10.545 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:10.537+0000 7f9389d9b700 1 --2- 172.21.15.110:0/2235354066 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9384037300 0x7f93840dc170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:10.545 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:10.537+0000 7f938ad9d700 1 --2- 172.21.15.110:0/2235354066 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f93840dfcf0 0x7f93840ddbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:10.545 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:10.537+0000 7f938a59c700 1 -- 172.21.15.110:0/2235354066 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9384037300 msgr2=0x7f93840dc170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:10.545 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:10.537+0000 7f938a59c700 1 --2- 172.21.15.110:0/2235354066 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9384037300 0x7f93840dc170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:10.545 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:10.537+0000 7f938a59c700 1 -- 172.21.15.110:0/2235354066 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f93840dfcf0 msgr2=0x7f93840ddbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:10.545 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:10.537+0000 7f938a59c700 1 --2- 172.21.15.110:0/2235354066 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f93840dfcf0 0x7f93840ddbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:10.545 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:10.537+0000 7f938a59c700 1 -- 172.21.15.110:0/2235354066 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f938c063bb0 con 0x7f9384007810 2023-10-06T03:30:10.546 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:10.537+0000 7f938a59c700 1 --2- 172.21.15.110:0/2235354066 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9384007810 0x7f93840508d0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f937c00c590 tx=0x7f937c007e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:30:10.546 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:10.537+0000 7f937a7fc700 1 -- 172.21.15.110:0/2235354066 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f937c005b30 con 0x7f9384007810 2023-10-06T03:30:10.546 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:10.537+0000 7f937a7fc700 1 -- 172.21.15.110:0/2235354066 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f937c005cb0 con 0x7f9384007810 2023-10-06T03:30:10.546 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:10.537+0000 7f938b59e700 1 -- 172.21.15.110:0/2235354066 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93840dd110 con 0x7f9384007810 2023-10-06T03:30:10.546 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:10.537+0000 7f937a7fc700 1 -- 172.21.15.110:0/2235354066 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f937c007320 con 0x7f9384007810 2023-10-06T03:30:10.546 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:10.537+0000 7f938b59e700 1 -- 172.21.15.110:0/2235354066 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9384051cb0 con 0x7f9384007810 2023-10-06T03:30:10.547 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:10.541+0000 7f938b59e700 1 -- 172.21.15.110:0/2235354066 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f93840054b0 con 0x7f9384007810 2023-10-06T03:30:10.547 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:10.541+0000 7f937a7fc700 1 -- 172.21.15.110:0/2235354066 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7f937c01b3b0 con 0x7f9384007810 2023-10-06T03:30:10.547 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:10.541+0000 7f937a7fc700 1 --2- 172.21.15.110:0/2235354066 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f936c060550 0x7f936c062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:10.547 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:10.541+0000 7f937a7fc700 1 -- 172.21.15.110:0/2235354066 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6809+0+0 (secure 0 0 0) 0x7f937c07c060 con 0x7f9384007810 2023-10-06T03:30:10.547 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:10.541+0000 7f9389d9b700 1 --2- 172.21.15.110:0/2235354066 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f936c060550 0x7f936c062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:10.547 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:10.541+0000 7f9389d9b700 1 --2- 172.21.15.110:0/2235354066 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f936c060550 0x7f936c062a00 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f93840f4520 tx=0x7f9380007ec0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:30:10.551 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:10.549+0000 7f937a7fc700 1 -- 172.21.15.110:0/2235354066 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f937c04c200 con 0x7f9384007810 2023-10-06T03:30:10.713 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:10.709+0000 7f938b59e700 1 -- 172.21.15.110:0/2235354066 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f9384007010 con 0x7f9384007810 2023-10-06T03:30:10.713 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:10.709+0000 7f937a7fc700 1 -- 172.21.15.110:0/2235354066 <== mon.0 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v28) v1 ==== 103+0+11 (secure 0 0 0) 0x7f937c04f820 con 0x7f9384007810 2023-10-06T03:30:10.713 INFO:teuthology.orchestra.run.smithi110.stdout:pg_num: 32 2023-10-06T03:30:10.715 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:10.713+0000 7f938b59e700 1 -- 172.21.15.110:0/2235354066 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f936c060550 msgr2=0x7f936c062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:10.716 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:10.713+0000 7f938b59e700 1 --2- 172.21.15.110:0/2235354066 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f936c060550 0x7f936c062a00 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f93840f4520 tx=0x7f9380007ec0 comp rx=0 tx=0).stop 2023-10-06T03:30:10.716 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:10.713+0000 7f938b59e700 1 -- 172.21.15.110:0/2235354066 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9384007810 msgr2=0x7f93840508d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:10.716 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:10.713+0000 7f938b59e700 1 --2- 172.21.15.110:0/2235354066 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9384007810 0x7f93840508d0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f937c00c590 tx=0x7f937c007e00 comp rx=0 tx=0).stop 2023-10-06T03:30:10.716 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:10.713+0000 7f938b59e700 1 -- 172.21.15.110:0/2235354066 shutdown_connections 2023-10-06T03:30:10.716 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:10.713+0000 7f938b59e700 1 --2- 172.21.15.110:0/2235354066 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f936c060550 0x7f936c062a00 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:10.716 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:10.713+0000 7f938b59e700 1 --2- 172.21.15.110:0/2235354066 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f93840dfcf0 0x7f93840ddbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:10.716 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:10.713+0000 7f938b59e700 1 --2- 172.21.15.110:0/2235354066 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9384037300 0x7f93840dc170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:10.716 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:10.713+0000 7f938b59e700 1 --2- 172.21.15.110:0/2235354066 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9384007810 0x7f93840508d0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:10.716 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:10.713+0000 7f938b59e700 1 -- 172.21.15.110:0/2235354066 >> 172.21.15.110:0/2235354066 conn(0x7f9384054da0 msgr2=0x7f938404be50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:30:10.716 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:10.713+0000 7f938b59e700 1 -- 172.21.15.110:0/2235354066 shutdown_connections 2023-10-06T03:30:10.716 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:10.713+0000 7f938b59e700 1 -- 172.21.15.110:0/2235354066 wait complete. 2023-10-06T03:30:10.727 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-06T03:30:10.835 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:10.829+0000 7f28cf23a700 1 Processor -- start 2023-10-06T03:30:10.835 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:10.829+0000 7f28cf23a700 1 -- start start 2023-10-06T03:30:10.837 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:10.833+0000 7f28cf23a700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f28c80add30 0x7f28c80ae100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:10.838 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:10.833+0000 7f28cf23a700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f28c80cdb30 0x7f28c80cdf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:10.838 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:10.833+0000 7f28cf23a700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f28c80af8e0 0x7f28c80afd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:10.838 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:10.833+0000 7f28cf23a700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f28c80ce4d0 con 0x7f28c80cdb30 2023-10-06T03:30:10.838 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:10.833+0000 7f28cf23a700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f28c8157ba0 con 0x7f28c80add30 2023-10-06T03:30:10.838 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:10.833+0000 7f28cf23a700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28c80acda0 con 0x7f28c80af8e0 2023-10-06T03:30:10.838 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:10.833+0000 7f28cda37700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f28c80cdb30 0x7f28c80cdf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:10.838 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:10.833+0000 7f28cda37700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f28c80cdb30 0x7f28c80cdf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:40560/0 (socket says 172.21.15.110:40560) 2023-10-06T03:30:10.838 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:10.833+0000 7f28cda37700 1 -- 172.21.15.110:0/2553105845 learned_addr learned my addr 172.21.15.110:0/2553105845 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:30:10.838 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:10.833+0000 7f28cea39700 1 --2- 172.21.15.110:0/2553105845 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f28c80af8e0 0x7f28c80afd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:10.838 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:10.833+0000 7f28ce238700 1 --2- 172.21.15.110:0/2553105845 >> v2:172.21.15.192:3300/0 conn(0x7f28c80add30 0x7f28c80ae100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:10.840 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:10.833+0000 7f28ce238700 1 -- 172.21.15.110:0/2553105845 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f28c80af8e0 msgr2=0x7f28c80afd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:30:10.840 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:10.833+0000 7f28ce238700 1 --2- 172.21.15.110:0/2553105845 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f28c80af8e0 0x7f28c80afd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:10.840 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:10.833+0000 7f28ce238700 1 -- 172.21.15.110:0/2553105845 >> v2:172.21.15.110:3300/0 conn(0x7f28c80cdb30 msgr2=0x7f28c80cdf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:30:10.840 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:10.833+0000 7f28ce238700 1 --2- 172.21.15.110:0/2553105845 >> v2:172.21.15.110:3300/0 conn(0x7f28c80cdb30 0x7f28c80cdf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:10.840 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:10.833+0000 7f28ce238700 1 -- 172.21.15.110:0/2553105845 --> v2:172.21.15.192:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28c81572e0 con 0x7f28c80add30 2023-10-06T03:30:10.840 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:10.833+0000 7f28cda37700 1 --2- 172.21.15.110:0/2553105845 >> v2:172.21.15.110:3300/0 conn(0x7f28c80cdb30 0x7f28c80cdf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-06T03:30:10.840 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:10.833+0000 7f28ce238700 1 --2- 172.21.15.110:0/2553105845 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f28c80add30 0x7f28c80ae100 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f28c8069750 tx=0x7f28bc0091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=48d2704ebc4df37c server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:30:10.841 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:10.833+0000 7f28cd236700 1 -- 172.21.15.110:0/2553105845 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28bc011630 con 0x7f28c80add30 2023-10-06T03:30:10.841 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:10.833+0000 7f28cd236700 1 -- 172.21.15.110:0/2553105845 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28bc0117b0 con 0x7f28c80add30 2023-10-06T03:30:10.841 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:10.833+0000 7f28cd236700 1 -- 172.21.15.110:0/2553105845 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28bc011b70 con 0x7f28c80add30 2023-10-06T03:30:10.841 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:10.837+0000 7f28cf23a700 1 -- 172.21.15.110:0/2553105845 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f28c80add30 msgr2=0x7f28c80ae100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:10.841 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:10.837+0000 7f28cf23a700 1 --2- 172.21.15.110:0/2553105845 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f28c80add30 0x7f28c80ae100 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f28c8069750 tx=0x7f28bc0091a0 comp rx=0 tx=0).stop 2023-10-06T03:30:10.841 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:10.837+0000 7f28cf23a700 1 -- 172.21.15.110:0/2553105845 shutdown_connections 2023-10-06T03:30:10.841 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:10.837+0000 7f28cf23a700 1 --2- 172.21.15.110:0/2553105845 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f28c80af8e0 0x7f28c80afd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:10.842 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:10.837+0000 7f28cf23a700 1 --2- 172.21.15.110:0/2553105845 >> v2:172.21.15.110:3300/0 conn(0x7f28c80cdb30 0x7f28c80cdf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:10.842 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:10.837+0000 7f28cf23a700 1 --2- 172.21.15.110:0/2553105845 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f28c80add30 0x7f28c80ae100 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:10.842 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:10.837+0000 7f28cf23a700 1 -- 172.21.15.110:0/2553105845 >> 172.21.15.110:0/2553105845 conn(0x7f28c806b7c0 msgr2=0x7f28c806bbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:30:10.842 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:10.837+0000 7f28cf23a700 1 -- 172.21.15.110:0/2553105845 shutdown_connections 2023-10-06T03:30:10.842 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:10.837+0000 7f28cf23a700 1 -- 172.21.15.110:0/2553105845 wait complete. 2023-10-06T03:30:10.843 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:10.841+0000 7f28cf23a700 1 Processor -- start 2023-10-06T03:30:10.843 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:10.841+0000 7f28cf23a700 1 -- start start 2023-10-06T03:30:10.847 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:10.841+0000 7f28cf23a700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f28c80add30 0x7f28c81521b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:10.847 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:10.841+0000 7f28cf23a700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f28c80af8e0 0x7f28c80c6cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:10.847 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:10.841+0000 7f28cf23a700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f28c80cdb30 0x7f28c80b1fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:10.847 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:10.841+0000 7f28cf23a700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28c80c6860 con 0x7f28c80add30 2023-10-06T03:30:10.847 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:10.841+0000 7f28cf23a700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28c80c5c50 con 0x7f28c80cdb30 2023-10-06T03:30:10.847 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:10.841+0000 7f28cf23a700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28c80c5f30 con 0x7f28c80af8e0 2023-10-06T03:30:10.847 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:10.841+0000 7f28ce238700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f28c80add30 0x7f28c81521b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:10.847 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:10.841+0000 7f28ce238700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f28c80add30 0x7f28c81521b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:40576/0 (socket says 172.21.15.110:40576) 2023-10-06T03:30:10.847 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:10.841+0000 7f28ce238700 1 -- 172.21.15.110:0/590918202 learned_addr learned my addr 172.21.15.110:0/590918202 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:30:10.847 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:10.841+0000 7f28cda37700 1 --2- 172.21.15.110:0/590918202 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f28c80af8e0 0x7f28c80c6cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:10.848 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:10.841+0000 7f28cea39700 1 --2- 172.21.15.110:0/590918202 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f28c80cdb30 0x7f28c80b1fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:10.848 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:10.841+0000 7f28ce238700 1 -- 172.21.15.110:0/590918202 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f28c80af8e0 msgr2=0x7f28c80c6cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:10.848 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:10.841+0000 7f28ce238700 1 --2- 172.21.15.110:0/590918202 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f28c80af8e0 0x7f28c80c6cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:10.848 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:10.841+0000 7f28ce238700 1 -- 172.21.15.110:0/590918202 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f28c80cdb30 msgr2=0x7f28c80b1fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:10.848 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:10.841+0000 7f28ce238700 1 --2- 172.21.15.110:0/590918202 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f28c80cdb30 0x7f28c80b1fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:10.848 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:10.841+0000 7f28ce238700 1 -- 172.21.15.110:0/590918202 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28c40094f0 con 0x7f28c80add30 2023-10-06T03:30:10.848 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:10.841+0000 7f28ce238700 1 --2- 172.21.15.110:0/590918202 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f28c80add30 0x7f28c81521b0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f28c8157ce0 tx=0x7f28bc017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:30:10.848 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:10.841+0000 7f28ba7fc700 1 -- 172.21.15.110:0/590918202 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28bc011920 con 0x7f28c80add30 2023-10-06T03:30:10.848 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:10.841+0000 7f28ba7fc700 1 -- 172.21.15.110:0/590918202 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28bc011aa0 con 0x7f28c80add30 2023-10-06T03:30:10.848 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:10.841+0000 7f28cf23a700 1 -- 172.21.15.110:0/590918202 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28bc009420 con 0x7f28c80add30 2023-10-06T03:30:10.849 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:10.841+0000 7f28ba7fc700 1 -- 172.21.15.110:0/590918202 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28bc01a680 con 0x7f28c80add30 2023-10-06T03:30:10.849 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:10.841+0000 7f28cf23a700 1 -- 172.21.15.110:0/590918202 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28c80c64f0 con 0x7f28c80add30 2023-10-06T03:30:10.849 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:10.845+0000 7f28cf23a700 1 -- 172.21.15.110:0/590918202 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f28c8070ff0 con 0x7f28c80add30 2023-10-06T03:30:10.850 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:10.845+0000 7f28ba7fc700 1 -- 172.21.15.110:0/590918202 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7f28bc01aa30 con 0x7f28c80add30 2023-10-06T03:30:10.850 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:10.845+0000 7f28ba7fc700 1 --2- 172.21.15.110:0/590918202 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f28ac060550 0x7f28ac062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:10.850 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:10.845+0000 7f28cda37700 1 --2- 172.21.15.110:0/590918202 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f28ac060550 0x7f28ac062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:10.850 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:10.845+0000 7f28ba7fc700 1 -- 172.21.15.110:0/590918202 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6809+0+0 (secure 0 0 0) 0x7f28bc07ec10 con 0x7f28c80add30 2023-10-06T03:30:10.850 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:10.845+0000 7f28cda37700 1 --2- 172.21.15.110:0/590918202 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f28ac060550 0x7f28ac062a00 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f28c816a3b0 tx=0x7f28c4009250 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:30:10.856 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:10.853+0000 7f28ba7fc700 1 -- 172.21.15.110:0/590918202 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f28bc04ee10 con 0x7f28c80add30 2023-10-06T03:30:11.015 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:11.013+0000 7f28cf23a700 1 -- 172.21.15.110:0/590918202 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f28c8157550 con 0x7f28c80add30 2023-10-06T03:30:11.017 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:11.013+0000 7f28ba7fc700 1 -- 172.21.15.110:0/590918202 <== mon.0 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v28) v1 ==== 74+0+18755 (secure 0 0 0) 0x7f28bc015090 con 0x7f28c80add30 2023-10-06T03:30:11.018 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-10-06T03:30:11.018 INFO:teuthology.orchestra.run.smithi110.stdout:{"epoch":28,"fsid":"93b8f6ac-4c92-4b45-b8e4-b59617024aa2","created":"2023-10-06T03:29:10.519583+0000","modified":"2023-10-06T03:29:38.951740+0000","last_up_change":"2023-10-06T03:29:21.960511+0000","last_in_change":"2023-10-06T03:29:12.560436+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-06T03:29:20.703743+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2023-10-06T03:29:28.307281+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2023-10-06T03:29:32.247251+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data","create_time":"2023-10-06T03:29:32.880086+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}}},{"pool":5,"pool_name":"cephfs_data_ec","create_time":"2023-10-06T03:29:36.613779+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"28","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"5866eb6c-2adc-46e4-be7a-e5bcda0f4065","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6824","nonce":33394},{"type":"v1","addr":"172.21.15.110:6825","nonce":33394}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6826","nonce":33394},{"type":"v1","addr":"172.21.15.110:6827","nonce":33394}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6830","nonce":33394},{"type":"v1","addr":"172.21.15.110:6831","nonce":33394}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6828","nonce":33394},{"type":"v1","addr":"172.21.15.110:6829","nonce":33394}]},"public_addr":"172.21.15.110:6825/33394","cluster_addr":"172.21.15.110:6827/33394","heartbeat_back_addr":"172.21.15.110:6831/33394","heartbeat_front_addr":"172.21.15.110:6829/33394","state":["exists","up"]},{"osd":1,"uuid":"a38de118-062b-4a05-9782-2ceddf6ff4df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6800","nonce":33396},{"type":"v1","addr":"172.21.15.110:6801","nonce":33396}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6802","nonce":33396},{"type":"v1","addr":"172.21.15.110:6803","nonce":33396}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6806","nonce":33396},{"type":"v1","addr":"172.21.15.110:6807","nonce":33396}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6804","nonce":33396},{"type":"v1","addr":"172.21.15.110:6805","nonce":33396}]},"public_addr":"172.21.15.110:6801/33396","cluster_addr":"172.21.15.110:6803/33396","heartbeat_back_addr":"172.21.15.110:6807/33396","heartbeat_front_addr":"172.21.15.110:6805/33396","state":["exists","up"]},{"osd":2,"uuid":"56e39be2-3407-4b69-877e-cf16617b06d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6816","nonce":33397},{"type":"v1","addr":"172.21.15.110:6817","nonce":33397}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6818","nonce":33397},{"type":"v1","addr":"172.21.15.110:6819","nonce":33397}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6822","nonce":33397},{"type":"v1","addr":"172.21.15.110:6823","nonce":33397}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6820","nonce":33397},{"type":"v1","addr":"172.21.15.110:6821","nonce":33397}]},"public_addr":"172.21.15.110:6817/33397","cluster_addr":"172.21.15.110:6819/33397","heartbeat_back_addr":"172.21.15.110:6823/33397","heartbeat_front_addr":"172.21.15.110:6821/33397","state":["exists","up"]},{"osd":3,"uuid":"01c82d13-c1d4-4bde-b55e-3e0089b40115","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6808","nonce":33395},{"type":"v1","addr":"172.21.15.110:6809","nonce":33395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6810","nonce":33395},{"type":"v1","addr":"172.21.15.110:6811","nonce":33395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6814","nonce":33395},{"type":"v1","addr":"172.21.15.110:6815","nonce":33395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6812","nonce":33395},{"type":"v1","addr":"172.21.15.110:6813","nonce":33395}]},"public_addr":"172.21.15.110:6809/33395","cluster_addr":"172.21.15.110:6811/33395","heartbeat_back_addr":"172.21.15.110:6815/33395","heartbeat_front_addr":"172.21.15.110:6813/33395","state":["exists","up"]},{"osd":4,"uuid":"86983430-5fa3-4d17-998f-03e2cd843237","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":33683},{"type":"v1","addr":"172.21.15.192:6817","nonce":33683}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":33683},{"type":"v1","addr":"172.21.15.192:6819","nonce":33683}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":33683},{"type":"v1","addr":"172.21.15.192:6823","nonce":33683}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6820","nonce":33683},{"type":"v1","addr":"172.21.15.192:6821","nonce":33683}]},"public_addr":"172.21.15.192:6817/33683","cluster_addr":"172.21.15.192:6819/33683","heartbeat_back_addr":"172.21.15.192:6823/33683","heartbeat_front_addr":"172.21.15.192:6821/33683","state":["exists","up"]},{"osd":5,"uuid":"6945257f-821d-4619-a63f-ebe857eba6f4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":33684},{"type":"v1","addr":"172.21.15.192:6801","nonce":33684}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":33684},{"type":"v1","addr":"172.21.15.192:6803","nonce":33684}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":33684},{"type":"v1","addr":"172.21.15.192:6807","nonce":33684}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6804","nonce":33684},{"type":"v1","addr":"172.21.15.192:6805","nonce":33684}]},"public_addr":"172.21.15.192:6801/33684","cluster_addr":"172.21.15.192:6803/33684","heartbeat_back_addr":"172.21.15.192:6807/33684","heartbeat_front_addr":"172.21.15.192:6805/33684","state":["exists","up"]},{"osd":6,"uuid":"7854d1ae-7060-4780-8f97-73cea5e7a12f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6824","nonce":33686},{"type":"v1","addr":"172.21.15.192:6825","nonce":33686}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":33686},{"type":"v1","addr":"172.21.15.192:6827","nonce":33686}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":33686},{"type":"v1","addr":"172.21.15.192:6831","nonce":33686}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6828","nonce":33686},{"type":"v1","addr":"172.21.15.192:6829","nonce":33686}]},"public_addr":"172.21.15.192:6825/33686","cluster_addr":"172.21.15.192:6827/33686","heartbeat_back_addr":"172.21.15.192:6831/33686","heartbeat_front_addr":"172.21.15.192:6829/33686","state":["exists","up"]},{"osd":7,"uuid":"f1ebdfd5-b859-482b-9f97-c18c4a26f445","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6808","nonce":33685},{"type":"v1","addr":"172.21.15.192:6809","nonce":33685}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":33685},{"type":"v1","addr":"172.21.15.192:6811","nonce":33685}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":33685},{"type":"v1","addr":"172.21.15.192:6815","nonce":33685}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6812","nonce":33685},{"type":"v1","addr":"172.21.15.192:6813","nonce":33685}]},"public_addr":"172.21.15.192:6809/33685","cluster_addr":"172.21.15.192:6811/33685","heartbeat_back_addr":"172.21.15.192:6815/33685","heartbeat_front_addr":"172.21.15.192:6813/33685","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T03:29:18.002048+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T03:29:17.478576+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T03:29:18.034944+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T03:29:17.628513+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T03:29:18.047338+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T03:29:17.772349+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T03:29:17.883958+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T03:29:18.151838+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-06T03:30:11.020 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:11.017+0000 7f28cf23a700 1 -- 172.21.15.110:0/590918202 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f28ac060550 msgr2=0x7f28ac062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:11.020 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:11.017+0000 7f28cf23a700 1 --2- 172.21.15.110:0/590918202 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f28ac060550 0x7f28ac062a00 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f28c816a3b0 tx=0x7f28c4009250 comp rx=0 tx=0).stop 2023-10-06T03:30:11.020 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:11.017+0000 7f28cf23a700 1 -- 172.21.15.110:0/590918202 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f28c80add30 msgr2=0x7f28c81521b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:11.020 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:11.017+0000 7f28cf23a700 1 --2- 172.21.15.110:0/590918202 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f28c80add30 0x7f28c81521b0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f28c8157ce0 tx=0x7f28bc017000 comp rx=0 tx=0).stop 2023-10-06T03:30:11.020 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:11.017+0000 7f28cf23a700 1 -- 172.21.15.110:0/590918202 shutdown_connections 2023-10-06T03:30:11.021 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:11.017+0000 7f28cf23a700 1 --2- 172.21.15.110:0/590918202 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f28ac060550 0x7f28ac062a00 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:11.021 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:11.017+0000 7f28cf23a700 1 --2- 172.21.15.110:0/590918202 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f28c80cdb30 0x7f28c80b1fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:11.021 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:11.017+0000 7f28cf23a700 1 --2- 172.21.15.110:0/590918202 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f28c80af8e0 0x7f28c80c6cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:11.021 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:11.017+0000 7f28cf23a700 1 --2- 172.21.15.110:0/590918202 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f28c80add30 0x7f28c81521b0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:11.021 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:11.017+0000 7f28cf23a700 1 -- 172.21.15.110:0/590918202 >> 172.21.15.110:0/590918202 conn(0x7f28c806b7c0 msgr2=0x7f28c8152aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:30:11.021 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:11.017+0000 7f28cf23a700 1 -- 172.21.15.110:0/590918202 shutdown_connections 2023-10-06T03:30:11.021 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:11.017+0000 7f28cf23a700 1 -- 172.21.15.110:0/590918202 wait complete. 2023-10-06T03:30:11.032 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-06T03:29:20.703743+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-10-06T03:29:28.307281+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 3, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-06T03:29:32.247251+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 4, 'pool_name': 'cephfs_data', 'create_time': '2023-10-06T03:29:32.880086+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}, {'pool': 5, 'pool_name': 'cephfs_data_ec', 'create_time': '2023-10-06T03:29:36.613779+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '28', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-10-06T03:30:11.032 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-06T03:30:11.145 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:11.141+0000 7ff7d3fff700 1 Processor -- start 2023-10-06T03:30:11.145 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:11.141+0000 7ff7d3fff700 1 -- start start 2023-10-06T03:30:11.146 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:11.141+0000 7ff7d3fff700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7ff7d406b7c0 0x7ff7d40ce170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:11.146 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:11.141+0000 7ff7d3fff700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff7d406c2a0 0x7ff7d40b0a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:11.147 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:11.141+0000 7ff7d3fff700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7d40cfa00 con 0x7ff7d40ce6b0 2023-10-06T03:30:11.147 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:11.141+0000 7ff7d3fff700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff7d40cb1c0 con 0x7ff7d406b7c0 2023-10-06T03:30:11.147 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:11.141+0000 7ff7d3fff700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7d40cedf0 con 0x7ff7d406c2a0 2023-10-06T03:30:11.147 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:11.141+0000 7ff7d2ffd700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7ff7d406b7c0 0x7ff7d40ce170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:11.147 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:11.141+0000 7ff7d2ffd700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7ff7d406b7c0 0x7ff7d40ce170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:52026/0 (socket says 172.21.15.110:52026) 2023-10-06T03:30:11.147 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:11.141+0000 7ff7d2ffd700 1 -- 172.21.15.110:0/895255591 learned_addr learned my addr 172.21.15.110:0/895255591 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:30:11.147 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:11.141+0000 7ff7d37fe700 1 --2- 172.21.15.110:0/895255591 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff7d406c2a0 0x7ff7d40b0a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:11.148 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:11.141+0000 7ff7d2ffd700 1 -- 172.21.15.110:0/895255591 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff7d406c2a0 msgr2=0x7ff7d40b0a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:30:11.148 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:11.141+0000 7ff7d2ffd700 1 --2- 172.21.15.110:0/895255591 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff7d406c2a0 0x7ff7d40b0a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:11.148 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:11.141+0000 7ff7d2ffd700 1 -- 172.21.15.110:0/895255591 >> v1:172.21.15.192:6789/0 conn(0x7ff7d40ce6b0 legacy=0x7ff7d406bb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:11.149 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:11.141+0000 7ff7d2ffd700 1 -- 172.21.15.110:0/895255591 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7d40cf0d0 con 0x7ff7d406b7c0 2023-10-06T03:30:11.149 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:11.145+0000 7ff7d2ffd700 1 --2- 172.21.15.110:0/895255591 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff7d406b7c0 0x7ff7d40ce170 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7ff7c80099b0 tx=0x7ff7c8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f80a2f26e91666d6 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:30:11.149 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:11.145+0000 7ff7d1ffb700 1 -- 172.21.15.110:0/895255591 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7c8009ea0 con 0x7ff7d406b7c0 2023-10-06T03:30:11.149 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:11.145+0000 7ff7d1ffb700 1 -- 172.21.15.110:0/895255591 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff7c800b860 con 0x7ff7d406b7c0 2023-10-06T03:30:11.149 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:11.145+0000 7ff7d1ffb700 1 -- 172.21.15.110:0/895255591 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7c800bc00 con 0x7ff7d406b7c0 2023-10-06T03:30:11.149 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:11.145+0000 7ff7d3fff700 1 -- 172.21.15.110:0/895255591 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff7d406b7c0 msgr2=0x7ff7d40ce170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:11.149 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:11.145+0000 7ff7d3fff700 1 --2- 172.21.15.110:0/895255591 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff7d406b7c0 0x7ff7d40ce170 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7ff7c80099b0 tx=0x7ff7c8009250 comp rx=0 tx=0).stop 2023-10-06T03:30:11.149 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:11.145+0000 7ff7d3fff700 1 -- 172.21.15.110:0/895255591 shutdown_connections 2023-10-06T03:30:11.149 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:11.145+0000 7ff7d3fff700 1 --2- 172.21.15.110:0/895255591 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff7d406c2a0 0x7ff7d40b0a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:11.149 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:11.145+0000 7ff7d3fff700 1 --2- 172.21.15.110:0/895255591 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff7d406b7c0 0x7ff7d40ce170 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:11.149 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:11.145+0000 7ff7d3fff700 1 -- 172.21.15.110:0/895255591 >> 172.21.15.110:0/895255591 conn(0x7ff7d40cfea0 msgr2=0x7ff7d40d02a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:30:11.149 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:11.145+0000 7ff7d3fff700 1 -- 172.21.15.110:0/895255591 shutdown_connections 2023-10-06T03:30:11.150 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:11.145+0000 7ff7d3fff700 1 -- 172.21.15.110:0/895255591 wait complete. 2023-10-06T03:30:11.151 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:11.145+0000 7ff7d3fff700 1 Processor -- start 2023-10-06T03:30:11.151 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:11.145+0000 7ff7d3fff700 1 -- start start 2023-10-06T03:30:11.151 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:11.145+0000 7ff7d3fff700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff7d406b7c0 0x7ff7d413efe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:11.152 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:11.145+0000 7ff7d3fff700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff7d406c2a0 0x7ff7d40b2c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:11.152 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:11.145+0000 7ff7d3fff700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff7d40ce6b0 0x7ff7d40b3180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:11.152 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:11.145+0000 7ff7d3fff700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7d40b4780 con 0x7ff7d406c2a0 2023-10-06T03:30:11.152 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:11.145+0000 7ff7d3fff700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7d41598b0 con 0x7ff7d40ce6b0 2023-10-06T03:30:11.152 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:11.145+0000 7ff7d3fff700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7d415a3c0 con 0x7ff7d406b7c0 2023-10-06T03:30:11.152 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:11.145+0000 7ff7d27fc700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff7d406c2a0 0x7ff7d40b2c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:11.152 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:11.145+0000 7ff7d27fc700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff7d406c2a0 0x7ff7d40b2c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:52028/0 (socket says 172.21.15.110:52028) 2023-10-06T03:30:11.152 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:11.145+0000 7ff7d27fc700 1 -- 172.21.15.110:0/2154699849 learned_addr learned my addr 172.21.15.110:0/2154699849 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:30:11.152 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:11.145+0000 7ff7d37fe700 1 --2- 172.21.15.110:0/2154699849 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff7d40ce6b0 0x7ff7d40b3180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:11.152 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:11.145+0000 7ff7d2ffd700 1 --2- 172.21.15.110:0/2154699849 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff7d406b7c0 0x7ff7d413efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:11.152 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:11.145+0000 7ff7d27fc700 1 -- 172.21.15.110:0/2154699849 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff7d406b7c0 msgr2=0x7ff7d413efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:11.152 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:11.145+0000 7ff7d27fc700 1 --2- 172.21.15.110:0/2154699849 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff7d406b7c0 0x7ff7d413efe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:11.153 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:11.145+0000 7ff7d27fc700 1 -- 172.21.15.110:0/2154699849 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff7d40ce6b0 msgr2=0x7ff7d40b3180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:11.153 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:11.145+0000 7ff7d27fc700 1 --2- 172.21.15.110:0/2154699849 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff7d40ce6b0 0x7ff7d40b3180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:11.153 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:11.145+0000 7ff7d27fc700 1 -- 172.21.15.110:0/2154699849 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7c80094f0 con 0x7ff7d406c2a0 2023-10-06T03:30:11.153 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:11.145+0000 7ff7d2ffd700 1 --2- 172.21.15.110:0/2154699849 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff7d406b7c0 0x7ff7d413efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-06T03:30:11.153 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:11.149+0000 7ff7d27fc700 1 --2- 172.21.15.110:0/2154699849 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff7d406c2a0 0x7ff7d40b2c40 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7ff7c400c510 tx=0x7ff7c400c820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:30:11.153 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:11.149+0000 7ff7baffd700 1 -- 172.21.15.110:0/2154699849 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7c4011590 con 0x7ff7d406c2a0 2023-10-06T03:30:11.153 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:11.149+0000 7ff7baffd700 1 -- 172.21.15.110:0/2154699849 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff7c4011710 con 0x7ff7d406c2a0 2023-10-06T03:30:11.153 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:11.149+0000 7ff7d3fff700 1 -- 172.21.15.110:0/2154699849 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff7d4159a90 con 0x7ff7d406c2a0 2023-10-06T03:30:11.154 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:11.149+0000 7ff7d3fff700 1 -- 172.21.15.110:0/2154699849 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7d413f520 con 0x7ff7d406c2a0 2023-10-06T03:30:11.154 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:11.149+0000 7ff7baffd700 1 -- 172.21.15.110:0/2154699849 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7c4011ad0 con 0x7ff7d406c2a0 2023-10-06T03:30:11.154 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:11.149+0000 7ff7baffd700 1 -- 172.21.15.110:0/2154699849 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7ff7c401b3b0 con 0x7ff7d406c2a0 2023-10-06T03:30:11.155 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:11.149+0000 7ff7d3fff700 1 -- 172.21.15.110:0/2154699849 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff7d4159c80 con 0x7ff7d406c2a0 2023-10-06T03:30:11.155 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:11.149+0000 7ff7baffd700 1 --2- 172.21.15.110:0/2154699849 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7ff7c0060550 0x7ff7c0062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:11.155 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:11.149+0000 7ff7d2ffd700 1 --2- 172.21.15.110:0/2154699849 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7ff7c0060550 0x7ff7c0062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:11.155 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:11.149+0000 7ff7baffd700 1 -- 172.21.15.110:0/2154699849 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6809+0+0 (secure 0 0 0) 0x7ff7c407bd80 con 0x7ff7d406c2a0 2023-10-06T03:30:11.156 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:11.153+0000 7ff7d2ffd700 1 --2- 172.21.15.110:0/2154699849 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7ff7c0060550 0x7ff7c0062a00 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7ff7c80099b0 tx=0x7ff7c8017000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:30:11.163 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:11.157+0000 7ff7baffd700 1 -- 172.21.15.110:0/2154699849 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7ff7c404be90 con 0x7ff7d406c2a0 2023-10-06T03:30:11.318 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:11.313+0000 7ff7d3fff700 1 -- 172.21.15.110:0/2154699849 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7ff7d4159eb0 con 0x7ff7d406c2a0 2023-10-06T03:30:11.319 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:11.313+0000 7ff7baffd700 1 -- 172.21.15.110:0/2154699849 <== mon.0 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v28) v1 ==== 93+0+10 (secure 0 0 0) 0x7ff7d4159eb0 con 0x7ff7d406c2a0 2023-10-06T03:30:11.319 INFO:teuthology.orchestra.run.smithi110.stdout:pg_num: 1 2023-10-06T03:30:11.321 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:11.317+0000 7ff7d3fff700 1 -- 172.21.15.110:0/2154699849 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7ff7c0060550 msgr2=0x7ff7c0062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:11.321 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:11.317+0000 7ff7d3fff700 1 --2- 172.21.15.110:0/2154699849 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7ff7c0060550 0x7ff7c0062a00 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7ff7c80099b0 tx=0x7ff7c8017000 comp rx=0 tx=0).stop 2023-10-06T03:30:11.321 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:11.317+0000 7ff7d3fff700 1 -- 172.21.15.110:0/2154699849 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff7d406c2a0 msgr2=0x7ff7d40b2c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:11.321 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:11.317+0000 7ff7d3fff700 1 --2- 172.21.15.110:0/2154699849 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff7d406c2a0 0x7ff7d40b2c40 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7ff7c400c510 tx=0x7ff7c400c820 comp rx=0 tx=0).stop 2023-10-06T03:30:11.321 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:11.317+0000 7ff7d3fff700 1 -- 172.21.15.110:0/2154699849 shutdown_connections 2023-10-06T03:30:11.322 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:11.317+0000 7ff7d3fff700 1 --2- 172.21.15.110:0/2154699849 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7ff7c0060550 0x7ff7c0062a00 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:11.322 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:11.317+0000 7ff7d3fff700 1 --2- 172.21.15.110:0/2154699849 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff7d40ce6b0 0x7ff7d40b3180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:11.322 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:11.317+0000 7ff7d3fff700 1 --2- 172.21.15.110:0/2154699849 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff7d406c2a0 0x7ff7d40b2c40 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:11.322 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:11.317+0000 7ff7d3fff700 1 --2- 172.21.15.110:0/2154699849 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff7d406b7c0 0x7ff7d413efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:11.322 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:11.317+0000 7ff7d3fff700 1 -- 172.21.15.110:0/2154699849 >> 172.21.15.110:0/2154699849 conn(0x7ff7d40cfea0 msgr2=0x7ff7d406bb90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:30:11.322 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:11.317+0000 7ff7d3fff700 1 -- 172.21.15.110:0/2154699849 shutdown_connections 2023-10-06T03:30:11.322 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:11.317+0000 7ff7d3fff700 1 -- 172.21.15.110:0/2154699849 wait complete. 2023-10-06T03:30:11.333 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-10-06T03:30:11.445 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:11.441+0000 7fcf83d23700 1 Processor -- start 2023-10-06T03:30:11.445 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:11.441+0000 7fcf83d23700 1 -- start start 2023-10-06T03:30:11.448 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:11.445+0000 7fcf83d23700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fcf7c15d360 0x7fcf7c0af440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:11.448 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:11.445+0000 7fcf83d23700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf7c0cf5b0 con 0x7fcf7c0cfa40 2023-10-06T03:30:11.448 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:11.445+0000 7fcf83d23700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf7c0ce9a0 con 0x7fcf7c0af070 2023-10-06T03:30:11.448 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:11.445+0000 7fcf83d23700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf7c0cec80 con 0x7fcf7c15d360 2023-10-06T03:30:11.448 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:11.445+0000 7fcf822c0700 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7fcf7c0cfa40 0x7fcf7c0afbb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:53226/0 (socket says 172.21.15.110:53226) 2023-10-06T03:30:11.448 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:11.445+0000 7fcf822c0700 1 -- 172.21.15.110:0/3744756066 learned_addr learned my addr 172.21.15.110:0/3744756066 (peer_addr_for_me v1:172.21.15.110:0/0) 2023-10-06T03:30:11.448 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:11.445+0000 7fcf812be700 1 --2- 172.21.15.110:0/3744756066 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fcf7c15d360 0x7fcf7c0af440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:11.449 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:11.445+0000 7fcf80abd700 1 -- 172.21.15.110:0/3744756066 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1206299776 0 0) 0x7fcf7c0cf5b0 con 0x7fcf7c0cfa40 2023-10-06T03:30:11.449 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:11.445+0000 7fcf80abd700 1 -- 172.21.15.110:0/3744756066 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcf68003570 con 0x7fcf7c0cfa40 2023-10-06T03:30:11.449 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:11.445+0000 7fcf812be700 1 -- 172.21.15.110:0/3744756066 >> v1:172.21.15.192:6789/0 conn(0x7fcf7c0af070 legacy=0x7fcf7c15cc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:11.450 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:11.445+0000 7fcf812be700 1 -- 172.21.15.110:0/3744756066 >> v1:172.21.15.110:6789/0 conn(0x7fcf7c0cfa40 legacy=0x7fcf7c0afbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:11.450 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:11.445+0000 7fcf812be700 1 -- 172.21.15.110:0/3744756066 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf7c0cef60 con 0x7fcf7c15d360 2023-10-06T03:30:11.450 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:11.445+0000 7fcf812be700 1 --2- 172.21.15.110:0/3744756066 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fcf7c15d360 0x7fcf7c0af440 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fcf6c009910 tx=0x7fcf6c0091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=48cdf069ed8f0deb server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:30:11.450 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:11.445+0000 7fcf80abd700 1 -- 172.21.15.110:0/3744756066 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf6c00b680 con 0x7fcf7c15d360 2023-10-06T03:30:11.450 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:11.445+0000 7fcf80abd700 1 -- 172.21.15.110:0/3744756066 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf6c00b7e0 con 0x7fcf7c15d360 2023-10-06T03:30:11.451 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:11.445+0000 7fcf80abd700 1 -- 172.21.15.110:0/3744756066 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf6c00bba0 con 0x7fcf7c15d360 2023-10-06T03:30:11.451 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:11.445+0000 7fcf83d23700 1 -- 172.21.15.110:0/3744756066 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fcf7c15d360 msgr2=0x7fcf7c0af440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:11.451 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:11.445+0000 7fcf83d23700 1 --2- 172.21.15.110:0/3744756066 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fcf7c15d360 0x7fcf7c0af440 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fcf6c009910 tx=0x7fcf6c0091a0 comp rx=0 tx=0).stop 2023-10-06T03:30:11.451 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:11.449+0000 7fcf83d23700 1 -- 172.21.15.110:0/3744756066 shutdown_connections 2023-10-06T03:30:11.452 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:11.449+0000 7fcf83d23700 1 --2- 172.21.15.110:0/3744756066 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fcf7c15d360 0x7fcf7c0af440 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:11.452 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:11.449+0000 7fcf83d23700 1 -- 172.21.15.110:0/3744756066 >> 172.21.15.110:0/3744756066 conn(0x7fcf7c06e840 msgr2=0x7fcf7c0d0010 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:30:11.452 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:11.449+0000 7fcf83d23700 1 -- 172.21.15.110:0/3744756066 shutdown_connections 2023-10-06T03:30:11.452 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:11.449+0000 7fcf83d23700 1 -- 172.21.15.110:0/3744756066 wait complete. 2023-10-06T03:30:11.453 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:11.449+0000 7fcf83d23700 1 Processor -- start 2023-10-06T03:30:11.453 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:11.449+0000 7fcf83d23700 1 -- start start 2023-10-06T03:30:11.455 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:11.449+0000 7fcf83d23700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcf7c0af070 0x7fcf7c15bf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:11.455 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:11.449+0000 7fcf83d23700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcf7c0cfa40 0x7fcf7c159500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:11.455 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:11.449+0000 7fcf83d23700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fcf7c15d360 0x7fcf7c155480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:11.456 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:11.449+0000 7fcf83d23700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf7c0c82b0 con 0x7fcf7c0cfa40 2023-10-06T03:30:11.456 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:11.449+0000 7fcf83d23700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf7c159a40 con 0x7fcf7c0af070 2023-10-06T03:30:11.456 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:11.449+0000 7fcf83d23700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf7c159d10 con 0x7fcf7c15d360 2023-10-06T03:30:11.456 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:11.449+0000 7fcf812be700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcf7c0cfa40 0x7fcf7c159500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:11.456 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:11.449+0000 7fcf812be700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcf7c0cfa40 0x7fcf7c159500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:52042/0 (socket says 172.21.15.110:52042) 2023-10-06T03:30:11.456 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:11.449+0000 7fcf812be700 1 -- 172.21.15.110:0/1603939519 learned_addr learned my addr 172.21.15.110:0/1603939519 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:30:11.456 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:11.449+0000 7fcf822c0700 1 --2- 172.21.15.110:0/1603939519 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fcf7c15d360 0x7fcf7c155480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:11.456 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:11.449+0000 7fcf812be700 1 -- 172.21.15.110:0/1603939519 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fcf7c15d360 msgr2=0x7fcf7c155480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:11.456 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:11.449+0000 7fcf812be700 1 --2- 172.21.15.110:0/1603939519 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fcf7c15d360 0x7fcf7c155480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:11.456 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:11.449+0000 7fcf812be700 1 -- 172.21.15.110:0/1603939519 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcf7c0af070 msgr2=0x7fcf7c15bf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:11.456 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:11.449+0000 7fcf812be700 1 --2- 172.21.15.110:0/1603939519 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcf7c0af070 0x7fcf7c15bf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:11.457 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:11.449+0000 7fcf812be700 1 -- 172.21.15.110:0/1603939519 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf6c009420 con 0x7fcf7c0cfa40 2023-10-06T03:30:11.457 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:11.449+0000 7fcf822c0700 1 --2- 172.21.15.110:0/1603939519 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fcf7c15d360 0x7fcf7c155480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-06T03:30:11.457 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:11.449+0000 7fcf812be700 1 --2- 172.21.15.110:0/1603939519 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcf7c0cfa40 0x7fcf7c159500 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fcf7c06f840 tx=0x7fcf6c017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:30:11.457 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:11.453+0000 7fcf71ffb700 1 -- 172.21.15.110:0/1603939519 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf6c00bb60 con 0x7fcf7c0cfa40 2023-10-06T03:30:11.457 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:11.453+0000 7fcf71ffb700 1 -- 172.21.15.110:0/1603939519 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf6c01a430 con 0x7fcf7c0cfa40 2023-10-06T03:30:11.457 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:11.453+0000 7fcf71ffb700 1 -- 172.21.15.110:0/1603939519 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf6c01a7d0 con 0x7fcf7c0cfa40 2023-10-06T03:30:11.457 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:11.453+0000 7fcf83d23700 1 -- 172.21.15.110:0/1603939519 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf7c159ff0 con 0x7fcf7c0cfa40 2023-10-06T03:30:11.457 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:11.453+0000 7fcf83d23700 1 -- 172.21.15.110:0/1603939519 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf7c15a5b0 con 0x7fcf7c0cfa40 2023-10-06T03:30:11.458 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:11.453+0000 7fcf83d23700 1 -- 172.21.15.110:0/1603939519 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf7c06ff60 con 0x7fcf7c0cfa40 2023-10-06T03:30:11.458 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:11.453+0000 7fcf71ffb700 1 -- 172.21.15.110:0/1603939519 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7fcf6c01ab40 con 0x7fcf7c0cfa40 2023-10-06T03:30:11.458 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:11.453+0000 7fcf71ffb700 1 --2- 172.21.15.110:0/1603939519 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fcf60060550 0x7fcf60062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:11.459 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:11.453+0000 7fcf81abf700 1 --2- 172.21.15.110:0/1603939519 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fcf60060550 0x7fcf60062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:11.459 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:11.453+0000 7fcf71ffb700 1 -- 172.21.15.110:0/1603939519 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6809+0+0 (secure 0 0 0) 0x7fcf6c07e750 con 0x7fcf7c0cfa40 2023-10-06T03:30:11.459 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:11.457+0000 7fcf81abf700 1 --2- 172.21.15.110:0/1603939519 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fcf60060550 0x7fcf60062a00 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fcf78001730 tx=0x7fcf78007b40 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:30:11.467 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:11.465+0000 7fcf71ffb700 1 -- 172.21.15.110:0/1603939519 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fcf6c04e8f0 con 0x7fcf7c0cfa40 2023-10-06T03:30:11.628 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:11.625+0000 7fcf83d23700 1 -- 172.21.15.110:0/1603939519 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7fcf7c0c83f0 con 0x7fcf7c0cfa40 2023-10-06T03:30:11.628 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:11.625+0000 7fcf71ffb700 1 -- 172.21.15.110:0/1603939519 <== mon.0 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v28) v1 ==== 92+0+10 (secure 0 0 0) 0x7fcf6c02eed0 con 0x7fcf7c0cfa40 2023-10-06T03:30:11.629 INFO:teuthology.orchestra.run.smithi110.stdout:pg_num: 8 2023-10-06T03:30:11.633 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:11.629+0000 7fcf83d23700 1 -- 172.21.15.110:0/1603939519 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fcf60060550 msgr2=0x7fcf60062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:11.633 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:11.629+0000 7fcf83d23700 1 --2- 172.21.15.110:0/1603939519 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fcf60060550 0x7fcf60062a00 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fcf78001730 tx=0x7fcf78007b40 comp rx=0 tx=0).stop 2023-10-06T03:30:11.633 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:11.629+0000 7fcf83d23700 1 -- 172.21.15.110:0/1603939519 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcf7c0cfa40 msgr2=0x7fcf7c159500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:11.633 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:11.629+0000 7fcf83d23700 1 --2- 172.21.15.110:0/1603939519 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcf7c0cfa40 0x7fcf7c159500 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fcf7c06f840 tx=0x7fcf6c017000 comp rx=0 tx=0).stop 2023-10-06T03:30:11.634 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:11.629+0000 7fcf83d23700 1 -- 172.21.15.110:0/1603939519 shutdown_connections 2023-10-06T03:30:11.634 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:11.629+0000 7fcf83d23700 1 --2- 172.21.15.110:0/1603939519 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fcf60060550 0x7fcf60062a00 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:11.634 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:11.629+0000 7fcf83d23700 1 --2- 172.21.15.110:0/1603939519 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fcf7c15d360 0x7fcf7c155480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:11.634 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:11.629+0000 7fcf83d23700 1 --2- 172.21.15.110:0/1603939519 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcf7c0cfa40 0x7fcf7c159500 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:11.634 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:11.629+0000 7fcf83d23700 1 --2- 172.21.15.110:0/1603939519 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcf7c0af070 0x7fcf7c15bf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:11.634 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:11.629+0000 7fcf83d23700 1 -- 172.21.15.110:0/1603939519 >> 172.21.15.110:0/1603939519 conn(0x7fcf7c06e840 msgr2=0x7fcf7c1562f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:30:11.634 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:11.629+0000 7fcf83d23700 1 -- 172.21.15.110:0/1603939519 shutdown_connections 2023-10-06T03:30:11.634 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:11.629+0000 7fcf83d23700 1 -- 172.21.15.110:0/1603939519 wait complete. 2023-10-06T03:30:11.647 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-06T03:30:11.756 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:11.753+0000 7fef08c0a700 1 Processor -- start 2023-10-06T03:30:11.756 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:11.753+0000 7fef08c0a700 1 -- start start 2023-10-06T03:30:11.758 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:11.753+0000 7fef08c0a700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fef040af070 0x7fef040af440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:11.758 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:11.753+0000 7fef08c0a700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fef0415d360 0x7fef040cb220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:11.758 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:11.753+0000 7fef08c0a700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef040cf5b0 con 0x7fef040cfa40 2023-10-06T03:30:11.758 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:11.753+0000 7fef08c0a700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fef040ce9a0 con 0x7fef040af070 2023-10-06T03:30:11.758 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:11.753+0000 7fef08c0a700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef040cec80 con 0x7fef0415d360 2023-10-06T03:30:11.758 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:11.753+0000 7fef01d9b700 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7fef040cfa40 0x7fef0415cc50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:53232/0 (socket says 172.21.15.110:53232) 2023-10-06T03:30:11.758 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:11.753+0000 7fef01d9b700 1 -- 172.21.15.110:0/2417494086 learned_addr learned my addr 172.21.15.110:0/2417494086 (peer_addr_for_me v1:172.21.15.110:0/0) 2023-10-06T03:30:11.759 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:11.753+0000 7fef02d9d700 1 --2- 172.21.15.110:0/2417494086 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fef0415d360 0x7fef040cb220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:11.759 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:11.753+0000 7fef0259c700 1 --2- 172.21.15.110:0/2417494086 >> v2:172.21.15.192:3300/0 conn(0x7fef040af070 0x7fef040af440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:11.759 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:11.753+0000 7fef0159a700 1 -- 172.21.15.110:0/2417494086 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3664806750 0 0) 0x7fef040cf5b0 con 0x7fef040cfa40 2023-10-06T03:30:11.759 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:11.753+0000 7fef0159a700 1 -- 172.21.15.110:0/2417494086 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feeec003570 con 0x7fef040cfa40 2023-10-06T03:30:11.759 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:11.753+0000 7fef02d9d700 1 -- 172.21.15.110:0/2417494086 >> v2:172.21.15.192:3300/0 conn(0x7fef040af070 msgr2=0x7fef040af440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:30:11.759 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:11.753+0000 7fef02d9d700 1 --2- 172.21.15.110:0/2417494086 >> v2:172.21.15.192:3300/0 conn(0x7fef040af070 0x7fef040af440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:11.759 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:11.753+0000 7fef02d9d700 1 -- 172.21.15.110:0/2417494086 >> v1:172.21.15.110:6789/0 conn(0x7fef040cfa40 legacy=0x7fef0415cc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:11.759 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:11.753+0000 7fef02d9d700 1 -- 172.21.15.110:0/2417494086 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef040cef60 con 0x7fef0415d360 2023-10-06T03:30:11.760 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:11.753+0000 7fef02d9d700 1 --2- 172.21.15.110:0/2417494086 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fef0415d360 0x7fef040cb220 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7feefc009e60 tx=0x7feefc009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cfc2e5e2c9fd0238 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:30:11.760 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:11.753+0000 7fef0159a700 1 -- 172.21.15.110:0/2417494086 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feefc00e070 con 0x7fef0415d360 2023-10-06T03:30:11.761 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:11.753+0000 7fef0159a700 1 -- 172.21.15.110:0/2417494086 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feefc00bab0 con 0x7fef0415d360 2023-10-06T03:30:11.761 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:11.753+0000 7fef0159a700 1 -- 172.21.15.110:0/2417494086 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feefc00be30 con 0x7fef0415d360 2023-10-06T03:30:11.761 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:11.757+0000 7fef08c0a700 1 -- 172.21.15.110:0/2417494086 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fef0415d360 msgr2=0x7fef040cb220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:11.761 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:11.757+0000 7fef08c0a700 1 --2- 172.21.15.110:0/2417494086 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fef0415d360 0x7fef040cb220 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7feefc009e60 tx=0x7feefc009780 comp rx=0 tx=0).stop 2023-10-06T03:30:11.761 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:11.757+0000 7fef08c0a700 1 -- 172.21.15.110:0/2417494086 shutdown_connections 2023-10-06T03:30:11.761 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:11.757+0000 7fef08c0a700 1 --2- 172.21.15.110:0/2417494086 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fef0415d360 0x7fef040cb220 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:11.761 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:11.757+0000 7fef08c0a700 1 --2- 172.21.15.110:0/2417494086 >> v2:172.21.15.192:3300/0 conn(0x7fef040af070 0x7fef040af440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:11.761 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:11.757+0000 7fef08c0a700 1 -- 172.21.15.110:0/2417494086 >> 172.21.15.110:0/2417494086 conn(0x7fef0406e840 msgr2=0x7fef040d0010 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:30:11.761 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:11.757+0000 7fef08c0a700 1 -- 172.21.15.110:0/2417494086 shutdown_connections 2023-10-06T03:30:11.761 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:11.757+0000 7fef08c0a700 1 -- 172.21.15.110:0/2417494086 wait complete. 2023-10-06T03:30:11.762 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:11.757+0000 7fef08c0a700 1 Processor -- start 2023-10-06T03:30:11.762 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:11.757+0000 7fef08c0a700 1 -- start start 2023-10-06T03:30:11.764 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:11.761+0000 7fef08c0a700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fef040af070 0x7fef04153890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:11.765 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:11.761+0000 7fef08c0a700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fef040cfa40 0x7fef0415ba50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:11.765 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:11.761+0000 7fef08c0a700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fef0415d360 0x7fef040b16a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:11.765 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:11.761+0000 7fef08c0a700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef041597e0 con 0x7fef040cfa40 2023-10-06T03:30:11.765 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:11.761+0000 7fef08c0a700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef04159ac0 con 0x7fef040af070 2023-10-06T03:30:11.765 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:11.761+0000 7fef08c0a700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef04159da0 con 0x7fef0415d360 2023-10-06T03:30:11.765 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:11.761+0000 7fef01d9b700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fef040cfa40 0x7fef0415ba50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:11.765 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:11.761+0000 7fef01d9b700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fef040cfa40 0x7fef0415ba50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:52046/0 (socket says 172.21.15.110:52046) 2023-10-06T03:30:11.765 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:11.761+0000 7fef01d9b700 1 -- 172.21.15.110:0/478857575 learned_addr learned my addr 172.21.15.110:0/478857575 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:30:11.766 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:11.761+0000 7fef0259c700 1 --2- 172.21.15.110:0/478857575 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fef040af070 0x7fef04153890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:11.766 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:11.761+0000 7fef02d9d700 1 --2- 172.21.15.110:0/478857575 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fef0415d360 0x7fef040b16a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:11.766 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:11.761+0000 7fef01d9b700 1 -- 172.21.15.110:0/478857575 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fef0415d360 msgr2=0x7fef040b16a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:11.766 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:11.761+0000 7fef01d9b700 1 --2- 172.21.15.110:0/478857575 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fef0415d360 0x7fef040b16a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:11.766 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:11.761+0000 7fef01d9b700 1 -- 172.21.15.110:0/478857575 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fef040af070 msgr2=0x7fef04153890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:11.766 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:11.761+0000 7fef01d9b700 1 --2- 172.21.15.110:0/478857575 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fef040af070 0x7fef04153890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:11.766 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:11.761+0000 7fef01d9b700 1 -- 172.21.15.110:0/478857575 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feefc009a20 con 0x7fef040cfa40 2023-10-06T03:30:11.766 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:11.761+0000 7fef01d9b700 1 --2- 172.21.15.110:0/478857575 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fef040cfa40 0x7fef0415ba50 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7feef400c7c0 tx=0x7feef4007d90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:30:11.767 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:11.761+0000 7feef27fc700 1 -- 172.21.15.110:0/478857575 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feef4005c00 con 0x7fef040cfa40 2023-10-06T03:30:11.767 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:11.761+0000 7feef27fc700 1 -- 172.21.15.110:0/478857575 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feef4005d60 con 0x7fef040cfa40 2023-10-06T03:30:11.767 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:11.761+0000 7fef08c0a700 1 -- 172.21.15.110:0/478857575 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef0415a080 con 0x7fef040cfa40 2023-10-06T03:30:11.767 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:11.761+0000 7feef27fc700 1 -- 172.21.15.110:0/478857575 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feef40072e0 con 0x7fef040cfa40 2023-10-06T03:30:11.768 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:11.761+0000 7fef08c0a700 1 -- 172.21.15.110:0/478857575 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef040b1be0 con 0x7fef040cfa40 2023-10-06T03:30:11.768 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:11.761+0000 7fef08c0a700 1 -- 172.21.15.110:0/478857575 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef0415a580 con 0x7fef040cfa40 2023-10-06T03:30:11.768 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:11.765+0000 7feef27fc700 1 -- 172.21.15.110:0/478857575 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7feef4007690 con 0x7fef040cfa40 2023-10-06T03:30:11.768 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:11.765+0000 7feef27fc700 1 --2- 172.21.15.110:0/478857575 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7feee4060550 0x7feee4062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:11.769 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:11.765+0000 7fef0259c700 1 --2- 172.21.15.110:0/478857575 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7feee4060550 0x7feee4062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:11.769 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:11.765+0000 7feef27fc700 1 -- 172.21.15.110:0/478857575 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6809+0+0 (secure 0 0 0) 0x7feef407beb0 con 0x7fef040cfa40 2023-10-06T03:30:11.769 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:11.765+0000 7fef0259c700 1 --2- 172.21.15.110:0/478857575 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7feee4060550 0x7feee4062a00 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7feef8009db0 tx=0x7feef80091a0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:30:11.775 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:11.769+0000 7feef27fc700 1 -- 172.21.15.110:0/478857575 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7feef4049be0 con 0x7fef040cfa40 2023-10-06T03:30:11.934 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:11.929+0000 7fef08c0a700 1 -- 172.21.15.110:0/478857575 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fef0406ff60 con 0x7fef040cfa40 2023-10-06T03:30:11.935 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:11.929+0000 7feef27fc700 1 -- 172.21.15.110:0/478857575 <== mon.0 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v28) v1 ==== 104+0+11 (secure 0 0 0) 0x7feef4017020 con 0x7fef040cfa40 2023-10-06T03:30:11.935 INFO:teuthology.orchestra.run.smithi110.stdout:pg_num: 32 2023-10-06T03:30:11.939 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:11.933+0000 7fef08c0a700 1 -- 172.21.15.110:0/478857575 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7feee4060550 msgr2=0x7feee4062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:11.939 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:11.933+0000 7fef08c0a700 1 --2- 172.21.15.110:0/478857575 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7feee4060550 0x7feee4062a00 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7feef8009db0 tx=0x7feef80091a0 comp rx=0 tx=0).stop 2023-10-06T03:30:11.939 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:11.933+0000 7fef08c0a700 1 -- 172.21.15.110:0/478857575 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fef040cfa40 msgr2=0x7fef0415ba50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:11.939 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:11.933+0000 7fef08c0a700 1 --2- 172.21.15.110:0/478857575 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fef040cfa40 0x7fef0415ba50 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7feef400c7c0 tx=0x7feef4007d90 comp rx=0 tx=0).stop 2023-10-06T03:30:11.939 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:11.933+0000 7fef08c0a700 1 -- 172.21.15.110:0/478857575 shutdown_connections 2023-10-06T03:30:11.939 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:11.933+0000 7fef08c0a700 1 --2- 172.21.15.110:0/478857575 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7feee4060550 0x7feee4062a00 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:11.939 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:11.933+0000 7fef08c0a700 1 --2- 172.21.15.110:0/478857575 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fef0415d360 0x7fef040b16a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:11.939 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:11.933+0000 7fef08c0a700 1 --2- 172.21.15.110:0/478857575 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fef040cfa40 0x7fef0415ba50 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:11.939 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:11.933+0000 7fef08c0a700 1 --2- 172.21.15.110:0/478857575 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fef040af070 0x7fef04153890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:11.940 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:11.933+0000 7fef08c0a700 1 -- 172.21.15.110:0/478857575 >> 172.21.15.110:0/478857575 conn(0x7fef0406e840 msgr2=0x7fef040c5640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:30:11.940 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:11.933+0000 7fef08c0a700 1 -- 172.21.15.110:0/478857575 shutdown_connections 2023-10-06T03:30:11.940 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:11.933+0000 7fef08c0a700 1 -- 172.21.15.110:0/478857575 wait complete. 2023-10-06T03:30:11.953 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-06T03:30:12.063 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:12.057+0000 7f5c44675700 1 Processor -- start 2023-10-06T03:30:12.063 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:12.061+0000 7f5c44675700 1 -- start start 2023-10-06T03:30:12.064 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:12.061+0000 7f5c44675700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5c3c156d40 0x7f5c3c15a560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:12.065 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:12.061+0000 7f5c44675700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f5c3c157110 0x7f5c3c06d120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:12.065 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:12.061+0000 7f5c44675700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f5c3c06b7c0 0x7f5c3c0670a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:12.065 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:12.061+0000 7f5c44675700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5c3c06c080 con 0x7f5c3c06b7c0 2023-10-06T03:30:12.065 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:12.061+0000 7f5c44675700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5c3c06c360 con 0x7f5c3c157110 2023-10-06T03:30:12.065 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:12.061+0000 7f5c44675700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c3c06c640 con 0x7f5c3c156d40 2023-10-06T03:30:12.066 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:12.061+0000 7f5c42c12700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f5c3c06b7c0 0x7f5c3c0670a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:12.066 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:12.061+0000 7f5c42c12700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f5c3c06b7c0 0x7f5c3c0670a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:52052/0 (socket says 172.21.15.110:52052) 2023-10-06T03:30:12.066 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:12.061+0000 7f5c42c12700 1 -- 172.21.15.110:0/3606585068 learned_addr learned my addr 172.21.15.110:0/3606585068 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:30:12.066 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:12.061+0000 7f5c42411700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5c3c156d40 0x7f5c3c15a560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:12.066 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:12.061+0000 7f5c41c10700 1 --2- 172.21.15.110:0/3606585068 >> v2:172.21.15.192:3300/0 conn(0x7f5c3c157110 0x7f5c3c06d120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:12.067 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:12.061+0000 7f5c42c12700 1 -- 172.21.15.110:0/3606585068 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5c3c156d40 msgr2=0x7f5c3c15a560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:30:12.067 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:12.061+0000 7f5c42c12700 1 --2- 172.21.15.110:0/3606585068 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5c3c156d40 0x7f5c3c15a560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:12.067 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:12.061+0000 7f5c42c12700 1 -- 172.21.15.110:0/3606585068 >> v2:172.21.15.192:3300/0 conn(0x7f5c3c157110 msgr2=0x7f5c3c06d120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:30:12.067 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:12.061+0000 7f5c42c12700 1 --2- 172.21.15.110:0/3606585068 >> v2:172.21.15.192:3300/0 conn(0x7f5c3c157110 0x7f5c3c06d120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:12.067 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:12.061+0000 7f5c42c12700 1 -- 172.21.15.110:0/3606585068 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c3c06c920 con 0x7f5c3c06b7c0 2023-10-06T03:30:12.067 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:12.061+0000 7f5c41c10700 1 --2- 172.21.15.110:0/3606585068 >> v2:172.21.15.192:3300/0 conn(0x7f5c3c157110 0x7f5c3c06d120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-06T03:30:12.067 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:12.061+0000 7f5c42c12700 1 --2- 172.21.15.110:0/3606585068 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5c3c06b7c0 0x7f5c3c0670a0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f5c3c0638e0 tx=0x7f5c34009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=23df8db2c421f531 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:30:12.067 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:12.061+0000 7f5c4140f700 1 -- 172.21.15.110:0/3606585068 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c3400e070 con 0x7f5c3c06b7c0 2023-10-06T03:30:12.067 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:12.061+0000 7f5c4140f700 1 -- 172.21.15.110:0/3606585068 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c3400b970 con 0x7f5c3c06b7c0 2023-10-06T03:30:12.067 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:12.061+0000 7f5c4140f700 1 -- 172.21.15.110:0/3606585068 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c340092e0 con 0x7f5c3c06b7c0 2023-10-06T03:30:12.068 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:12.061+0000 7f5c44675700 1 -- 172.21.15.110:0/3606585068 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5c3c06b7c0 msgr2=0x7f5c3c0670a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:12.068 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:12.061+0000 7f5c44675700 1 --2- 172.21.15.110:0/3606585068 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5c3c06b7c0 0x7f5c3c0670a0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f5c3c0638e0 tx=0x7f5c34009840 comp rx=0 tx=0).stop 2023-10-06T03:30:12.068 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:12.061+0000 7f5c44675700 1 -- 172.21.15.110:0/3606585068 shutdown_connections 2023-10-06T03:30:12.069 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:12.061+0000 7f5c44675700 1 --2- 172.21.15.110:0/3606585068 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5c3c06b7c0 0x7f5c3c0670a0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:12.069 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:12.061+0000 7f5c44675700 1 --2- 172.21.15.110:0/3606585068 >> v2:172.21.15.192:3300/0 conn(0x7f5c3c157110 0x7f5c3c06d120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:12.069 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:12.061+0000 7f5c44675700 1 --2- 172.21.15.110:0/3606585068 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5c3c156d40 0x7f5c3c15a560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:12.069 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:12.061+0000 7f5c44675700 1 -- 172.21.15.110:0/3606585068 >> 172.21.15.110:0/3606585068 conn(0x7f5c3c0ccd30 msgr2=0x7f5c3c0cd130 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:30:12.069 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:12.061+0000 7f5c44675700 1 -- 172.21.15.110:0/3606585068 shutdown_connections 2023-10-06T03:30:12.069 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:12.061+0000 7f5c44675700 1 -- 172.21.15.110:0/3606585068 wait complete. 2023-10-06T03:30:12.069 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:12.065+0000 7f5c44675700 1 Processor -- start 2023-10-06T03:30:12.069 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:12.065+0000 7f5c44675700 1 -- start start 2023-10-06T03:30:12.071 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:12.065+0000 7f5c44675700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5c3c06b7c0 0x7f5c3c0b1c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:12.071 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:12.065+0000 7f5c44675700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5c3c156d40 0x7f5c3c156440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:12.071 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:12.065+0000 7f5c44675700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5c3c157110 0x7f5c3c0670a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:12.071 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:12.065+0000 7f5c44675700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c3c0c2520 con 0x7f5c3c06b7c0 2023-10-06T03:30:12.071 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:12.065+0000 7f5c44675700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c3c0b2160 con 0x7f5c3c157110 2023-10-06T03:30:12.071 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:12.065+0000 7f5c44675700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c3c156980 con 0x7f5c3c156d40 2023-10-06T03:30:12.072 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:12.065+0000 7f5c42411700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5c3c06b7c0 0x7f5c3c0b1c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:12.072 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:12.065+0000 7f5c42411700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5c3c06b7c0 0x7f5c3c0b1c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:52064/0 (socket says 172.21.15.110:52064) 2023-10-06T03:30:12.072 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:12.065+0000 7f5c42411700 1 -- 172.21.15.110:0/515934378 learned_addr learned my addr 172.21.15.110:0/515934378 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:30:12.072 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:12.065+0000 7f5c42c12700 1 --2- 172.21.15.110:0/515934378 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5c3c157110 0x7f5c3c0670a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:12.072 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:12.065+0000 7f5c41c10700 1 --2- 172.21.15.110:0/515934378 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5c3c156d40 0x7f5c3c156440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:12.072 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:12.065+0000 7f5c42411700 1 -- 172.21.15.110:0/515934378 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5c3c156d40 msgr2=0x7f5c3c156440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:12.072 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:12.065+0000 7f5c42411700 1 --2- 172.21.15.110:0/515934378 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5c3c156d40 0x7f5c3c156440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:12.072 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:12.065+0000 7f5c42411700 1 -- 172.21.15.110:0/515934378 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5c3c157110 msgr2=0x7f5c3c0670a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:12.072 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:12.065+0000 7f5c42411700 1 --2- 172.21.15.110:0/515934378 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5c3c157110 0x7f5c3c0670a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:12.073 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:12.065+0000 7f5c42411700 1 -- 172.21.15.110:0/515934378 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c30009420 con 0x7f5c3c06b7c0 2023-10-06T03:30:12.073 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:12.065+0000 7f5c42411700 1 --2- 172.21.15.110:0/515934378 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5c3c06b7c0 0x7f5c3c0b1c20 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f5c3800ec80 tx=0x7f5c380091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:30:12.073 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:12.065+0000 7f5c2e7fc700 1 -- 172.21.15.110:0/515934378 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c3800c850 con 0x7f5c3c06b7c0 2023-10-06T03:30:12.073 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:12.065+0000 7f5c2e7fc700 1 -- 172.21.15.110:0/515934378 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c3800c9b0 con 0x7f5c3c06b7c0 2023-10-06T03:30:12.073 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:12.065+0000 7f5c44675700 1 -- 172.21.15.110:0/515934378 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c34009ae0 con 0x7f5c3c06b7c0 2023-10-06T03:30:12.073 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:12.065+0000 7f5c2e7fc700 1 -- 172.21.15.110:0/515934378 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c3800cd50 con 0x7f5c3c06b7c0 2023-10-06T03:30:12.073 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:12.065+0000 7f5c44675700 1 -- 172.21.15.110:0/515934378 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c3c0c18a0 con 0x7f5c3c06b7c0 2023-10-06T03:30:12.074 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:12.069+0000 7f5c2e7fc700 1 -- 172.21.15.110:0/515934378 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7f5c38013680 con 0x7f5c3c06b7c0 2023-10-06T03:30:12.074 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:12.069+0000 7f5c44675700 1 -- 172.21.15.110:0/515934378 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c3c06ff60 con 0x7f5c3c06b7c0 2023-10-06T03:30:12.074 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:12.069+0000 7f5c2e7fc700 1 --2- 172.21.15.110:0/515934378 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f5c20060580 0x7f5c20062a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:12.074 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:12.069+0000 7f5c2e7fc700 1 -- 172.21.15.110:0/515934378 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6809+0+0 (secure 0 0 0) 0x7f5c3807ce70 con 0x7f5c3c06b7c0 2023-10-06T03:30:12.074 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:12.069+0000 7f5c41c10700 1 --2- 172.21.15.110:0/515934378 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f5c20060580 0x7f5c20062a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:12.075 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:12.069+0000 7f5c41c10700 1 --2- 172.21.15.110:0/515934378 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f5c20060580 0x7f5c20062a30 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f5c3c16c650 tx=0x7f5c300091a0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:30:12.081 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:12.077+0000 7f5c2e7fc700 1 -- 172.21.15.110:0/515934378 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f5c3804d010 con 0x7f5c3c06b7c0 2023-10-06T03:30:12.241 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:12.237+0000 7f5c44675700 1 -- 172.21.15.110:0/515934378 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f5c3c06cc00 con 0x7f5c3c06b7c0 2023-10-06T03:30:12.242 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:12.237+0000 7f5c2e7fc700 1 -- 172.21.15.110:0/515934378 <== mon.0 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v28) v1 ==== 100+0+11 (secure 0 0 0) 0x7f5c38050630 con 0x7f5c3c06b7c0 2023-10-06T03:30:12.242 INFO:teuthology.orchestra.run.smithi110.stdout:pg_num: 64 2023-10-06T03:30:12.244 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:12.241+0000 7f5c44675700 1 -- 172.21.15.110:0/515934378 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f5c20060580 msgr2=0x7f5c20062a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:12.244 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:12.241+0000 7f5c44675700 1 --2- 172.21.15.110:0/515934378 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f5c20060580 0x7f5c20062a30 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f5c3c16c650 tx=0x7f5c300091a0 comp rx=0 tx=0).stop 2023-10-06T03:30:12.244 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:12.241+0000 7f5c44675700 1 -- 172.21.15.110:0/515934378 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5c3c06b7c0 msgr2=0x7f5c3c0b1c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:12.244 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:12.241+0000 7f5c44675700 1 --2- 172.21.15.110:0/515934378 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5c3c06b7c0 0x7f5c3c0b1c20 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f5c3800ec80 tx=0x7f5c380091a0 comp rx=0 tx=0).stop 2023-10-06T03:30:12.245 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:12.241+0000 7f5c44675700 1 -- 172.21.15.110:0/515934378 shutdown_connections 2023-10-06T03:30:12.245 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:12.241+0000 7f5c44675700 1 --2- 172.21.15.110:0/515934378 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f5c20060580 0x7f5c20062a30 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:12.245 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:12.241+0000 7f5c44675700 1 --2- 172.21.15.110:0/515934378 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5c3c157110 0x7f5c3c0670a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:12.245 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:12.241+0000 7f5c44675700 1 --2- 172.21.15.110:0/515934378 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5c3c156d40 0x7f5c3c156440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:12.245 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:12.241+0000 7f5c44675700 1 --2- 172.21.15.110:0/515934378 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5c3c06b7c0 0x7f5c3c0b1c20 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:12.245 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:12.241+0000 7f5c44675700 1 -- 172.21.15.110:0/515934378 >> 172.21.15.110:0/515934378 conn(0x7f5c3c0ccd30 msgr2=0x7f5c3c15c340 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:30:12.245 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:12.241+0000 7f5c44675700 1 -- 172.21.15.110:0/515934378 shutdown_connections 2023-10-06T03:30:12.246 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:12.241+0000 7f5c44675700 1 -- 172.21.15.110:0/515934378 wait complete. 2023-10-06T03:30:12.257 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2023-10-06T03:30:12.371 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:12.365+0000 7f5c4a6c7700 1 Processor -- start 2023-10-06T03:30:12.371 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:12.365+0000 7f5c4a6c7700 1 -- start start 2023-10-06T03:30:12.373 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:12.369+0000 7f5c4a6c7700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f5c440aee70 0x7f5c4415a500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:12.373 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:12.369+0000 7f5c4a6c7700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c440b0fa0 con 0x7f5c441591f0 2023-10-06T03:30:12.373 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:12.369+0000 7f5c4a6c7700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c440ca420 con 0x7f5c440af240 2023-10-06T03:30:12.373 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:12.369+0000 7f5c4a6c7700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5c440ce660 con 0x7f5c440aee70 2023-10-06T03:30:12.373 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:12.369+0000 7f5c48c64700 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f5c441591f0 0x7f5c440b1430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:53234/0 (socket says 172.21.15.110:53234) 2023-10-06T03:30:12.373 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:12.369+0000 7f5c48c64700 1 -- 172.21.15.110:0/2192630883 learned_addr learned my addr 172.21.15.110:0/2192630883 (peer_addr_for_me v1:172.21.15.110:0/0) 2023-10-06T03:30:12.373 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:12.369+0000 7f5c43fff700 1 --2- 172.21.15.110:0/2192630883 >> v2:172.21.15.192:3300/0 conn(0x7f5c440aee70 0x7f5c4415a500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:12.375 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:12.369+0000 7f5c437fe700 1 -- 172.21.15.110:0/2192630883 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2664347189 0 0) 0x7f5c440b0fa0 con 0x7f5c441591f0 2023-10-06T03:30:12.376 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:12.369+0000 7f5c437fe700 1 -- 172.21.15.110:0/2192630883 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5c24003570 con 0x7f5c441591f0 2023-10-06T03:30:12.376 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:12.369+0000 7f5c437fe700 1 -- 172.21.15.110:0/2192630883 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3185237462 0 0) 0x7f5c440ca420 con 0x7f5c440af240 2023-10-06T03:30:12.376 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:12.369+0000 7f5c437fe700 1 -- 172.21.15.110:0/2192630883 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5c440b0fa0 con 0x7f5c440af240 2023-10-06T03:30:12.376 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:12.369+0000 7f5c43fff700 1 -- 172.21.15.110:0/2192630883 >> v1:172.21.15.192:6789/0 conn(0x7f5c440af240 legacy=0x7f5c44158ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:12.376 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:12.369+0000 7f5c437fe700 1 -- 172.21.15.110:0/2192630883 <== mon.0 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1981220096 0 0) 0x7f5c24003570 con 0x7f5c441591f0 2023-10-06T03:30:12.376 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:12.369+0000 7f5c43fff700 1 -- 172.21.15.110:0/2192630883 >> v1:172.21.15.110:6789/0 conn(0x7f5c441591f0 legacy=0x7f5c440b1430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:12.376 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:12.369+0000 7f5c43fff700 1 -- 172.21.15.110:0/2192630883 --> v2:172.21.15.192:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c440ce940 con 0x7f5c440aee70 2023-10-06T03:30:12.376 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:12.369+0000 7f5c43fff700 1 --2- 172.21.15.110:0/2192630883 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5c440aee70 0x7f5c4415a500 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f5c340098e0 tx=0x7f5c340091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fd664b1498041b5 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:30:12.377 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:12.369+0000 7f5c437fe700 1 -- 172.21.15.110:0/2192630883 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c3c0017f0 con 0x7f5c440aee70 2023-10-06T03:30:12.377 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:12.369+0000 7f5c437fe700 1 -- 172.21.15.110:0/2192630883 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c34011420 con 0x7f5c440aee70 2023-10-06T03:30:12.377 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:12.369+0000 7f5c437fe700 1 -- 172.21.15.110:0/2192630883 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c340117c0 con 0x7f5c440aee70 2023-10-06T03:30:12.377 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:12.373+0000 7f5c4a6c7700 1 -- 172.21.15.110:0/2192630883 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5c440aee70 msgr2=0x7f5c4415a500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:12.377 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:12.373+0000 7f5c4a6c7700 1 --2- 172.21.15.110:0/2192630883 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5c440aee70 0x7f5c4415a500 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f5c340098e0 tx=0x7f5c340091a0 comp rx=0 tx=0).stop 2023-10-06T03:30:12.377 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:12.373+0000 7f5c4a6c7700 1 -- 172.21.15.110:0/2192630883 shutdown_connections 2023-10-06T03:30:12.377 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:12.373+0000 7f5c4a6c7700 1 --2- 172.21.15.110:0/2192630883 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5c440aee70 0x7f5c4415a500 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:12.377 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:12.373+0000 7f5c4a6c7700 1 -- 172.21.15.110:0/2192630883 >> 172.21.15.110:0/2192630883 conn(0x7f5c4406bfa0 msgr2=0x7f5c440cdeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:30:12.377 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:12.373+0000 7f5c4a6c7700 1 -- 172.21.15.110:0/2192630883 shutdown_connections 2023-10-06T03:30:12.378 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:12.373+0000 7f5c4a6c7700 1 -- 172.21.15.110:0/2192630883 wait complete. 2023-10-06T03:30:12.380 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:12.373+0000 7f5c4a6c7700 1 Processor -- start 2023-10-06T03:30:12.381 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:12.373+0000 7f5c4a6c7700 1 -- start start 2023-10-06T03:30:12.381 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:12.373+0000 7f5c4a6c7700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5c440aee70 0x7f5c4414c110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:12.381 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:12.373+0000 7f5c4a6c7700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5c440af240 0x7f5c44150800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:12.381 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:12.373+0000 7f5c4a6c7700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5c441591f0 0x7f5c4415c580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:12.381 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:12.373+0000 7f5c4a6c7700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c4414c650 con 0x7f5c441591f0 2023-10-06T03:30:12.381 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:12.373+0000 7f5c4a6c7700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c441526d0 con 0x7f5c440aee70 2023-10-06T03:30:12.381 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:12.373+0000 7f5c4a6c7700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c441529b0 con 0x7f5c440af240 2023-10-06T03:30:12.381 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:12.373+0000 7f5c48c64700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5c441591f0 0x7f5c4415c580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:12.382 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:12.373+0000 7f5c48c64700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5c441591f0 0x7f5c4415c580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:52076/0 (socket says 172.21.15.110:52076) 2023-10-06T03:30:12.382 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:12.373+0000 7f5c48c64700 1 -- 172.21.15.110:0/532451352 learned_addr learned my addr 172.21.15.110:0/532451352 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:30:12.382 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:12.373+0000 7f5c43fff700 1 --2- 172.21.15.110:0/532451352 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5c440aee70 0x7f5c4414c110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:12.382 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:12.373+0000 7f5c3bfff700 1 --2- 172.21.15.110:0/532451352 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5c440af240 0x7f5c44150800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:12.382 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:12.373+0000 7f5c48c64700 1 -- 172.21.15.110:0/532451352 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5c440af240 msgr2=0x7f5c44150800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:12.382 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:12.373+0000 7f5c48c64700 1 --2- 172.21.15.110:0/532451352 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5c440af240 0x7f5c44150800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:12.382 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:12.373+0000 7f5c48c64700 1 -- 172.21.15.110:0/532451352 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5c440aee70 msgr2=0x7f5c4414c110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:12.382 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:12.373+0000 7f5c48c64700 1 --2- 172.21.15.110:0/532451352 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5c440aee70 0x7f5c4414c110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:12.383 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:12.373+0000 7f5c48c64700 1 -- 172.21.15.110:0/532451352 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c34009420 con 0x7f5c441591f0 2023-10-06T03:30:12.383 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:12.373+0000 7f5c48c64700 1 --2- 172.21.15.110:0/532451352 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5c441591f0 0x7f5c4415c580 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f5c3c00cbd0 tx=0x7f5c3c007d20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:30:12.383 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:12.373+0000 7f5c40ff9700 1 -- 172.21.15.110:0/532451352 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c3c004dc0 con 0x7f5c441591f0 2023-10-06T03:30:12.383 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:12.373+0000 7f5c40ff9700 1 -- 172.21.15.110:0/532451352 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c3c00a6b0 con 0x7f5c441591f0 2023-10-06T03:30:12.383 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:12.373+0000 7f5c4a6c7700 1 -- 172.21.15.110:0/532451352 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c44152c90 con 0x7f5c441591f0 2023-10-06T03:30:12.383 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:12.373+0000 7f5c40ff9700 1 -- 172.21.15.110:0/532451352 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c3c00aa70 con 0x7f5c441591f0 2023-10-06T03:30:12.383 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:12.373+0000 7f5c4a6c7700 1 -- 172.21.15.110:0/532451352 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c44150d40 con 0x7f5c441591f0 2023-10-06T03:30:12.383 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:12.377+0000 7f5c4a6c7700 1 -- 172.21.15.110:0/532451352 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c440aeb30 con 0x7f5c441591f0 2023-10-06T03:30:12.383 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:12.377+0000 7f5c40ff9700 1 -- 172.21.15.110:0/532451352 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7f5c3c007150 con 0x7f5c441591f0 2023-10-06T03:30:12.384 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:12.377+0000 7f5c40ff9700 1 --2- 172.21.15.110:0/532451352 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f5c28060550 0x7f5c28062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:12.384 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:12.377+0000 7f5c43fff700 1 --2- 172.21.15.110:0/532451352 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f5c28060550 0x7f5c28062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:12.384 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:12.377+0000 7f5c40ff9700 1 -- 172.21.15.110:0/532451352 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6809+0+0 (secure 0 0 0) 0x7f5c3c07c5f0 con 0x7f5c441591f0 2023-10-06T03:30:12.385 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:12.377+0000 7f5c43fff700 1 --2- 172.21.15.110:0/532451352 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f5c28060550 0x7f5c28062a00 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f5c340093f0 tx=0x7f5c34016000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:30:12.386 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:12.381+0000 7f5c40ff9700 1 -- 172.21.15.110:0/532451352 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f5c440aeb30 con 0x7f5c441591f0 2023-10-06T03:30:12.551 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:12.545+0000 7f5c4a6c7700 1 -- 172.21.15.110:0/532451352 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f5c440aeb30 con 0x7f5c441591f0 2023-10-06T03:30:12.551 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:12.549+0000 7f5c40ff9700 1 -- 172.21.15.110:0/532451352 <== mon.0 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v28) v1 ==== 103+0+11 (secure 0 0 0) 0x7f5c440aeb30 con 0x7f5c441591f0 2023-10-06T03:30:12.551 INFO:teuthology.orchestra.run.smithi110.stdout:pg_num: 32 2023-10-06T03:30:12.554 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:12.549+0000 7f5c4a6c7700 1 -- 172.21.15.110:0/532451352 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f5c28060550 msgr2=0x7f5c28062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:12.554 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:12.549+0000 7f5c4a6c7700 1 --2- 172.21.15.110:0/532451352 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f5c28060550 0x7f5c28062a00 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f5c340093f0 tx=0x7f5c34016000 comp rx=0 tx=0).stop 2023-10-06T03:30:12.554 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:12.549+0000 7f5c4a6c7700 1 -- 172.21.15.110:0/532451352 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5c441591f0 msgr2=0x7f5c4415c580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:12.554 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:12.549+0000 7f5c4a6c7700 1 --2- 172.21.15.110:0/532451352 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5c441591f0 0x7f5c4415c580 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f5c3c00cbd0 tx=0x7f5c3c007d20 comp rx=0 tx=0).stop 2023-10-06T03:30:12.554 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:12.549+0000 7f5c4a6c7700 1 -- 172.21.15.110:0/532451352 shutdown_connections 2023-10-06T03:30:12.555 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:12.549+0000 7f5c4a6c7700 1 --2- 172.21.15.110:0/532451352 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f5c28060550 0x7f5c28062a00 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:12.555 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:12.549+0000 7f5c4a6c7700 1 --2- 172.21.15.110:0/532451352 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5c441591f0 0x7f5c4415c580 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:12.555 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:12.549+0000 7f5c4a6c7700 1 --2- 172.21.15.110:0/532451352 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5c440af240 0x7f5c44150800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:12.555 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:12.549+0000 7f5c4a6c7700 1 --2- 172.21.15.110:0/532451352 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5c440aee70 0x7f5c4414c110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:12.555 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:12.549+0000 7f5c4a6c7700 1 -- 172.21.15.110:0/532451352 >> 172.21.15.110:0/532451352 conn(0x7f5c4406bfa0 msgr2=0x7f5c440b1430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:30:12.555 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:12.549+0000 7f5c4a6c7700 1 -- 172.21.15.110:0/532451352 shutdown_connections 2023-10-06T03:30:12.555 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:12.549+0000 7f5c4a6c7700 1 -- 172.21.15.110:0/532451352 wait complete. 2023-10-06T03:30:12.566 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-06T03:30:12.672 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:12.669+0000 7f1c8a536700 1 Processor -- start 2023-10-06T03:30:12.672 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:12.669+0000 7f1c8a536700 1 -- start start 2023-10-06T03:30:12.673 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:12.669+0000 7f1c8a536700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f1c840b23e0 0x7f1c840aff50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:12.673 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:12.669+0000 7f1c8a536700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f1c84158a50 0x7f1c84158ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:12.673 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:12.669+0000 7f1c8a536700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c840cac30 con 0x7f1c840afb80 2023-10-06T03:30:12.673 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:12.669+0000 7f1c8a536700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1c840caf10 con 0x7f1c84158a50 2023-10-06T03:30:12.673 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:12.669+0000 7f1c8a536700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1c840cb1f0 con 0x7f1c840b23e0 2023-10-06T03:30:12.673 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:12.669+0000 7f1c89d35700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f1c84158a50 0x7f1c84158ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:12.674 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:12.669+0000 7f1c88d33700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f1c840b23e0 0x7f1c840aff50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:12.674 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:12.669+0000 7f1c89d35700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f1c84158a50 0x7f1c84158ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:52090/0 (socket says 172.21.15.110:52090) 2023-10-06T03:30:12.674 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:12.669+0000 7f1c89d35700 1 -- 172.21.15.110:0/1091097826 learned_addr learned my addr 172.21.15.110:0/1091097826 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:30:12.675 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:12.669+0000 7f1c88d33700 1 -- 172.21.15.110:0/1091097826 >> v2:172.21.15.110:3300/0 conn(0x7f1c84158a50 msgr2=0x7f1c84158ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:30:12.675 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:12.669+0000 7f1c88d33700 1 --2- 172.21.15.110:0/1091097826 >> v2:172.21.15.110:3300/0 conn(0x7f1c84158a50 0x7f1c84158ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:12.675 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:12.669+0000 7f1c88d33700 1 -- 172.21.15.110:0/1091097826 >> v1:172.21.15.192:6789/0 conn(0x7f1c840afb80 legacy=0x7f1c840b1cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:12.675 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:12.669+0000 7f1c88d33700 1 -- 172.21.15.110:0/1091097826 --> v2:172.21.15.192:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c840cb4d0 con 0x7f1c840b23e0 2023-10-06T03:30:12.676 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:12.669+0000 7f1c89d35700 1 --2- 172.21.15.110:0/1091097826 >> v2:172.21.15.110:3300/0 conn(0x7f1c84158a50 0x7f1c84158ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-06T03:30:12.676 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:12.669+0000 7f1c88d33700 1 --2- 172.21.15.110:0/1091097826 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1c840b23e0 0x7f1c840aff50 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f1c840b04c0 tx=0x7f1c7c009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5d8c5d715e657ddd server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:30:12.676 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:12.669+0000 7f1c7bfff700 1 -- 172.21.15.110:0/1091097826 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c7c00e070 con 0x7f1c840b23e0 2023-10-06T03:30:12.676 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:12.669+0000 7f1c7bfff700 1 -- 172.21.15.110:0/1091097826 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c7c00bb60 con 0x7f1c840b23e0 2023-10-06T03:30:12.676 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:12.669+0000 7f1c7bfff700 1 -- 172.21.15.110:0/1091097826 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c7c012430 con 0x7f1c840b23e0 2023-10-06T03:30:12.676 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:12.669+0000 7f1c8a536700 1 -- 172.21.15.110:0/1091097826 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1c840b23e0 msgr2=0x7f1c840aff50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:12.677 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:12.669+0000 7f1c8a536700 1 --2- 172.21.15.110:0/1091097826 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1c840b23e0 0x7f1c840aff50 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f1c840b04c0 tx=0x7f1c7c009780 comp rx=0 tx=0).stop 2023-10-06T03:30:12.677 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:12.669+0000 7f1c8a536700 1 -- 172.21.15.110:0/1091097826 shutdown_connections 2023-10-06T03:30:12.677 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:12.669+0000 7f1c8a536700 1 --2- 172.21.15.110:0/1091097826 >> v2:172.21.15.110:3300/0 conn(0x7f1c84158a50 0x7f1c84158ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:12.677 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:12.669+0000 7f1c8a536700 1 --2- 172.21.15.110:0/1091097826 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1c840b23e0 0x7f1c840aff50 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:12.677 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:12.669+0000 7f1c8a536700 1 -- 172.21.15.110:0/1091097826 >> 172.21.15.110:0/1091097826 conn(0x7f1c8406dc70 msgr2=0x7f1c8406e070 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:30:12.677 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:12.669+0000 7f1c8a536700 1 -- 172.21.15.110:0/1091097826 shutdown_connections 2023-10-06T03:30:12.677 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:12.669+0000 7f1c8a536700 1 -- 172.21.15.110:0/1091097826 wait complete. 2023-10-06T03:30:12.680 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:12.673+0000 7f1c8a536700 1 Processor -- start 2023-10-06T03:30:12.680 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:12.673+0000 7f1c8a536700 1 -- start start 2023-10-06T03:30:12.680 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:12.673+0000 7f1c8a536700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1c840afb80 0x7f1c84156680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:12.680 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:12.673+0000 7f1c8a536700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1c840b23e0 0x7f1c840c6e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:12.680 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:12.673+0000 7f1c8a536700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1c84158a50 0x7f1c840cbbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:12.680 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:12.673+0000 7f1c8a536700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c840c3860 con 0x7f1c840afb80 2023-10-06T03:30:12.681 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:12.673+0000 7f1c8a536700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c840c3ab0 con 0x7f1c84158a50 2023-10-06T03:30:12.681 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:12.673+0000 7f1c8a536700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c840c3d90 con 0x7f1c840b23e0 2023-10-06T03:30:12.681 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:12.673+0000 7f1c89534700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1c840afb80 0x7f1c84156680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:12.681 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:12.673+0000 7f1c89534700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1c840afb80 0x7f1c84156680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:52096/0 (socket says 172.21.15.110:52096) 2023-10-06T03:30:12.681 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:12.673+0000 7f1c89534700 1 -- 172.21.15.110:0/3972457469 learned_addr learned my addr 172.21.15.110:0/3972457469 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:30:12.681 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:12.673+0000 7f1c88d33700 1 --2- 172.21.15.110:0/3972457469 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1c840b23e0 0x7f1c840c6e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:12.681 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:12.673+0000 7f1c89d35700 1 --2- 172.21.15.110:0/3972457469 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1c84158a50 0x7f1c840cbbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:12.681 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:12.673+0000 7f1c89534700 1 -- 172.21.15.110:0/3972457469 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1c840b23e0 msgr2=0x7f1c840c6e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:12.681 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:12.673+0000 7f1c89534700 1 --2- 172.21.15.110:0/3972457469 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1c840b23e0 0x7f1c840c6e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:12.682 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:12.673+0000 7f1c89534700 1 -- 172.21.15.110:0/3972457469 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1c84158a50 msgr2=0x7f1c840cbbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:12.682 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:12.673+0000 7f1c89534700 1 --2- 172.21.15.110:0/3972457469 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1c84158a50 0x7f1c840cbbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:12.682 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:12.673+0000 7f1c89534700 1 -- 172.21.15.110:0/3972457469 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c800094f0 con 0x7f1c840afb80 2023-10-06T03:30:12.682 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:12.673+0000 7f1c88d33700 1 --2- 172.21.15.110:0/3972457469 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1c840b23e0 0x7f1c840c6e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-06T03:30:12.682 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:12.673+0000 7f1c89d35700 1 --2- 172.21.15.110:0/3972457469 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1c84158a50 0x7f1c840cbbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-06T03:30:12.682 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:12.677+0000 7f1c89534700 1 --2- 172.21.15.110:0/3972457469 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1c840afb80 0x7f1c84156680 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f1c7400c4e0 tx=0x7f1c7400c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:30:12.682 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:12.677+0000 7f1c797fa700 1 -- 172.21.15.110:0/3972457469 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c74011590 con 0x7f1c840afb80 2023-10-06T03:30:12.682 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:12.677+0000 7f1c797fa700 1 -- 172.21.15.110:0/3972457469 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c74011710 con 0x7f1c840afb80 2023-10-06T03:30:12.682 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:12.677+0000 7f1c8a536700 1 -- 172.21.15.110:0/3972457469 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c7c009a20 con 0x7f1c840afb80 2023-10-06T03:30:12.682 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:12.677+0000 7f1c797fa700 1 -- 172.21.15.110:0/3972457469 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c74011ad0 con 0x7f1c840afb80 2023-10-06T03:30:12.683 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:12.677+0000 7f1c8a536700 1 -- 172.21.15.110:0/3972457469 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c840c4350 con 0x7f1c840afb80 2023-10-06T03:30:12.683 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:12.677+0000 7f1c8a536700 1 -- 172.21.15.110:0/3972457469 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c840c4540 con 0x7f1c840afb80 2023-10-06T03:30:12.683 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:12.677+0000 7f1c797fa700 1 -- 172.21.15.110:0/3972457469 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7f1c7401b3b0 con 0x7f1c840afb80 2023-10-06T03:30:12.683 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:12.677+0000 7f1c797fa700 1 --2- 172.21.15.110:0/3972457469 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f1c68060550 0x7f1c68062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:12.684 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:12.681+0000 7f1c88d33700 1 --2- 172.21.15.110:0/3972457469 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f1c68060550 0x7f1c68062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:12.684 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:12.681+0000 7f1c797fa700 1 -- 172.21.15.110:0/3972457469 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6809+0+0 (secure 0 0 0) 0x7f1c7407c200 con 0x7f1c840afb80 2023-10-06T03:30:12.684 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:12.681+0000 7f1c88d33700 1 --2- 172.21.15.110:0/3972457469 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f1c68060550 0x7f1c68062a00 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f1c7c0097c0 tx=0x7f1c7c019000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:30:12.688 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:12.685+0000 7f1c797fa700 1 -- 172.21.15.110:0/3972457469 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f1c7404c300 con 0x7f1c840afb80 2023-10-06T03:30:12.849 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:12.845+0000 7f1c8a536700 1 -- 172.21.15.110:0/3972457469 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1c84070ff0 con 0x7f1c840afb80 2023-10-06T03:30:12.849 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:12.845+0000 7f1c797fa700 1 -- 172.21.15.110:0/3972457469 <== mon.0 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v28) v1 ==== 74+0+18755 (secure 0 0 0) 0x7f1c7404f920 con 0x7f1c840afb80 2023-10-06T03:30:12.850 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-10-06T03:30:12.850 INFO:teuthology.orchestra.run.smithi110.stdout:{"epoch":28,"fsid":"93b8f6ac-4c92-4b45-b8e4-b59617024aa2","created":"2023-10-06T03:29:10.519583+0000","modified":"2023-10-06T03:29:38.951740+0000","last_up_change":"2023-10-06T03:29:21.960511+0000","last_in_change":"2023-10-06T03:29:12.560436+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-06T03:29:20.703743+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2023-10-06T03:29:28.307281+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2023-10-06T03:29:32.247251+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data","create_time":"2023-10-06T03:29:32.880086+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}}},{"pool":5,"pool_name":"cephfs_data_ec","create_time":"2023-10-06T03:29:36.613779+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"28","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"5866eb6c-2adc-46e4-be7a-e5bcda0f4065","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6824","nonce":33394},{"type":"v1","addr":"172.21.15.110:6825","nonce":33394}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6826","nonce":33394},{"type":"v1","addr":"172.21.15.110:6827","nonce":33394}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6830","nonce":33394},{"type":"v1","addr":"172.21.15.110:6831","nonce":33394}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6828","nonce":33394},{"type":"v1","addr":"172.21.15.110:6829","nonce":33394}]},"public_addr":"172.21.15.110:6825/33394","cluster_addr":"172.21.15.110:6827/33394","heartbeat_back_addr":"172.21.15.110:6831/33394","heartbeat_front_addr":"172.21.15.110:6829/33394","state":["exists","up"]},{"osd":1,"uuid":"a38de118-062b-4a05-9782-2ceddf6ff4df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6800","nonce":33396},{"type":"v1","addr":"172.21.15.110:6801","nonce":33396}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6802","nonce":33396},{"type":"v1","addr":"172.21.15.110:6803","nonce":33396}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6806","nonce":33396},{"type":"v1","addr":"172.21.15.110:6807","nonce":33396}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6804","nonce":33396},{"type":"v1","addr":"172.21.15.110:6805","nonce":33396}]},"public_addr":"172.21.15.110:6801/33396","cluster_addr":"172.21.15.110:6803/33396","heartbeat_back_addr":"172.21.15.110:6807/33396","heartbeat_front_addr":"172.21.15.110:6805/33396","state":["exists","up"]},{"osd":2,"uuid":"56e39be2-3407-4b69-877e-cf16617b06d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6816","nonce":33397},{"type":"v1","addr":"172.21.15.110:6817","nonce":33397}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6818","nonce":33397},{"type":"v1","addr":"172.21.15.110:6819","nonce":33397}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6822","nonce":33397},{"type":"v1","addr":"172.21.15.110:6823","nonce":33397}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6820","nonce":33397},{"type":"v1","addr":"172.21.15.110:6821","nonce":33397}]},"public_addr":"172.21.15.110:6817/33397","cluster_addr":"172.21.15.110:6819/33397","heartbeat_back_addr":"172.21.15.110:6823/33397","heartbeat_front_addr":"172.21.15.110:6821/33397","state":["exists","up"]},{"osd":3,"uuid":"01c82d13-c1d4-4bde-b55e-3e0089b40115","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6808","nonce":33395},{"type":"v1","addr":"172.21.15.110:6809","nonce":33395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6810","nonce":33395},{"type":"v1","addr":"172.21.15.110:6811","nonce":33395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6814","nonce":33395},{"type":"v1","addr":"172.21.15.110:6815","nonce":33395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6812","nonce":33395},{"type":"v1","addr":"172.21.15.110:6813","nonce":33395}]},"public_addr":"172.21.15.110:6809/33395","cluster_addr":"172.21.15.110:6811/33395","heartbeat_back_addr":"172.21.15.110:6815/33395","heartbeat_front_addr":"172.21.15.110:6813/33395","state":["exists","up"]},{"osd":4,"uuid":"86983430-5fa3-4d17-998f-03e2cd843237","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":33683},{"type":"v1","addr":"172.21.15.192:6817","nonce":33683}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":33683},{"type":"v1","addr":"172.21.15.192:6819","nonce":33683}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":33683},{"type":"v1","addr":"172.21.15.192:6823","nonce":33683}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6820","nonce":33683},{"type":"v1","addr":"172.21.15.192:6821","nonce":33683}]},"public_addr":"172.21.15.192:6817/33683","cluster_addr":"172.21.15.192:6819/33683","heartbeat_back_addr":"172.21.15.192:6823/33683","heartbeat_front_addr":"172.21.15.192:6821/33683","state":["exists","up"]},{"osd":5,"uuid":"6945257f-821d-4619-a63f-ebe857eba6f4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":33684},{"type":"v1","addr":"172.21.15.192:6801","nonce":33684}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":33684},{"type":"v1","addr":"172.21.15.192:6803","nonce":33684}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":33684},{"type":"v1","addr":"172.21.15.192:6807","nonce":33684}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6804","nonce":33684},{"type":"v1","addr":"172.21.15.192:6805","nonce":33684}]},"public_addr":"172.21.15.192:6801/33684","cluster_addr":"172.21.15.192:6803/33684","heartbeat_back_addr":"172.21.15.192:6807/33684","heartbeat_front_addr":"172.21.15.192:6805/33684","state":["exists","up"]},{"osd":6,"uuid":"7854d1ae-7060-4780-8f97-73cea5e7a12f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6824","nonce":33686},{"type":"v1","addr":"172.21.15.192:6825","nonce":33686}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":33686},{"type":"v1","addr":"172.21.15.192:6827","nonce":33686}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":33686},{"type":"v1","addr":"172.21.15.192:6831","nonce":33686}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6828","nonce":33686},{"type":"v1","addr":"172.21.15.192:6829","nonce":33686}]},"public_addr":"172.21.15.192:6825/33686","cluster_addr":"172.21.15.192:6827/33686","heartbeat_back_addr":"172.21.15.192:6831/33686","heartbeat_front_addr":"172.21.15.192:6829/33686","state":["exists","up"]},{"osd":7,"uuid":"f1ebdfd5-b859-482b-9f97-c18c4a26f445","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6808","nonce":33685},{"type":"v1","addr":"172.21.15.192:6809","nonce":33685}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":33685},{"type":"v1","addr":"172.21.15.192:6811","nonce":33685}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":33685},{"type":"v1","addr":"172.21.15.192:6815","nonce":33685}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6812","nonce":33685},{"type":"v1","addr":"172.21.15.192:6813","nonce":33685}]},"public_addr":"172.21.15.192:6809/33685","cluster_addr":"172.21.15.192:6811/33685","heartbeat_back_addr":"172.21.15.192:6815/33685","heartbeat_front_addr":"172.21.15.192:6813/33685","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T03:29:18.002048+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T03:29:17.478576+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T03:29:18.034944+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T03:29:17.628513+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T03:29:18.047338+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T03:29:17.772349+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T03:29:17.883958+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T03:29:18.151838+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-06T03:30:12.852 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:12.849+0000 7f1c8a536700 1 -- 172.21.15.110:0/3972457469 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f1c68060550 msgr2=0x7f1c68062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:12.853 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:12.849+0000 7f1c8a536700 1 --2- 172.21.15.110:0/3972457469 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f1c68060550 0x7f1c68062a00 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f1c7c0097c0 tx=0x7f1c7c019000 comp rx=0 tx=0).stop 2023-10-06T03:30:12.853 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:12.849+0000 7f1c8a536700 1 -- 172.21.15.110:0/3972457469 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1c840afb80 msgr2=0x7f1c84156680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:12.853 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:12.849+0000 7f1c8a536700 1 --2- 172.21.15.110:0/3972457469 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1c840afb80 0x7f1c84156680 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f1c7400c4e0 tx=0x7f1c7400c8a0 comp rx=0 tx=0).stop 2023-10-06T03:30:12.853 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:12.849+0000 7f1c8a536700 1 -- 172.21.15.110:0/3972457469 shutdown_connections 2023-10-06T03:30:12.853 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:12.849+0000 7f1c8a536700 1 --2- 172.21.15.110:0/3972457469 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f1c68060550 0x7f1c68062a00 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:12.853 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:12.849+0000 7f1c8a536700 1 --2- 172.21.15.110:0/3972457469 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1c84158a50 0x7f1c840cbbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:12.853 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:12.849+0000 7f1c8a536700 1 --2- 172.21.15.110:0/3972457469 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1c840b23e0 0x7f1c840c6e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:12.853 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:12.849+0000 7f1c8a536700 1 --2- 172.21.15.110:0/3972457469 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1c840afb80 0x7f1c84156680 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:12.853 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:12.849+0000 7f1c8a536700 1 -- 172.21.15.110:0/3972457469 >> 172.21.15.110:0/3972457469 conn(0x7f1c8406dc70 msgr2=0x7f1c84157590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:30:12.854 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:12.849+0000 7f1c8a536700 1 -- 172.21.15.110:0/3972457469 shutdown_connections 2023-10-06T03:30:12.854 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:12.849+0000 7f1c8a536700 1 -- 172.21.15.110:0/3972457469 wait complete. 2023-10-06T03:30:12.864 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-06T03:29:20.703743+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-10-06T03:29:28.307281+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 3, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-06T03:29:32.247251+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 4, 'pool_name': 'cephfs_data', 'create_time': '2023-10-06T03:29:32.880086+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}, {'pool': 5, 'pool_name': 'cephfs_data_ec', 'create_time': '2023-10-06T03:29:36.613779+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '28', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-10-06T03:30:12.866 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-06T03:30:12.976 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:12.973+0000 7f160050b700 1 Processor -- start 2023-10-06T03:30:12.976 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:12.973+0000 7f160050b700 1 -- start start 2023-10-06T03:30:12.978 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:12.973+0000 7f160050b700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f15f80b1d60 0x7f15f8156e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:12.978 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:12.973+0000 7f160050b700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15f80c6af0 con 0x7f15f80cbfb0 2023-10-06T03:30:12.978 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:12.973+0000 7f160050b700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15f806bd40 con 0x7f15f80c8880 2023-10-06T03:30:12.978 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:12.973+0000 7f160050b700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15f80c7300 con 0x7f15f80b1d60 2023-10-06T03:30:12.978 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:12.973+0000 7f15feaa8700 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f15f80cbfb0 0x7f15f806c1d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:53248/0 (socket says 172.21.15.110:53248) 2023-10-06T03:30:12.978 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:12.973+0000 7f15feaa8700 1 -- 172.21.15.110:0/367460861 learned_addr learned my addr 172.21.15.110:0/367460861 (peer_addr_for_me v1:172.21.15.110:0/0) 2023-10-06T03:30:12.978 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:12.973+0000 7f15fe2a7700 1 --2- 172.21.15.110:0/367460861 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f15f80b1d60 0x7f15f8156e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:12.980 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:12.973+0000 7f15fd2a5700 1 -- 172.21.15.110:0/367460861 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2246094319 0 0) 0x7f15f80c6af0 con 0x7f15f80cbfb0 2023-10-06T03:30:12.980 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:12.973+0000 7f15fd2a5700 1 -- 172.21.15.110:0/367460861 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f15e4003570 con 0x7f15f80cbfb0 2023-10-06T03:30:12.980 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:12.973+0000 7f15fd2a5700 1 -- 172.21.15.110:0/367460861 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 444900011 0 0) 0x7f15f806bd40 con 0x7f15f80c8880 2023-10-06T03:30:12.980 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:12.973+0000 7f15fd2a5700 1 -- 172.21.15.110:0/367460861 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f15f80c6af0 con 0x7f15f80c8880 2023-10-06T03:30:12.980 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:12.973+0000 7f15fe2a7700 1 -- 172.21.15.110:0/367460861 >> v1:172.21.15.192:6789/0 conn(0x7f15f80c8880 legacy=0x7f15f80cb8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:12.981 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:12.973+0000 7f15fe2a7700 1 -- 172.21.15.110:0/367460861 >> v1:172.21.15.110:6789/0 conn(0x7f15f80cbfb0 legacy=0x7f15f806c1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:12.981 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:12.973+0000 7f15fe2a7700 1 -- 172.21.15.110:0/367460861 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15f8157340 con 0x7f15f80b1d60 2023-10-06T03:30:12.981 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:12.977+0000 7f15fe2a7700 1 --2- 172.21.15.110:0/367460861 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f15f80b1d60 0x7f15f8156e00 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f15f4009910 tx=0x7f15f40091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5157c57ced6f672b server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:30:12.981 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:12.977+0000 7f15fd2a5700 1 -- 172.21.15.110:0/367460861 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15f4011630 con 0x7f15f80b1d60 2023-10-06T03:30:12.981 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:12.977+0000 7f15fd2a5700 1 -- 172.21.15.110:0/367460861 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f15f4011790 con 0x7f15f80b1d60 2023-10-06T03:30:12.981 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:12.977+0000 7f15fd2a5700 1 -- 172.21.15.110:0/367460861 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15f4011b10 con 0x7f15f80b1d60 2023-10-06T03:30:12.982 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:12.977+0000 7f160050b700 1 -- 172.21.15.110:0/367460861 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f15f80b1d60 msgr2=0x7f15f8156e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:12.982 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:12.977+0000 7f160050b700 1 --2- 172.21.15.110:0/367460861 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f15f80b1d60 0x7f15f8156e00 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f15f4009910 tx=0x7f15f40091a0 comp rx=0 tx=0).stop 2023-10-06T03:30:12.982 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:12.977+0000 7f160050b700 1 -- 172.21.15.110:0/367460861 shutdown_connections 2023-10-06T03:30:12.982 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:12.977+0000 7f160050b700 1 --2- 172.21.15.110:0/367460861 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f15f80b1d60 0x7f15f8156e00 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:12.983 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:12.977+0000 7f160050b700 1 -- 172.21.15.110:0/367460861 >> 172.21.15.110:0/367460861 conn(0x7f15f80cc4b0 msgr2=0x7f15f80cc8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:30:12.983 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:12.977+0000 7f160050b700 1 -- 172.21.15.110:0/367460861 shutdown_connections 2023-10-06T03:30:12.983 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:12.977+0000 7f160050b700 1 -- 172.21.15.110:0/367460861 wait complete. 2023-10-06T03:30:12.984 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:12.981+0000 7f160050b700 1 Processor -- start 2023-10-06T03:30:12.984 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:12.981+0000 7f160050b700 1 -- start start 2023-10-06T03:30:12.987 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:12.981+0000 7f160050b700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f15f80b1d60 0x7f15f814dea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:12.987 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:12.981+0000 7f160050b700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f15f80c8880 0x7f15f80c1b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:12.987 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:12.981+0000 7f160050b700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f15f80cbfb0 0x7f15f80c2080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:12.987 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:12.981+0000 7f160050b700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15f8157530 con 0x7f15f80c8880 2023-10-06T03:30:12.987 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:12.981+0000 7f160050b700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15f80c3e50 con 0x7f15f80b1d60 2023-10-06T03:30:12.988 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:12.981+0000 7f160050b700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15f80c3560 con 0x7f15f80cbfb0 2023-10-06T03:30:12.988 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:12.981+0000 7f15fdaa6700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f15f80c8880 0x7f15f80c1b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:12.988 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:12.981+0000 7f15fdaa6700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f15f80c8880 0x7f15f80c1b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:52098/0 (socket says 172.21.15.110:52098) 2023-10-06T03:30:12.988 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:12.981+0000 7f15fdaa6700 1 -- 172.21.15.110:0/1581272510 learned_addr learned my addr 172.21.15.110:0/1581272510 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:30:12.988 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:12.981+0000 7f15fe2a7700 1 --2- 172.21.15.110:0/1581272510 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f15f80b1d60 0x7f15f814dea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:12.988 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:12.981+0000 7f15feaa8700 1 --2- 172.21.15.110:0/1581272510 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f15f80cbfb0 0x7f15f80c2080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:12.988 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:12.981+0000 7f15fdaa6700 1 -- 172.21.15.110:0/1581272510 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f15f80cbfb0 msgr2=0x7f15f80c2080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:12.988 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:12.981+0000 7f15fdaa6700 1 --2- 172.21.15.110:0/1581272510 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f15f80cbfb0 0x7f15f80c2080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:12.988 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:12.981+0000 7f15fdaa6700 1 -- 172.21.15.110:0/1581272510 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f15f80b1d60 msgr2=0x7f15f814dea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:12.989 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:12.981+0000 7f15fdaa6700 1 --2- 172.21.15.110:0/1581272510 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f15f80b1d60 0x7f15f814dea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:12.989 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:12.981+0000 7f15fdaa6700 1 -- 172.21.15.110:0/1581272510 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15f4009420 con 0x7f15f80c8880 2023-10-06T03:30:12.989 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:12.981+0000 7f15fdaa6700 1 --2- 172.21.15.110:0/1581272510 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f15f80c8880 0x7f15f80c1b40 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f15e800c580 tx=0x7f15e8007e10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:30:12.989 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:12.981+0000 7f15ee7fc700 1 -- 172.21.15.110:0/1581272510 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15e8010510 con 0x7f15f80c8880 2023-10-06T03:30:12.989 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:12.985+0000 7f15ee7fc700 1 -- 172.21.15.110:0/1581272510 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f15e8010670 con 0x7f15f80c8880 2023-10-06T03:30:12.989 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:12.985+0000 7f15ee7fc700 1 -- 172.21.15.110:0/1581272510 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15e8010a30 con 0x7f15f80c8880 2023-10-06T03:30:12.989 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:12.985+0000 7f160050b700 1 -- 172.21.15.110:0/1581272510 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15f816b3b0 con 0x7f15f80c8880 2023-10-06T03:30:12.990 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:12.985+0000 7f160050b700 1 -- 172.21.15.110:0/1581272510 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15f80c3ae0 con 0x7f15f80c8880 2023-10-06T03:30:12.991 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:12.985+0000 7f15ee7fc700 1 -- 172.21.15.110:0/1581272510 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7f15e801b400 con 0x7f15f80c8880 2023-10-06T03:30:12.991 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:12.985+0000 7f160050b700 1 -- 172.21.15.110:0/1581272510 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f15f806ff60 con 0x7f15f80c8880 2023-10-06T03:30:12.991 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:12.985+0000 7f15ee7fc700 1 --2- 172.21.15.110:0/1581272510 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f15dc060550 0x7f15dc062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:12.991 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:12.985+0000 7f15fe2a7700 1 --2- 172.21.15.110:0/1581272510 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f15dc060550 0x7f15dc062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:12.991 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:12.985+0000 7f15ee7fc700 1 -- 172.21.15.110:0/1581272510 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6809+0+0 (secure 0 0 0) 0x7f15e807c790 con 0x7f15f80c8880 2023-10-06T03:30:12.991 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:12.985+0000 7f15fe2a7700 1 --2- 172.21.15.110:0/1581272510 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f15dc060550 0x7f15dc062a00 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f15f4009860 tx=0x7f15f4017000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:30:12.998 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:12.993+0000 7f15ee7fc700 1 -- 172.21.15.110:0/1581272510 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f15e804c930 con 0x7f15f80c8880 2023-10-06T03:30:13.154 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:13.149+0000 7f160050b700 1 -- 172.21.15.110:0/1581272510 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f15f806c3b0 con 0x7f15f80c8880 2023-10-06T03:30:13.154 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:13.149+0000 7f15ee7fc700 1 -- 172.21.15.110:0/1581272510 <== mon.0 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v28) v1 ==== 93+0+10 (secure 0 0 0) 0x7f15e804ff50 con 0x7f15f80c8880 2023-10-06T03:30:13.155 INFO:teuthology.orchestra.run.smithi110.stdout:pg_num: 1 2023-10-06T03:30:13.157 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:13.153+0000 7f160050b700 1 -- 172.21.15.110:0/1581272510 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f15dc060550 msgr2=0x7f15dc062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:13.157 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:13.153+0000 7f160050b700 1 --2- 172.21.15.110:0/1581272510 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f15dc060550 0x7f15dc062a00 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f15f4009860 tx=0x7f15f4017000 comp rx=0 tx=0).stop 2023-10-06T03:30:13.157 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:13.153+0000 7f160050b700 1 -- 172.21.15.110:0/1581272510 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f15f80c8880 msgr2=0x7f15f80c1b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:13.158 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:13.153+0000 7f160050b700 1 --2- 172.21.15.110:0/1581272510 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f15f80c8880 0x7f15f80c1b40 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f15e800c580 tx=0x7f15e8007e10 comp rx=0 tx=0).stop 2023-10-06T03:30:13.158 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:13.153+0000 7f160050b700 1 -- 172.21.15.110:0/1581272510 shutdown_connections 2023-10-06T03:30:13.158 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:13.153+0000 7f160050b700 1 --2- 172.21.15.110:0/1581272510 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f15dc060550 0x7f15dc062a00 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:13.158 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:13.153+0000 7f160050b700 1 --2- 172.21.15.110:0/1581272510 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f15f80cbfb0 0x7f15f80c2080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:13.158 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:13.153+0000 7f160050b700 1 --2- 172.21.15.110:0/1581272510 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f15f80c8880 0x7f15f80c1b40 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:13.158 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:13.153+0000 7f160050b700 1 --2- 172.21.15.110:0/1581272510 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f15f80b1d60 0x7f15f814dea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:13.158 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:13.153+0000 7f160050b700 1 -- 172.21.15.110:0/1581272510 >> 172.21.15.110:0/1581272510 conn(0x7f15f80cc4b0 msgr2=0x7f15f806cbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:30:13.158 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:13.153+0000 7f160050b700 1 -- 172.21.15.110:0/1581272510 shutdown_connections 2023-10-06T03:30:13.158 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:13.153+0000 7f160050b700 1 -- 172.21.15.110:0/1581272510 wait complete. 2023-10-06T03:30:13.169 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-10-06T03:30:13.284 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:13.281+0000 7fde727f3700 1 Processor -- start 2023-10-06T03:30:13.284 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:13.281+0000 7fde727f3700 1 -- start start 2023-10-06T03:30:13.287 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:13.281+0000 7fde727f3700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fde6c06b7c0 0x7fde6c0ce170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:13.287 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:13.281+0000 7fde727f3700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fde6c0ce6b0 0x7fde6c06bb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:13.287 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:13.281+0000 7fde727f3700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fde6c06c0d0 0x7fde6c0afa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:13.287 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:13.281+0000 7fde727f3700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fde6c0cb1c0 con 0x7fde6c06c0d0 2023-10-06T03:30:13.287 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:13.281+0000 7fde727f3700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fde6c0cfa00 con 0x7fde6c0ce6b0 2023-10-06T03:30:13.287 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:13.281+0000 7fde727f3700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde6c0cedf0 con 0x7fde6c06b7c0 2023-10-06T03:30:13.287 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:13.281+0000 7fde71ff2700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fde6c06c0d0 0x7fde6c0afa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:13.287 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:13.281+0000 7fde71ff2700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fde6c06c0d0 0x7fde6c0afa20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:52114/0 (socket says 172.21.15.110:52114) 2023-10-06T03:30:13.287 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:13.281+0000 7fde71ff2700 1 -- 172.21.15.110:0/475952291 learned_addr learned my addr 172.21.15.110:0/475952291 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:30:13.288 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:13.281+0000 7fde717f1700 1 --2- 172.21.15.110:0/475952291 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fde6c06b7c0 0x7fde6c0ce170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:13.288 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:13.281+0000 7fde70ff0700 1 --2- 172.21.15.110:0/475952291 >> v2:172.21.15.192:3300/0 conn(0x7fde6c0ce6b0 0x7fde6c06bb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:13.288 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:13.281+0000 7fde71ff2700 1 -- 172.21.15.110:0/475952291 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fde6c06b7c0 msgr2=0x7fde6c0ce170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:30:13.288 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:13.281+0000 7fde71ff2700 1 --2- 172.21.15.110:0/475952291 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fde6c06b7c0 0x7fde6c0ce170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:13.288 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:13.281+0000 7fde71ff2700 1 -- 172.21.15.110:0/475952291 >> v2:172.21.15.192:3300/0 conn(0x7fde6c0ce6b0 msgr2=0x7fde6c06bb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:30:13.288 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:13.281+0000 7fde71ff2700 1 --2- 172.21.15.110:0/475952291 >> v2:172.21.15.192:3300/0 conn(0x7fde6c0ce6b0 0x7fde6c06bb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:13.288 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:13.281+0000 7fde71ff2700 1 -- 172.21.15.110:0/475952291 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde6c0cf0d0 con 0x7fde6c06c0d0 2023-10-06T03:30:13.288 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:13.281+0000 7fde70ff0700 1 --2- 172.21.15.110:0/475952291 >> v2:172.21.15.192:3300/0 conn(0x7fde6c0ce6b0 0x7fde6c06bb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-06T03:30:13.288 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:13.281+0000 7fde71ff2700 1 --2- 172.21.15.110:0/475952291 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fde6c06c0d0 0x7fde6c0afa20 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fde6c06fe70 tx=0x7fde64009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=97d454998aad6827 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:30:13.288 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:13.281+0000 7fde63fff700 1 -- 172.21.15.110:0/475952291 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde6400e070 con 0x7fde6c06c0d0 2023-10-06T03:30:13.289 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:13.281+0000 7fde63fff700 1 -- 172.21.15.110:0/475952291 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde6400ba20 con 0x7fde6c06c0d0 2023-10-06T03:30:13.289 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:13.281+0000 7fde63fff700 1 -- 172.21.15.110:0/475952291 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde640092e0 con 0x7fde6c06c0d0 2023-10-06T03:30:13.290 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:13.285+0000 7fde727f3700 1 -- 172.21.15.110:0/475952291 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fde6c06c0d0 msgr2=0x7fde6c0afa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:13.290 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:13.285+0000 7fde727f3700 1 --2- 172.21.15.110:0/475952291 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fde6c06c0d0 0x7fde6c0afa20 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fde6c06fe70 tx=0x7fde64009840 comp rx=0 tx=0).stop 2023-10-06T03:30:13.290 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:13.285+0000 7fde727f3700 1 -- 172.21.15.110:0/475952291 shutdown_connections 2023-10-06T03:30:13.290 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:13.285+0000 7fde727f3700 1 --2- 172.21.15.110:0/475952291 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fde6c06c0d0 0x7fde6c0afa20 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:13.290 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:13.285+0000 7fde727f3700 1 --2- 172.21.15.110:0/475952291 >> v2:172.21.15.192:3300/0 conn(0x7fde6c0ce6b0 0x7fde6c06bb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:13.290 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:13.285+0000 7fde727f3700 1 --2- 172.21.15.110:0/475952291 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fde6c06b7c0 0x7fde6c0ce170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:13.291 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:13.285+0000 7fde727f3700 1 -- 172.21.15.110:0/475952291 >> 172.21.15.110:0/475952291 conn(0x7fde6c0cfea0 msgr2=0x7fde6c0d02a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:30:13.291 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:13.285+0000 7fde727f3700 1 -- 172.21.15.110:0/475952291 shutdown_connections 2023-10-06T03:30:13.291 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:13.285+0000 7fde727f3700 1 -- 172.21.15.110:0/475952291 wait complete. 2023-10-06T03:30:13.291 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:13.285+0000 7fde727f3700 1 Processor -- start 2023-10-06T03:30:13.291 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:13.285+0000 7fde727f3700 1 -- start start 2023-10-06T03:30:13.293 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:13.285+0000 7fde727f3700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fde6c06b7c0 0x7fde6c145900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:13.293 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:13.285+0000 7fde727f3700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fde6c06c0d0 0x7fde6c157b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:13.293 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:13.285+0000 7fde727f3700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fde6c0ce6b0 0x7fde6c158070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:13.293 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:13.285+0000 7fde727f3700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde6c0b3910 con 0x7fde6c0ce6b0 2023-10-06T03:30:13.293 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:13.285+0000 7fde727f3700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde6c145e40 con 0x7fde6c06c0d0 2023-10-06T03:30:13.293 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:13.285+0000 7fde727f3700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde6c0b2d00 con 0x7fde6c06b7c0 2023-10-06T03:30:13.294 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:13.285+0000 7fde71ff2700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fde6c0ce6b0 0x7fde6c158070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:13.294 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:13.285+0000 7fde71ff2700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fde6c0ce6b0 0x7fde6c158070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:52116/0 (socket says 172.21.15.110:52116) 2023-10-06T03:30:13.294 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:13.289+0000 7fde71ff2700 1 -- 172.21.15.110:0/1362505351 learned_addr learned my addr 172.21.15.110:0/1362505351 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:30:13.294 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:13.289+0000 7fde717f1700 1 --2- 172.21.15.110:0/1362505351 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fde6c06b7c0 0x7fde6c145900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:13.294 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:13.289+0000 7fde70ff0700 1 --2- 172.21.15.110:0/1362505351 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fde6c06c0d0 0x7fde6c157b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:13.294 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:13.289+0000 7fde71ff2700 1 -- 172.21.15.110:0/1362505351 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fde6c06b7c0 msgr2=0x7fde6c145900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:13.294 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:13.289+0000 7fde71ff2700 1 --2- 172.21.15.110:0/1362505351 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fde6c06b7c0 0x7fde6c145900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:13.294 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:13.289+0000 7fde71ff2700 1 -- 172.21.15.110:0/1362505351 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fde6c06c0d0 msgr2=0x7fde6c157b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:13.294 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:13.289+0000 7fde71ff2700 1 --2- 172.21.15.110:0/1362505351 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fde6c06c0d0 0x7fde6c157b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:13.294 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:13.289+0000 7fde71ff2700 1 -- 172.21.15.110:0/1362505351 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde64009ae0 con 0x7fde6c0ce6b0 2023-10-06T03:30:13.295 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:13.289+0000 7fde717f1700 1 --2- 172.21.15.110:0/1362505351 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fde6c06b7c0 0x7fde6c145900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-06T03:30:13.295 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:13.289+0000 7fde71ff2700 1 --2- 172.21.15.110:0/1362505351 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fde6c0ce6b0 0x7fde6c158070 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fde6c06fef0 tx=0x7fde64018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:30:13.295 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:13.289+0000 7fde617fa700 1 -- 172.21.15.110:0/1362505351 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde640032f0 con 0x7fde6c0ce6b0 2023-10-06T03:30:13.295 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:13.289+0000 7fde617fa700 1 -- 172.21.15.110:0/1362505351 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde64008db0 con 0x7fde6c0ce6b0 2023-10-06T03:30:13.295 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:13.289+0000 7fde727f3700 1 -- 172.21.15.110:0/1362505351 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde6c0b2fe0 con 0x7fde6c0ce6b0 2023-10-06T03:30:13.295 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:13.289+0000 7fde617fa700 1 -- 172.21.15.110:0/1362505351 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde64026690 con 0x7fde6c0ce6b0 2023-10-06T03:30:13.295 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:13.289+0000 7fde727f3700 1 -- 172.21.15.110:0/1362505351 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde6c0b35a0 con 0x7fde6c0ce6b0 2023-10-06T03:30:13.297 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:13.289+0000 7fde727f3700 1 -- 172.21.15.110:0/1362505351 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde6c0b3ac0 con 0x7fde6c0ce6b0 2023-10-06T03:30:13.297 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:13.289+0000 7fde617fa700 1 -- 172.21.15.110:0/1362505351 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7fde64003040 con 0x7fde6c0ce6b0 2023-10-06T03:30:13.297 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:13.289+0000 7fde617fa700 1 --2- 172.21.15.110:0/1362505351 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fde50060550 0x7fde50062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:13.297 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:13.289+0000 7fde617fa700 1 -- 172.21.15.110:0/1362505351 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6809+0+0 (secure 0 0 0) 0x7fde6407ebd0 con 0x7fde6c0ce6b0 2023-10-06T03:30:13.297 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:13.289+0000 7fde717f1700 1 --2- 172.21.15.110:0/1362505351 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fde50060550 0x7fde50062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:13.297 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:13.293+0000 7fde717f1700 1 --2- 172.21.15.110:0/1362505351 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fde50060550 0x7fde50062a00 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fde68011480 tx=0x7fde680091a0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:30:13.304 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:13.301+0000 7fde617fa700 1 -- 172.21.15.110:0/1362505351 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fde6c0b3ac0 con 0x7fde6c0ce6b0 2023-10-06T03:30:13.458 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:13.453+0000 7fde727f3700 1 -- 172.21.15.110:0/1362505351 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7fde6c070ff0 con 0x7fde6c0ce6b0 2023-10-06T03:30:13.463 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:13.461+0000 7fde617fa700 1 -- 172.21.15.110:0/1362505351 <== mon.0 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v28) v1 ==== 92+0+10 (secure 0 0 0) 0x7fde6404ed70 con 0x7fde6c0ce6b0 2023-10-06T03:30:13.463 INFO:teuthology.orchestra.run.smithi110.stdout:pg_num: 8 2023-10-06T03:30:13.466 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:13.461+0000 7fde727f3700 1 -- 172.21.15.110:0/1362505351 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fde50060550 msgr2=0x7fde50062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:13.466 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:13.461+0000 7fde727f3700 1 --2- 172.21.15.110:0/1362505351 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fde50060550 0x7fde50062a00 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fde68011480 tx=0x7fde680091a0 comp rx=0 tx=0).stop 2023-10-06T03:30:13.466 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:13.461+0000 7fde727f3700 1 -- 172.21.15.110:0/1362505351 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fde6c0ce6b0 msgr2=0x7fde6c158070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:13.467 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:13.461+0000 7fde727f3700 1 --2- 172.21.15.110:0/1362505351 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fde6c0ce6b0 0x7fde6c158070 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fde6c06fef0 tx=0x7fde64018000 comp rx=0 tx=0).stop 2023-10-06T03:30:13.467 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:13.461+0000 7fde727f3700 1 -- 172.21.15.110:0/1362505351 shutdown_connections 2023-10-06T03:30:13.467 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:13.461+0000 7fde727f3700 1 --2- 172.21.15.110:0/1362505351 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fde50060550 0x7fde50062a00 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:13.467 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:13.461+0000 7fde727f3700 1 --2- 172.21.15.110:0/1362505351 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fde6c0ce6b0 0x7fde6c158070 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:13.467 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:13.461+0000 7fde727f3700 1 --2- 172.21.15.110:0/1362505351 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fde6c06c0d0 0x7fde6c157b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:13.467 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:13.461+0000 7fde727f3700 1 --2- 172.21.15.110:0/1362505351 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fde6c06b7c0 0x7fde6c145900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:13.467 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:13.461+0000 7fde727f3700 1 -- 172.21.15.110:0/1362505351 >> 172.21.15.110:0/1362505351 conn(0x7fde6c0cfea0 msgr2=0x7fde6c0b1bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:30:13.467 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:13.461+0000 7fde727f3700 1 -- 172.21.15.110:0/1362505351 shutdown_connections 2023-10-06T03:30:13.467 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:13.461+0000 7fde727f3700 1 -- 172.21.15.110:0/1362505351 wait complete. 2023-10-06T03:30:13.480 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-06T03:30:13.588 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:13.585+0000 7f268919e700 1 Processor -- start 2023-10-06T03:30:13.588 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:13.585+0000 7f268919e700 1 -- start start 2023-10-06T03:30:13.591 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:13.585+0000 7f268919e700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f268406c970 0x7f268406cd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:13.591 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:13.585+0000 7f268919e700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f2684159c20 0x7f2684155b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:13.591 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:13.585+0000 7f268919e700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f26841560d0 0x7f2684152ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:13.592 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:13.585+0000 7f268919e700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f26841591d0 con 0x7f268406c970 2023-10-06T03:30:13.592 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:13.585+0000 7f268919e700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2684156700 con 0x7f2684159c20 2023-10-06T03:30:13.592 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:13.585+0000 7f268919e700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26841569e0 con 0x7f26841560d0 2023-10-06T03:30:13.592 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:13.585+0000 7f2683fff700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f268406c970 0x7f268406cd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:13.592 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:13.585+0000 7f2683fff700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f268406c970 0x7f268406cd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:52130/0 (socket says 172.21.15.110:52130) 2023-10-06T03:30:13.592 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:13.585+0000 7f2683fff700 1 -- 172.21.15.110:0/2438381552 learned_addr learned my addr 172.21.15.110:0/2438381552 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:30:13.592 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:13.585+0000 7f268899d700 1 --2- 172.21.15.110:0/2438381552 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f26841560d0 0x7f2684152ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:13.592 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:13.585+0000 7f26837fe700 1 --2- 172.21.15.110:0/2438381552 >> v2:172.21.15.192:3300/0 conn(0x7f2684159c20 0x7f2684155b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:13.592 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:13.585+0000 7f2683fff700 1 -- 172.21.15.110:0/2438381552 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f26841560d0 msgr2=0x7f2684152ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:30:13.592 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:13.585+0000 7f2683fff700 1 --2- 172.21.15.110:0/2438381552 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f26841560d0 0x7f2684152ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:13.592 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:13.589+0000 7f2683fff700 1 -- 172.21.15.110:0/2438381552 >> v2:172.21.15.192:3300/0 conn(0x7f2684159c20 msgr2=0x7f2684155b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:30:13.592 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:13.589+0000 7f2683fff700 1 --2- 172.21.15.110:0/2438381552 >> v2:172.21.15.192:3300/0 conn(0x7f2684159c20 0x7f2684155b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:13.593 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:13.589+0000 7f2683fff700 1 -- 172.21.15.110:0/2438381552 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2684159420 con 0x7f268406c970 2023-10-06T03:30:13.593 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:13.589+0000 7f2683fff700 1 --2- 172.21.15.110:0/2438381552 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f268406c970 0x7f268406cd40 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f26740099b0 tx=0x7f2674009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c59e97c0c782ba57 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:30:13.594 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:13.589+0000 7f2682ffd700 1 -- 172.21.15.110:0/2438381552 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f267400b820 con 0x7f268406c970 2023-10-06T03:30:13.594 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:13.589+0000 7f2682ffd700 1 -- 172.21.15.110:0/2438381552 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f267400b980 con 0x7f268406c970 2023-10-06T03:30:13.594 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:13.589+0000 7f2682ffd700 1 -- 172.21.15.110:0/2438381552 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f267400bd40 con 0x7f268406c970 2023-10-06T03:30:13.594 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:13.589+0000 7f268919e700 1 -- 172.21.15.110:0/2438381552 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f268406c970 msgr2=0x7f268406cd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:13.594 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:13.589+0000 7f268919e700 1 --2- 172.21.15.110:0/2438381552 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f268406c970 0x7f268406cd40 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f26740099b0 tx=0x7f2674009250 comp rx=0 tx=0).stop 2023-10-06T03:30:13.594 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:13.589+0000 7f268919e700 1 -- 172.21.15.110:0/2438381552 shutdown_connections 2023-10-06T03:30:13.594 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:13.589+0000 7f268919e700 1 --2- 172.21.15.110:0/2438381552 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f26841560d0 0x7f2684152ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:13.594 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:13.589+0000 7f268919e700 1 --2- 172.21.15.110:0/2438381552 >> v2:172.21.15.192:3300/0 conn(0x7f2684159c20 0x7f2684155b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:13.594 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:13.589+0000 7f268919e700 1 --2- 172.21.15.110:0/2438381552 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f268406c970 0x7f268406cd40 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:13.595 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:13.589+0000 7f268919e700 1 -- 172.21.15.110:0/2438381552 >> 172.21.15.110:0/2438381552 conn(0x7f268406ab30 msgr2=0x7f26840adff0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:30:13.595 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:13.589+0000 7f268919e700 1 -- 172.21.15.110:0/2438381552 shutdown_connections 2023-10-06T03:30:13.595 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:13.589+0000 7f268919e700 1 -- 172.21.15.110:0/2438381552 wait complete. 2023-10-06T03:30:13.595 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:13.589+0000 7f268919e700 1 Processor -- start 2023-10-06T03:30:13.595 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:13.589+0000 7f268919e700 1 -- start start 2023-10-06T03:30:13.597 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:13.593+0000 7f268919e700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f268406c970 0x7f26840c7910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:13.597 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:13.593+0000 7f268919e700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f26841560d0 0x7f26840adb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:13.597 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:13.593+0000 7f268919e700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2684159c20 0x7f268414f360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:13.598 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:13.593+0000 7f268919e700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26840ae0c0 con 0x7f2684159c20 2023-10-06T03:30:13.598 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:13.593+0000 7f268919e700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26840c8010 con 0x7f268406c970 2023-10-06T03:30:13.598 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:13.593+0000 7f268919e700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26840c82f0 con 0x7f26841560d0 2023-10-06T03:30:13.598 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:13.593+0000 7f268899d700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2684159c20 0x7f268414f360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:13.598 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:13.593+0000 7f268899d700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2684159c20 0x7f268414f360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:52146/0 (socket says 172.21.15.110:52146) 2023-10-06T03:30:13.598 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:13.593+0000 7f268899d700 1 -- 172.21.15.110:0/3553893587 learned_addr learned my addr 172.21.15.110:0/3553893587 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:30:13.598 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:13.593+0000 7f2683fff700 1 --2- 172.21.15.110:0/3553893587 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f268406c970 0x7f26840c7910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:13.598 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:13.593+0000 7f26837fe700 1 --2- 172.21.15.110:0/3553893587 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f26841560d0 0x7f26840adb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:13.599 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:13.593+0000 7f268899d700 1 -- 172.21.15.110:0/3553893587 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f26841560d0 msgr2=0x7f26840adb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:13.599 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:13.593+0000 7f268899d700 1 --2- 172.21.15.110:0/3553893587 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f26841560d0 0x7f26840adb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:13.600 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:13.593+0000 7f268899d700 1 -- 172.21.15.110:0/3553893587 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f268406c970 msgr2=0x7f26840c7910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:13.600 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:13.593+0000 7f268899d700 1 --2- 172.21.15.110:0/3553893587 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f268406c970 0x7f26840c7910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:13.600 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:13.593+0000 7f268899d700 1 -- 172.21.15.110:0/3553893587 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26740094f0 con 0x7f2684159c20 2023-10-06T03:30:13.600 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:13.593+0000 7f268899d700 1 --2- 172.21.15.110:0/3553893587 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2684159c20 0x7f268414f360 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f267c007f30 tx=0x7f267c00c040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:30:13.600 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:13.593+0000 7f266bfff700 1 -- 172.21.15.110:0/3553893587 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f267c00f040 con 0x7f2684159c20 2023-10-06T03:30:13.600 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:13.593+0000 7f266bfff700 1 -- 172.21.15.110:0/3553893587 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f267c00d960 con 0x7f2684159c20 2023-10-06T03:30:13.600 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:13.593+0000 7f268919e700 1 -- 172.21.15.110:0/3553893587 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26840c5850 con 0x7f2684159c20 2023-10-06T03:30:13.600 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:13.593+0000 7f266bfff700 1 -- 172.21.15.110:0/3553893587 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f267c00dd20 con 0x7f2684159c20 2023-10-06T03:30:13.600 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:13.593+0000 7f268919e700 1 -- 172.21.15.110:0/3553893587 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26840c5e10 con 0x7f2684159c20 2023-10-06T03:30:13.601 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:13.597+0000 7f268919e700 1 -- 172.21.15.110:0/3553893587 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f26840c6030 con 0x7f2684159c20 2023-10-06T03:30:13.601 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:13.597+0000 7f266bfff700 1 -- 172.21.15.110:0/3553893587 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7f267c008450 con 0x7f2684159c20 2023-10-06T03:30:13.601 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:13.597+0000 7f266bfff700 1 --2- 172.21.15.110:0/3553893587 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f2670060550 0x7f2670062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:13.601 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:13.597+0000 7f266bfff700 1 -- 172.21.15.110:0/3553893587 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6809+0+0 (secure 0 0 0) 0x7f267c07cd70 con 0x7f2684159c20 2023-10-06T03:30:13.601 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:13.597+0000 7f2683fff700 1 --2- 172.21.15.110:0/3553893587 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f2670060550 0x7f2670062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:13.602 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:13.597+0000 7f2683fff700 1 --2- 172.21.15.110:0/3553893587 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f2670060550 0x7f2670062a00 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f26740099b0 tx=0x7f2674017000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:30:13.607 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:13.605+0000 7f266bfff700 1 -- 172.21.15.110:0/3553893587 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f267c04cf10 con 0x7f2684159c20 2023-10-06T03:30:13.773 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:13.769+0000 7f268919e700 1 -- 172.21.15.110:0/3553893587 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f2684070ff0 con 0x7f2684159c20 2023-10-06T03:30:13.773 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:13.769+0000 7f266bfff700 1 -- 172.21.15.110:0/3553893587 <== mon.0 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v28) v1 ==== 104+0+11 (secure 0 0 0) 0x7f267c050530 con 0x7f2684159c20 2023-10-06T03:30:13.774 INFO:teuthology.orchestra.run.smithi110.stdout:pg_num: 32 2023-10-06T03:30:13.776 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:13.773+0000 7f268919e700 1 -- 172.21.15.110:0/3553893587 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f2670060550 msgr2=0x7f2670062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:13.777 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:13.773+0000 7f268919e700 1 --2- 172.21.15.110:0/3553893587 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f2670060550 0x7f2670062a00 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f26740099b0 tx=0x7f2674017000 comp rx=0 tx=0).stop 2023-10-06T03:30:13.777 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:13.773+0000 7f268919e700 1 -- 172.21.15.110:0/3553893587 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2684159c20 msgr2=0x7f268414f360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:13.777 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:13.773+0000 7f268919e700 1 --2- 172.21.15.110:0/3553893587 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2684159c20 0x7f268414f360 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f267c007f30 tx=0x7f267c00c040 comp rx=0 tx=0).stop 2023-10-06T03:30:13.777 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:13.773+0000 7f268919e700 1 -- 172.21.15.110:0/3553893587 shutdown_connections 2023-10-06T03:30:13.777 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:13.773+0000 7f268919e700 1 --2- 172.21.15.110:0/3553893587 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f2670060550 0x7f2670062a00 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:13.777 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:13.773+0000 7f268919e700 1 --2- 172.21.15.110:0/3553893587 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2684159c20 0x7f268414f360 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:13.777 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:13.773+0000 7f268919e700 1 --2- 172.21.15.110:0/3553893587 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f26841560d0 0x7f26840adb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:13.777 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:13.773+0000 7f268919e700 1 --2- 172.21.15.110:0/3553893587 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f268406c970 0x7f26840c7910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:13.777 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:13.773+0000 7f268919e700 1 -- 172.21.15.110:0/3553893587 >> 172.21.15.110:0/3553893587 conn(0x7f268406ab30 msgr2=0x7f26840cbee0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:30:13.777 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:13.773+0000 7f268919e700 1 -- 172.21.15.110:0/3553893587 shutdown_connections 2023-10-06T03:30:13.778 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:13.773+0000 7f268919e700 1 -- 172.21.15.110:0/3553893587 wait complete. 2023-10-06T03:30:13.788 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-06T03:30:13.897 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:13.893+0000 7ffbcaf1b700 1 Processor -- start 2023-10-06T03:30:13.897 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:13.893+0000 7ffbcaf1b700 1 -- start start 2023-10-06T03:30:13.899 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:13.893+0000 7ffbcaf1b700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7ffbc4155890 0x7ffbc40af3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:13.899 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:13.893+0000 7ffbcaf1b700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7ffbc40b0e60 0x7ffbc406b7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:13.899 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:13.893+0000 7ffbcaf1b700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffbc40aef30 con 0x7ffbc40af900 2023-10-06T03:30:13.899 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:13.893+0000 7ffbcaf1b700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffbc4069790 con 0x7ffbc40b0e60 2023-10-06T03:30:13.899 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:13.893+0000 7ffbcaf1b700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffbc4156d70 con 0x7ffbc4155890 2023-10-06T03:30:13.899 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:13.893+0000 7ffbca71a700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7ffbc40b0e60 0x7ffbc406b7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:13.900 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:13.893+0000 7ffbc9718700 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7ffbc40af900 0x7ffbc40b0750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:53260/0 (socket says 172.21.15.110:53260) 2023-10-06T03:30:13.900 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:13.893+0000 7ffbc9718700 1 -- 172.21.15.110:0/2431101779 learned_addr learned my addr 172.21.15.110:0/2431101779 (peer_addr_for_me v1:172.21.15.110:0/0) 2023-10-06T03:30:13.900 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:13.893+0000 7ffbc9f19700 1 --2- 172.21.15.110:0/2431101779 >> v2:172.21.15.192:3300/0 conn(0x7ffbc4155890 0x7ffbc40af3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:13.900 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:13.893+0000 7ffbc8f17700 1 -- 172.21.15.110:0/2431101779 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2760148815 0 0) 0x7ffbc40aef30 con 0x7ffbc40af900 2023-10-06T03:30:13.900 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:13.893+0000 7ffbc8f17700 1 -- 172.21.15.110:0/2431101779 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffbb0003570 con 0x7ffbc40af900 2023-10-06T03:30:13.900 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:13.893+0000 7ffbc8f17700 1 -- 172.21.15.110:0/2431101779 <== mon.0 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1479679237 0 0) 0x7ffbb0003570 con 0x7ffbc40af900 2023-10-06T03:30:13.900 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:13.893+0000 7ffbc9f19700 1 -- 172.21.15.110:0/2431101779 >> v2:172.21.15.110:3300/0 conn(0x7ffbc40b0e60 msgr2=0x7ffbc406b7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:30:13.900 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:13.893+0000 7ffbc9f19700 1 --2- 172.21.15.110:0/2431101779 >> v2:172.21.15.110:3300/0 conn(0x7ffbc40b0e60 0x7ffbc406b7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:13.900 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:13.893+0000 7ffbc9f19700 1 -- 172.21.15.110:0/2431101779 >> v1:172.21.15.110:6789/0 conn(0x7ffbc40af900 legacy=0x7ffbc40b0750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:13.900 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:13.893+0000 7ffbc9f19700 1 -- 172.21.15.110:0/2431101779 --> v2:172.21.15.192:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffbc4157050 con 0x7ffbc4155890 2023-10-06T03:30:13.902 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:13.897+0000 7ffbc9f19700 1 --2- 172.21.15.110:0/2431101779 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ffbc4155890 0x7ffbc40af3c0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7ffbc40692e0 tx=0x7ffbb40091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f1291fb5c121161e server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:30:13.902 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:13.897+0000 7ffbc8f17700 1 -- 172.21.15.110:0/2431101779 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbc0001470 con 0x7ffbc4155890 2023-10-06T03:30:13.902 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:13.897+0000 7ffbc8f17700 1 -- 172.21.15.110:0/2431101779 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffbb4011420 con 0x7ffbc4155890 2023-10-06T03:30:13.902 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:13.897+0000 7ffbc8f17700 1 -- 172.21.15.110:0/2431101779 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbb40117c0 con 0x7ffbc4155890 2023-10-06T03:30:13.902 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:13.897+0000 7ffbcaf1b700 1 -- 172.21.15.110:0/2431101779 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ffbc4155890 msgr2=0x7ffbc40af3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:13.903 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:13.897+0000 7ffbcaf1b700 1 --2- 172.21.15.110:0/2431101779 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ffbc4155890 0x7ffbc40af3c0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7ffbc40692e0 tx=0x7ffbb40091a0 comp rx=0 tx=0).stop 2023-10-06T03:30:13.903 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:13.897+0000 7ffbcaf1b700 1 -- 172.21.15.110:0/2431101779 shutdown_connections 2023-10-06T03:30:13.903 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:13.897+0000 7ffbcaf1b700 1 --2- 172.21.15.110:0/2431101779 >> v2:172.21.15.110:3300/0 conn(0x7ffbc40b0e60 0x7ffbc406b7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:13.903 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:13.897+0000 7ffbcaf1b700 1 --2- 172.21.15.110:0/2431101779 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ffbc4155890 0x7ffbc40af3c0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:13.903 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:13.897+0000 7ffbcaf1b700 1 -- 172.21.15.110:0/2431101779 >> 172.21.15.110:0/2431101779 conn(0x7ffbc40cbac0 msgr2=0x7ffbc40cbec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:30:13.903 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:13.897+0000 7ffbcaf1b700 1 -- 172.21.15.110:0/2431101779 shutdown_connections 2023-10-06T03:30:13.903 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:13.897+0000 7ffbcaf1b700 1 -- 172.21.15.110:0/2431101779 wait complete. 2023-10-06T03:30:13.904 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:13.897+0000 7ffbcaf1b700 1 Processor -- start 2023-10-06T03:30:13.904 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:13.897+0000 7ffbcaf1b700 1 -- start start 2023-10-06T03:30:13.905 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:13.901+0000 7ffbcaf1b700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ffbc40af900 0x7ffbc4146f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:13.905 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:13.901+0000 7ffbcaf1b700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ffbc40b0e60 0x7ffbc414d4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:13.905 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:13.901+0000 7ffbcaf1b700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ffbc4155890 0x7ffbc4152e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:13.905 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:13.901+0000 7ffbcaf1b700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbc4148d00 con 0x7ffbc40af900 2023-10-06T03:30:13.905 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:13.901+0000 7ffbcaf1b700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbc4148fe0 con 0x7ffbc4155890 2023-10-06T03:30:13.905 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:13.901+0000 7ffbcaf1b700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffbc41492c0 con 0x7ffbc40b0e60 2023-10-06T03:30:13.905 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:13.901+0000 7ffbc9f19700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ffbc40af900 0x7ffbc4146f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:13.906 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:13.901+0000 7ffbc9f19700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ffbc40af900 0x7ffbc4146f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:52164/0 (socket says 172.21.15.110:52164) 2023-10-06T03:30:13.906 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:13.901+0000 7ffbc9f19700 1 -- 172.21.15.110:0/3695768266 learned_addr learned my addr 172.21.15.110:0/3695768266 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:30:13.906 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:13.901+0000 7ffbc9718700 1 --2- 172.21.15.110:0/3695768266 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ffbc40b0e60 0x7ffbc414d4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:13.906 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:13.901+0000 7ffbca71a700 1 --2- 172.21.15.110:0/3695768266 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ffbc4155890 0x7ffbc4152e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:13.907 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:13.901+0000 7ffbc9f19700 1 -- 172.21.15.110:0/3695768266 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ffbc40b0e60 msgr2=0x7ffbc414d4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:13.907 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:13.901+0000 7ffbc9f19700 1 --2- 172.21.15.110:0/3695768266 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ffbc40b0e60 0x7ffbc414d4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:13.907 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:13.901+0000 7ffbc9f19700 1 -- 172.21.15.110:0/3695768266 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ffbc4155890 msgr2=0x7ffbc4152e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:13.907 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:13.901+0000 7ffbc9f19700 1 --2- 172.21.15.110:0/3695768266 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ffbc4155890 0x7ffbc4152e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:13.907 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:13.901+0000 7ffbc9f19700 1 -- 172.21.15.110:0/3695768266 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffbb4009420 con 0x7ffbc40af900 2023-10-06T03:30:13.907 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:13.901+0000 7ffbc9f19700 1 --2- 172.21.15.110:0/3695768266 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ffbc40af900 0x7ffbc4146f00 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7ffbb4002000 tx=0x7ffbb4016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:30:13.908 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:13.901+0000 7ffbb9ffb700 1 -- 172.21.15.110:0/3695768266 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbb401cab0 con 0x7ffbc40af900 2023-10-06T03:30:13.908 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:13.901+0000 7ffbb9ffb700 1 -- 172.21.15.110:0/3695768266 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffbb401cc10 con 0x7ffbc40af900 2023-10-06T03:30:13.908 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:13.901+0000 7ffbcaf1b700 1 -- 172.21.15.110:0/3695768266 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffbc41495a0 con 0x7ffbc40af900 2023-10-06T03:30:13.908 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:13.901+0000 7ffbb9ffb700 1 -- 172.21.15.110:0/3695768266 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbb401a450 con 0x7ffbc40af900 2023-10-06T03:30:13.908 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:13.901+0000 7ffbcaf1b700 1 -- 172.21.15.110:0/3695768266 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffbc4147440 con 0x7ffbc40af900 2023-10-06T03:30:13.908 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:13.905+0000 7ffbcaf1b700 1 -- 172.21.15.110:0/3695768266 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffbc41497b0 con 0x7ffbc40af900 2023-10-06T03:30:13.909 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:13.905+0000 7ffbb9ffb700 1 -- 172.21.15.110:0/3695768266 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7ffbb401a800 con 0x7ffbc40af900 2023-10-06T03:30:13.909 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:13.905+0000 7ffbb9ffb700 1 --2- 172.21.15.110:0/3695768266 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7ffba8060550 0x7ffba8062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:13.911 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:13.905+0000 7ffbc9718700 1 --2- 172.21.15.110:0/3695768266 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7ffba8060550 0x7ffba8062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:13.911 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:13.905+0000 7ffbb9ffb700 1 -- 172.21.15.110:0/3695768266 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6809+0+0 (secure 0 0 0) 0x7ffbb407e8a0 con 0x7ffbc40af900 2023-10-06T03:30:13.911 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:13.905+0000 7ffbc9718700 1 --2- 172.21.15.110:0/3695768266 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7ffba8060550 0x7ffba8062a00 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7ffbc0001ed0 tx=0x7ffbc0000bc0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:30:13.917 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:13.913+0000 7ffbb9ffb700 1 -- 172.21.15.110:0/3695768266 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7ffbb404e9a0 con 0x7ffbc40af900 2023-10-06T03:30:14.079 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:14.073+0000 7ffbcaf1b700 1 -- 172.21.15.110:0/3695768266 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7ffbc4070ff0 con 0x7ffbc40af900 2023-10-06T03:30:14.079 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:14.077+0000 7ffbb9ffb700 1 -- 172.21.15.110:0/3695768266 <== mon.0 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v28) v1 ==== 100+0+11 (secure 0 0 0) 0x7ffbb4051fc0 con 0x7ffbc40af900 2023-10-06T03:30:14.079 INFO:teuthology.orchestra.run.smithi110.stdout:pg_num: 64 2023-10-06T03:30:14.082 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:14.077+0000 7ffbcaf1b700 1 -- 172.21.15.110:0/3695768266 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7ffba8060550 msgr2=0x7ffba8062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:14.082 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:14.077+0000 7ffbcaf1b700 1 --2- 172.21.15.110:0/3695768266 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7ffba8060550 0x7ffba8062a00 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7ffbc0001ed0 tx=0x7ffbc0000bc0 comp rx=0 tx=0).stop 2023-10-06T03:30:14.082 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:14.077+0000 7ffbcaf1b700 1 -- 172.21.15.110:0/3695768266 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ffbc40af900 msgr2=0x7ffbc4146f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:14.082 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:14.077+0000 7ffbcaf1b700 1 --2- 172.21.15.110:0/3695768266 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ffbc40af900 0x7ffbc4146f00 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7ffbb4002000 tx=0x7ffbb4016000 comp rx=0 tx=0).stop 2023-10-06T03:30:14.082 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:14.077+0000 7ffbcaf1b700 1 -- 172.21.15.110:0/3695768266 shutdown_connections 2023-10-06T03:30:14.083 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:14.077+0000 7ffbcaf1b700 1 --2- 172.21.15.110:0/3695768266 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7ffba8060550 0x7ffba8062a00 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:14.083 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:14.077+0000 7ffbcaf1b700 1 --2- 172.21.15.110:0/3695768266 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ffbc4155890 0x7ffbc4152e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:14.083 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:14.077+0000 7ffbcaf1b700 1 --2- 172.21.15.110:0/3695768266 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ffbc40b0e60 0x7ffbc414d4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:14.083 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:14.077+0000 7ffbcaf1b700 1 --2- 172.21.15.110:0/3695768266 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ffbc40af900 0x7ffbc4146f00 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:14.083 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:14.077+0000 7ffbcaf1b700 1 -- 172.21.15.110:0/3695768266 >> 172.21.15.110:0/3695768266 conn(0x7ffbc40cbac0 msgr2=0x7ffbc406b7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:30:14.083 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:14.077+0000 7ffbcaf1b700 1 -- 172.21.15.110:0/3695768266 shutdown_connections 2023-10-06T03:30:14.083 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:14.077+0000 7ffbcaf1b700 1 -- 172.21.15.110:0/3695768266 wait complete. 2023-10-06T03:30:14.094 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2023-10-06T03:30:14.208 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:14.205+0000 7fed7b59e700 1 Processor -- start 2023-10-06T03:30:14.209 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:14.205+0000 7fed7b59e700 1 -- start start 2023-10-06T03:30:14.210 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:14.205+0000 7fed7b59e700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fed7c0670a0 0x7fed7c067470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:14.210 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:14.205+0000 7fed7b59e700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fed7c15ab20 0x7fed7c06d380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:14.210 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:14.205+0000 7fed7b59e700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed7c06cef0 con 0x7fed7c06d8c0 2023-10-06T03:30:14.210 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:14.205+0000 7fed7b59e700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fed7c15ba50 con 0x7fed7c0670a0 2023-10-06T03:30:14.210 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:14.205+0000 7fed7b59e700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed7c0679b0 con 0x7fed7c15ab20 2023-10-06T03:30:14.210 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:14.205+0000 7fed7a59c700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fed7c0670a0 0x7fed7c067470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:14.210 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:14.205+0000 7fed7ad9d700 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7fed7c06d8c0 0x7fed7c158630 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:53276/0 (socket says 172.21.15.110:53276) 2023-10-06T03:30:14.211 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:14.205+0000 7fed7ad9d700 1 -- 172.21.15.110:0/1834836574 learned_addr learned my addr 172.21.15.110:0/1834836574 (peer_addr_for_me v1:172.21.15.110:0/0) 2023-10-06T03:30:14.211 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:14.205+0000 7fed79d9b700 1 --2- 172.21.15.110:0/1834836574 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fed7c15ab20 0x7fed7c06d380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:14.211 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:14.205+0000 7fed7a59c700 1 -- 172.21.15.110:0/1834836574 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fed7c15ab20 msgr2=0x7fed7c06d380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:30:14.211 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:14.205+0000 7fed7a59c700 1 --2- 172.21.15.110:0/1834836574 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fed7c15ab20 0x7fed7c06d380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:14.211 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:14.205+0000 7fed7a59c700 1 -- 172.21.15.110:0/1834836574 >> v1:172.21.15.110:6789/0 conn(0x7fed7c06d8c0 legacy=0x7fed7c158630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:14.211 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:14.205+0000 7fed7959a700 1 -- 172.21.15.110:0/1834836574 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 954818542 0 0) 0x7fed7c06cef0 con 0x7fed7c06d8c0 2023-10-06T03:30:14.211 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:14.205+0000 7fed7a59c700 1 -- 172.21.15.110:0/1834836574 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed7c15b490 con 0x7fed7c0670a0 2023-10-06T03:30:14.211 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:14.205+0000 7fed7a59c700 1 --2- 172.21.15.110:0/1834836574 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fed7c0670a0 0x7fed7c067470 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fed640098e0 tx=0x7fed640091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb51973165f7dcbc server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:30:14.212 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:14.205+0000 7fed7959a700 1 -- 172.21.15.110:0/1834836574 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed6400b640 con 0x7fed7c0670a0 2023-10-06T03:30:14.212 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:14.205+0000 7fed7959a700 1 -- 172.21.15.110:0/1834836574 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed6400b7a0 con 0x7fed7c0670a0 2023-10-06T03:30:14.212 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:14.205+0000 7fed7959a700 1 -- 172.21.15.110:0/1834836574 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed6400bb40 con 0x7fed7c0670a0 2023-10-06T03:30:14.213 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:14.209+0000 7fed7b59e700 1 -- 172.21.15.110:0/1834836574 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fed7c0670a0 msgr2=0x7fed7c067470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:14.213 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:14.209+0000 7fed7b59e700 1 --2- 172.21.15.110:0/1834836574 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fed7c0670a0 0x7fed7c067470 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fed640098e0 tx=0x7fed640091a0 comp rx=0 tx=0).stop 2023-10-06T03:30:14.213 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:14.209+0000 7fed7b59e700 1 -- 172.21.15.110:0/1834836574 shutdown_connections 2023-10-06T03:30:14.213 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:14.209+0000 7fed7b59e700 1 --2- 172.21.15.110:0/1834836574 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fed7c15ab20 0x7fed7c06d380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:14.213 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:14.209+0000 7fed7b59e700 1 --2- 172.21.15.110:0/1834836574 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fed7c0670a0 0x7fed7c067470 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:14.213 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:14.209+0000 7fed7b59e700 1 -- 172.21.15.110:0/1834836574 >> 172.21.15.110:0/1834836574 conn(0x7fed7c0ccdc0 msgr2=0x7fed7c0cd1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:30:14.213 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:14.209+0000 7fed7b59e700 1 -- 172.21.15.110:0/1834836574 shutdown_connections 2023-10-06T03:30:14.213 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:14.209+0000 7fed7b59e700 1 -- 172.21.15.110:0/1834836574 wait complete. 2023-10-06T03:30:14.214 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:14.209+0000 7fed7b59e700 1 Processor -- start 2023-10-06T03:30:14.214 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:14.209+0000 7fed7b59e700 1 -- start start 2023-10-06T03:30:14.217 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:14.213+0000 7fed7b59e700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fed7c0670a0 0x7fed7c0c87e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:14.217 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:14.213+0000 7fed7b59e700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fed7c06d8c0 0x7fed7c0c2e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:14.217 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:14.213+0000 7fed7b59e700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fed7c15ab20 0x7fed7c0c33a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:14.217 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:14.213+0000 7fed7b59e700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed7c0c38e0 con 0x7fed7c06d8c0 2023-10-06T03:30:14.217 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:14.213+0000 7fed7b59e700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed7c0c7740 con 0x7fed7c0670a0 2023-10-06T03:30:14.217 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:14.213+0000 7fed7b59e700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed7c0c7a20 con 0x7fed7c15ab20 2023-10-06T03:30:14.217 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:14.213+0000 7fed79d9b700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fed7c06d8c0 0x7fed7c0c2e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:14.217 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:14.213+0000 7fed79d9b700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fed7c06d8c0 0x7fed7c0c2e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:52192/0 (socket says 172.21.15.110:52192) 2023-10-06T03:30:14.217 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:14.213+0000 7fed79d9b700 1 -- 172.21.15.110:0/2397553195 learned_addr learned my addr 172.21.15.110:0/2397553195 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:30:14.217 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:14.213+0000 7fed7ad9d700 1 --2- 172.21.15.110:0/2397553195 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fed7c15ab20 0x7fed7c0c33a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:14.218 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:14.213+0000 7fed7a59c700 1 --2- 172.21.15.110:0/2397553195 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fed7c0670a0 0x7fed7c0c87e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:14.218 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:14.213+0000 7fed79d9b700 1 -- 172.21.15.110:0/2397553195 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fed7c15ab20 msgr2=0x7fed7c0c33a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:14.218 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:14.213+0000 7fed79d9b700 1 --2- 172.21.15.110:0/2397553195 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fed7c15ab20 0x7fed7c0c33a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:14.218 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:14.213+0000 7fed79d9b700 1 -- 172.21.15.110:0/2397553195 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fed7c0670a0 msgr2=0x7fed7c0c87e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:14.218 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:14.213+0000 7fed79d9b700 1 --2- 172.21.15.110:0/2397553195 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fed7c0670a0 0x7fed7c0c87e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:14.218 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:14.213+0000 7fed79d9b700 1 -- 172.21.15.110:0/2397553195 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed64009420 con 0x7fed7c06d8c0 2023-10-06T03:30:14.218 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:14.213+0000 7fed79d9b700 1 --2- 172.21.15.110:0/2397553195 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fed7c06d8c0 0x7fed7c0c2e60 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fed6c009e50 tx=0x7fed6c00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:30:14.218 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:14.213+0000 7fed727fc700 1 -- 172.21.15.110:0/2397553195 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed6c00cb50 con 0x7fed7c06d8c0 2023-10-06T03:30:14.218 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:14.213+0000 7fed727fc700 1 -- 172.21.15.110:0/2397553195 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed6c00f040 con 0x7fed7c06d8c0 2023-10-06T03:30:14.218 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:14.213+0000 7fed7b59e700 1 -- 172.21.15.110:0/2397553195 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed7c0c7d00 con 0x7fed7c06d8c0 2023-10-06T03:30:14.218 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:14.213+0000 7fed727fc700 1 -- 172.21.15.110:0/2397553195 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed6c014480 con 0x7fed7c06d8c0 2023-10-06T03:30:14.219 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:14.213+0000 7fed7b59e700 1 -- 172.21.15.110:0/2397553195 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed7c0c82c0 con 0x7fed7c06d8c0 2023-10-06T03:30:14.219 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:14.213+0000 7fed727fc700 1 -- 172.21.15.110:0/2397553195 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7fed6c0147f0 con 0x7fed7c06d8c0 2023-10-06T03:30:14.219 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:14.213+0000 7fed7b59e700 1 -- 172.21.15.110:0/2397553195 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed7c0c8d20 con 0x7fed7c06d8c0 2023-10-06T03:30:14.220 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:14.217+0000 7fed727fc700 1 --2- 172.21.15.110:0/2397553195 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fed5c060550 0x7fed5c062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:14.220 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:14.217+0000 7fed7a59c700 1 --2- 172.21.15.110:0/2397553195 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fed5c060550 0x7fed5c062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:14.220 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:14.217+0000 7fed727fc700 1 -- 172.21.15.110:0/2397553195 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6809+0+0 (secure 0 0 0) 0x7fed6c07d590 con 0x7fed7c06d8c0 2023-10-06T03:30:14.220 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:14.217+0000 7fed7a59c700 1 --2- 172.21.15.110:0/2397553195 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fed5c060550 0x7fed5c062a00 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fed6400a010 tx=0x7fed64017000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:30:14.227 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:14.221+0000 7fed727fc700 1 -- 172.21.15.110:0/2397553195 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fed6c04d690 con 0x7fed7c06d8c0 2023-10-06T03:30:14.387 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:14.385+0000 7fed7b59e700 1 -- 172.21.15.110:0/2397553195 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7fed7c070ff0 con 0x7fed7c06d8c0 2023-10-06T03:30:14.388 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:14.385+0000 7fed727fc700 1 -- 172.21.15.110:0/2397553195 <== mon.0 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v28) v1 ==== 103+0+11 (secure 0 0 0) 0x7fed6c050cb0 con 0x7fed7c06d8c0 2023-10-06T03:30:14.388 INFO:teuthology.orchestra.run.smithi110.stdout:pg_num: 32 2023-10-06T03:30:14.391 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:14.385+0000 7fed7b59e700 1 -- 172.21.15.110:0/2397553195 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fed5c060550 msgr2=0x7fed5c062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:14.391 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:14.385+0000 7fed7b59e700 1 --2- 172.21.15.110:0/2397553195 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fed5c060550 0x7fed5c062a00 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fed6400a010 tx=0x7fed64017000 comp rx=0 tx=0).stop 2023-10-06T03:30:14.391 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:14.385+0000 7fed7b59e700 1 -- 172.21.15.110:0/2397553195 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fed7c06d8c0 msgr2=0x7fed7c0c2e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:14.391 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:14.385+0000 7fed7b59e700 1 --2- 172.21.15.110:0/2397553195 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fed7c06d8c0 0x7fed7c0c2e60 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fed6c009e50 tx=0x7fed6c00b040 comp rx=0 tx=0).stop 2023-10-06T03:30:14.391 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:14.385+0000 7fed7b59e700 1 -- 172.21.15.110:0/2397553195 shutdown_connections 2023-10-06T03:30:14.391 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:14.385+0000 7fed7b59e700 1 --2- 172.21.15.110:0/2397553195 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fed5c060550 0x7fed5c062a00 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:14.391 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:14.385+0000 7fed7b59e700 1 --2- 172.21.15.110:0/2397553195 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fed7c15ab20 0x7fed7c0c33a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:14.391 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:14.385+0000 7fed7b59e700 1 --2- 172.21.15.110:0/2397553195 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fed7c06d8c0 0x7fed7c0c2e60 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:14.391 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:14.385+0000 7fed7b59e700 1 --2- 172.21.15.110:0/2397553195 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fed7c0670a0 0x7fed7c0c87e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:14.391 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:14.385+0000 7fed7b59e700 1 -- 172.21.15.110:0/2397553195 >> 172.21.15.110:0/2397553195 conn(0x7fed7c0ccdc0 msgr2=0x7fed7c0b0510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:30:14.391 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:14.385+0000 7fed7b59e700 1 -- 172.21.15.110:0/2397553195 shutdown_connections 2023-10-06T03:30:14.392 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:14.385+0000 7fed7b59e700 1 -- 172.21.15.110:0/2397553195 wait complete. 2023-10-06T03:30:14.402 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-06T03:30:14.504 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:14.501+0000 7f725d847700 1 Processor -- start 2023-10-06T03:30:14.505 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:14.501+0000 7f725d847700 1 -- start start 2023-10-06T03:30:14.506 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:14.501+0000 7f725d847700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f72580af3e0 0x7f72580af7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:14.506 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:14.501+0000 7f725d847700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f72580c8db0 0x7f72580c9250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:14.506 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:14.501+0000 7f725d847700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72580c9790 con 0x7f725806d030 2023-10-06T03:30:14.506 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:14.501+0000 7f725d847700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f725815bad0 con 0x7f72580c8db0 2023-10-06T03:30:14.506 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:14.501+0000 7f725d847700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f725815aec0 con 0x7f72580af3e0 2023-10-06T03:30:14.506 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:14.501+0000 7f72567fc700 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f725806d030 0x7f7258063430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:53290/0 (socket says 172.21.15.110:53290) 2023-10-06T03:30:14.506 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:14.501+0000 7f72567fc700 1 -- 172.21.15.110:0/2635716989 learned_addr learned my addr 172.21.15.110:0/2635716989 (peer_addr_for_me v1:172.21.15.110:0/0) 2023-10-06T03:30:14.506 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:14.501+0000 7f72577fe700 1 --2- 172.21.15.110:0/2635716989 >> v2:172.21.15.192:3300/0 conn(0x7f72580c8db0 0x7f72580c9250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:14.506 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:14.501+0000 7f7256ffd700 1 --2- 172.21.15.110:0/2635716989 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f72580af3e0 0x7f72580af7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:14.506 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:14.501+0000 7f7255ffb700 1 -- 172.21.15.110:0/2635716989 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2050396966 0 0) 0x7f7250001490 con 0x7f725806d030 2023-10-06T03:30:14.507 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:14.501+0000 7f7255ffb700 1 -- 172.21.15.110:0/2635716989 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7244003570 con 0x7f725806d030 2023-10-06T03:30:14.510 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:14.501+0000 7f7255ffb700 1 -- 172.21.15.110:0/2635716989 <== mon.0 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3768069332 0 0) 0x7f7244003570 con 0x7f725806d030 2023-10-06T03:30:14.510 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:14.501+0000 7f7255ffb700 1 -- 172.21.15.110:0/2635716989 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f72580af3e0 msgr2=0x7f72580af7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:30:14.510 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:14.501+0000 7f7255ffb700 1 --2- 172.21.15.110:0/2635716989 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f72580af3e0 0x7f72580af7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:14.510 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:14.501+0000 7f7255ffb700 1 -- 172.21.15.110:0/2635716989 >> v2:172.21.15.192:3300/0 conn(0x7f72580c8db0 msgr2=0x7f72580c9250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:30:14.510 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:14.501+0000 7f7255ffb700 1 --2- 172.21.15.110:0/2635716989 >> v2:172.21.15.192:3300/0 conn(0x7f72580c8db0 0x7f72580c9250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:14.510 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:14.501+0000 7f7255ffb700 1 -- 172.21.15.110:0/2635716989 --> v1:172.21.15.110:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f725815b1a0 con 0x7f725806d030 2023-10-06T03:30:14.510 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:14.501+0000 7f7255ffb700 1 -- 172.21.15.110:0/2635716989 <== mon.0 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 118603703 0 0) 0x7f7250001980 con 0x7f725806d030 2023-10-06T03:30:14.510 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:14.501+0000 7f7255ffb700 1 -- 172.21.15.110:0/2635716989 >> v1:172.21.15.110:6789/0 conn(0x7f725806d030 legacy=0x7f7258063430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:14.510 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:14.501+0000 7f7255ffb700 1 --2- 172.21.15.110:0/2635716989 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7244003a60 0x7f7244003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:14.510 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:14.501+0000 7f7255ffb700 1 --2- 172.21.15.110:0/2635716989 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f72580af3e0 0x7f7244004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:14.511 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:14.501+0000 7f7255ffb700 1 --2- 172.21.15.110:0/2635716989 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7244004d50 0x7f72440051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:14.511 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:14.501+0000 7f7255ffb700 1 -- 172.21.15.110:0/2635716989 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f725815bad0 con 0x7f7244003a60 2023-10-06T03:30:14.511 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:14.501+0000 7f7255ffb700 1 -- 172.21.15.110:0/2635716989 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f725815aec0 con 0x7f7244004d50 2023-10-06T03:30:14.511 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:14.501+0000 7f7255ffb700 1 -- 172.21.15.110:0/2635716989 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f724400bae0 con 0x7f72580af3e0 2023-10-06T03:30:14.511 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:14.501+0000 7f7256ffd700 1 --2- 172.21.15.110:0/2635716989 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7244003a60 0x7f7244003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:14.511 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:14.501+0000 7f72567fc700 1 --2- 172.21.15.110:0/2635716989 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f72580af3e0 0x7f7244004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:14.511 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:14.501+0000 7f72577fe700 1 --2- 172.21.15.110:0/2635716989 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7244004d50 0x7f72440051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:14.511 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:14.505+0000 7f7256ffd700 1 -- 172.21.15.110:0/2635716989 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f72580af3e0 msgr2=0x7f7244004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:30:14.511 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:14.505+0000 7f7256ffd700 1 --2- 172.21.15.110:0/2635716989 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f72580af3e0 0x7f7244004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:14.511 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:14.505+0000 7f7256ffd700 1 -- 172.21.15.110:0/2635716989 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7244004d50 msgr2=0x7f72440051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:30:14.511 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:14.505+0000 7f7256ffd700 1 --2- 172.21.15.110:0/2635716989 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7244004d50 0x7f72440051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:14.511 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:14.505+0000 7f7256ffd700 1 -- 172.21.15.110:0/2635716989 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7244003570 con 0x7f7244003a60 2023-10-06T03:30:14.512 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:14.505+0000 7f7256ffd700 1 --2- 172.21.15.110:0/2635716989 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7244003a60 0x7f7244003e70 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f7240009c00 tx=0x7f72400091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ee8309d3fff0762 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:30:14.512 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:14.505+0000 7f7255ffb700 1 -- 172.21.15.110:0/2635716989 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f724000e710 con 0x7f7244003a60 2023-10-06T03:30:14.512 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:14.505+0000 7f7255ffb700 1 -- 172.21.15.110:0/2635716989 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f724000e870 con 0x7f7244003a60 2023-10-06T03:30:14.512 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:14.505+0000 7f7255ffb700 1 -- 172.21.15.110:0/2635716989 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f724000ec30 con 0x7f7244003a60 2023-10-06T03:30:14.512 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:14.505+0000 7f725d847700 1 -- 172.21.15.110:0/2635716989 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7244003a60 msgr2=0x7f7244003e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:14.512 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:14.505+0000 7f725d847700 1 --2- 172.21.15.110:0/2635716989 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7244003a60 0x7f7244003e70 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f7240009c00 tx=0x7f72400091a0 comp rx=0 tx=0).stop 2023-10-06T03:30:14.512 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:14.505+0000 7f72577fe700 1 -- 172.21.15.110:0/2635716989 reap_dead start 2023-10-06T03:30:14.512 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:14.505+0000 7f725d847700 1 -- 172.21.15.110:0/2635716989 shutdown_connections 2023-10-06T03:30:14.512 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:14.505+0000 7f725d847700 1 -- 172.21.15.110:0/2635716989 >> 172.21.15.110:0/2635716989 conn(0x7f7258095a60 msgr2=0x7f72580670a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:30:14.512 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:14.505+0000 7f725d847700 1 -- 172.21.15.110:0/2635716989 shutdown_connections 2023-10-06T03:30:14.512 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:14.505+0000 7f725d847700 1 -- 172.21.15.110:0/2635716989 wait complete. 2023-10-06T03:30:14.515 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:14.509+0000 7f725d847700 1 Processor -- start 2023-10-06T03:30:14.515 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:14.509+0000 7f725d847700 1 -- start start 2023-10-06T03:30:14.515 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:14.509+0000 7f725d847700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7244003a60 0x7f7258152b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:14.515 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:14.509+0000 7f725d847700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f725814de60 0x7f7258155b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:14.515 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:14.509+0000 7f725d847700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f725814e2d0 0x7f7258156090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:14.515 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:14.509+0000 7f725d847700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7258157880 con 0x7f725814de60 2023-10-06T03:30:14.515 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:14.509+0000 7f725d847700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7258153f80 con 0x7f725814e2d0 2023-10-06T03:30:14.515 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:14.509+0000 7f725d847700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7258158b70 con 0x7f7244003a60 2023-10-06T03:30:14.515 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:14.509+0000 7f72567fc700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f725814de60 0x7f7258155b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:14.515 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:14.509+0000 7f72567fc700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f725814de60 0x7f7258155b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:52198/0 (socket says 172.21.15.110:52198) 2023-10-06T03:30:14.516 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:14.509+0000 7f7256ffd700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7244003a60 0x7f7258152b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:14.516 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:14.509+0000 7f72567fc700 1 -- 172.21.15.110:0/2956245975 learned_addr learned my addr 172.21.15.110:0/2956245975 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:30:14.516 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:14.509+0000 7f72577fe700 1 --2- 172.21.15.110:0/2956245975 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f725814e2d0 0x7f7258156090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:14.516 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:14.509+0000 7f72567fc700 1 -- 172.21.15.110:0/2956245975 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7244003a60 msgr2=0x7f7258152b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:14.516 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:14.509+0000 7f72567fc700 1 --2- 172.21.15.110:0/2956245975 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7244003a60 0x7f7258152b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:14.516 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:14.509+0000 7f72567fc700 1 -- 172.21.15.110:0/2956245975 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f725814e2d0 msgr2=0x7f7258156090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:14.516 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:14.509+0000 7f72567fc700 1 --2- 172.21.15.110:0/2956245975 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f725814e2d0 0x7f7258156090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:14.517 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:14.509+0000 7f72567fc700 1 -- 172.21.15.110:0/2956245975 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7258157ad0 con 0x7f725814de60 2023-10-06T03:30:14.517 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:14.509+0000 7f7256ffd700 1 --2- 172.21.15.110:0/2956245975 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7244003a60 0x7f7258152b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-06T03:30:14.517 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:14.509+0000 7f72567fc700 1 --2- 172.21.15.110:0/2956245975 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f725814de60 0x7f7258155b50 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f725000ace0 tx=0x7f725000a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:30:14.517 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:14.509+0000 7f724effd700 1 -- 172.21.15.110:0/2956245975 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7250007830 con 0x7f725814de60 2023-10-06T03:30:14.517 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:14.509+0000 7f724effd700 1 -- 172.21.15.110:0/2956245975 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72500079d0 con 0x7f725814de60 2023-10-06T03:30:14.517 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:14.509+0000 7f725d847700 1 -- 172.21.15.110:0/2956245975 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7258157db0 con 0x7f725814de60 2023-10-06T03:30:14.517 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:14.509+0000 7f724effd700 1 -- 172.21.15.110:0/2956245975 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7250013640 con 0x7f725814de60 2023-10-06T03:30:14.517 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:14.509+0000 7f725d847700 1 -- 172.21.15.110:0/2956245975 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7258158370 con 0x7f725814de60 2023-10-06T03:30:14.518 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:14.513+0000 7f725d847700 1 -- 172.21.15.110:0/2956245975 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7258063f50 con 0x7f725814de60 2023-10-06T03:30:14.518 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:14.513+0000 7f724effd700 1 -- 172.21.15.110:0/2956245975 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7f72500139f0 con 0x7f725814de60 2023-10-06T03:30:14.518 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:14.513+0000 7f724effd700 1 --2- 172.21.15.110:0/2956245975 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f7238060550 0x7f7238062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:14.518 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:14.513+0000 7f7256ffd700 1 --2- 172.21.15.110:0/2956245975 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f7238060550 0x7f7238062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:14.518 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:14.513+0000 7f724effd700 1 -- 172.21.15.110:0/2956245975 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6809+0+0 (secure 0 0 0) 0x7f725007d780 con 0x7f725814de60 2023-10-06T03:30:14.519 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:14.513+0000 7f7256ffd700 1 --2- 172.21.15.110:0/2956245975 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f7238060550 0x7f7238062a00 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f72400091e0 tx=0x7f724001a000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:30:14.526 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:14.521+0000 7f724effd700 1 -- 172.21.15.110:0/2956245975 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f725004d920 con 0x7f725814de60 2023-10-06T03:30:14.693 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:14.689+0000 7f725d847700 1 -- 172.21.15.110:0/2956245975 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f72580c9790 con 0x7f725814de60 2023-10-06T03:30:14.695 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:14.689+0000 7f724effd700 1 -- 172.21.15.110:0/2956245975 <== mon.0 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v28) v1 ==== 74+0+18755 (secure 0 0 0) 0x7f72580c9790 con 0x7f725814de60 2023-10-06T03:30:14.696 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-10-06T03:30:14.696 INFO:teuthology.orchestra.run.smithi110.stdout:{"epoch":28,"fsid":"93b8f6ac-4c92-4b45-b8e4-b59617024aa2","created":"2023-10-06T03:29:10.519583+0000","modified":"2023-10-06T03:29:38.951740+0000","last_up_change":"2023-10-06T03:29:21.960511+0000","last_in_change":"2023-10-06T03:29:12.560436+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-06T03:29:20.703743+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2023-10-06T03:29:28.307281+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2023-10-06T03:29:32.247251+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data","create_time":"2023-10-06T03:29:32.880086+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}}},{"pool":5,"pool_name":"cephfs_data_ec","create_time":"2023-10-06T03:29:36.613779+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"28","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"5866eb6c-2adc-46e4-be7a-e5bcda0f4065","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6824","nonce":33394},{"type":"v1","addr":"172.21.15.110:6825","nonce":33394}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6826","nonce":33394},{"type":"v1","addr":"172.21.15.110:6827","nonce":33394}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6830","nonce":33394},{"type":"v1","addr":"172.21.15.110:6831","nonce":33394}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6828","nonce":33394},{"type":"v1","addr":"172.21.15.110:6829","nonce":33394}]},"public_addr":"172.21.15.110:6825/33394","cluster_addr":"172.21.15.110:6827/33394","heartbeat_back_addr":"172.21.15.110:6831/33394","heartbeat_front_addr":"172.21.15.110:6829/33394","state":["exists","up"]},{"osd":1,"uuid":"a38de118-062b-4a05-9782-2ceddf6ff4df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6800","nonce":33396},{"type":"v1","addr":"172.21.15.110:6801","nonce":33396}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6802","nonce":33396},{"type":"v1","addr":"172.21.15.110:6803","nonce":33396}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6806","nonce":33396},{"type":"v1","addr":"172.21.15.110:6807","nonce":33396}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6804","nonce":33396},{"type":"v1","addr":"172.21.15.110:6805","nonce":33396}]},"public_addr":"172.21.15.110:6801/33396","cluster_addr":"172.21.15.110:6803/33396","heartbeat_back_addr":"172.21.15.110:6807/33396","heartbeat_front_addr":"172.21.15.110:6805/33396","state":["exists","up"]},{"osd":2,"uuid":"56e39be2-3407-4b69-877e-cf16617b06d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6816","nonce":33397},{"type":"v1","addr":"172.21.15.110:6817","nonce":33397}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6818","nonce":33397},{"type":"v1","addr":"172.21.15.110:6819","nonce":33397}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6822","nonce":33397},{"type":"v1","addr":"172.21.15.110:6823","nonce":33397}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6820","nonce":33397},{"type":"v1","addr":"172.21.15.110:6821","nonce":33397}]},"public_addr":"172.21.15.110:6817/33397","cluster_addr":"172.21.15.110:6819/33397","heartbeat_back_addr":"172.21.15.110:6823/33397","heartbeat_front_addr":"172.21.15.110:6821/33397","state":["exists","up"]},{"osd":3,"uuid":"01c82d13-c1d4-4bde-b55e-3e0089b40115","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6808","nonce":33395},{"type":"v1","addr":"172.21.15.110:6809","nonce":33395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6810","nonce":33395},{"type":"v1","addr":"172.21.15.110:6811","nonce":33395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6814","nonce":33395},{"type":"v1","addr":"172.21.15.110:6815","nonce":33395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6812","nonce":33395},{"type":"v1","addr":"172.21.15.110:6813","nonce":33395}]},"public_addr":"172.21.15.110:6809/33395","cluster_addr":"172.21.15.110:6811/33395","heartbeat_back_addr":"172.21.15.110:6815/33395","heartbeat_front_addr":"172.21.15.110:6813/33395","state":["exists","up"]},{"osd":4,"uuid":"86983430-5fa3-4d17-998f-03e2cd843237","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":33683},{"type":"v1","addr":"172.21.15.192:6817","nonce":33683}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":33683},{"type":"v1","addr":"172.21.15.192:6819","nonce":33683}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":33683},{"type":"v1","addr":"172.21.15.192:6823","nonce":33683}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6820","nonce":33683},{"type":"v1","addr":"172.21.15.192:6821","nonce":33683}]},"public_addr":"172.21.15.192:6817/33683","cluster_addr":"172.21.15.192:6819/33683","heartbeat_back_addr":"172.21.15.192:6823/33683","heartbeat_front_addr":"172.21.15.192:6821/33683","state":["exists","up"]},{"osd":5,"uuid":"6945257f-821d-4619-a63f-ebe857eba6f4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":33684},{"type":"v1","addr":"172.21.15.192:6801","nonce":33684}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":33684},{"type":"v1","addr":"172.21.15.192:6803","nonce":33684}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":33684},{"type":"v1","addr":"172.21.15.192:6807","nonce":33684}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6804","nonce":33684},{"type":"v1","addr":"172.21.15.192:6805","nonce":33684}]},"public_addr":"172.21.15.192:6801/33684","cluster_addr":"172.21.15.192:6803/33684","heartbeat_back_addr":"172.21.15.192:6807/33684","heartbeat_front_addr":"172.21.15.192:6805/33684","state":["exists","up"]},{"osd":6,"uuid":"7854d1ae-7060-4780-8f97-73cea5e7a12f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6824","nonce":33686},{"type":"v1","addr":"172.21.15.192:6825","nonce":33686}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":33686},{"type":"v1","addr":"172.21.15.192:6827","nonce":33686}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":33686},{"type":"v1","addr":"172.21.15.192:6831","nonce":33686}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6828","nonce":33686},{"type":"v1","addr":"172.21.15.192:6829","nonce":33686}]},"public_addr":"172.21.15.192:6825/33686","cluster_addr":"172.21.15.192:6827/33686","heartbeat_back_addr":"172.21.15.192:6831/33686","heartbeat_front_addr":"172.21.15.192:6829/33686","state":["exists","up"]},{"osd":7,"uuid":"f1ebdfd5-b859-482b-9f97-c18c4a26f445","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6808","nonce":33685},{"type":"v1","addr":"172.21.15.192:6809","nonce":33685}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":33685},{"type":"v1","addr":"172.21.15.192:6811","nonce":33685}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":33685},{"type":"v1","addr":"172.21.15.192:6815","nonce":33685}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6812","nonce":33685},{"type":"v1","addr":"172.21.15.192:6813","nonce":33685}]},"public_addr":"172.21.15.192:6809/33685","cluster_addr":"172.21.15.192:6811/33685","heartbeat_back_addr":"172.21.15.192:6815/33685","heartbeat_front_addr":"172.21.15.192:6813/33685","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T03:29:18.002048+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T03:29:17.478576+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T03:29:18.034944+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T03:29:17.628513+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T03:29:18.047338+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T03:29:17.772349+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T03:29:17.883958+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T03:29:18.151838+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-06T03:30:14.698 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:14.693+0000 7f725d847700 1 -- 172.21.15.110:0/2956245975 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f7238060550 msgr2=0x7f7238062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:14.698 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:14.693+0000 7f725d847700 1 --2- 172.21.15.110:0/2956245975 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f7238060550 0x7f7238062a00 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f72400091e0 tx=0x7f724001a000 comp rx=0 tx=0).stop 2023-10-06T03:30:14.698 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:14.693+0000 7f725d847700 1 -- 172.21.15.110:0/2956245975 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f725814de60 msgr2=0x7f7258155b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:14.698 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:14.693+0000 7f725d847700 1 --2- 172.21.15.110:0/2956245975 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f725814de60 0x7f7258155b50 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f725000ace0 tx=0x7f725000a1a0 comp rx=0 tx=0).stop 2023-10-06T03:30:14.698 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:14.693+0000 7f725d847700 1 -- 172.21.15.110:0/2956245975 shutdown_connections 2023-10-06T03:30:14.698 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:14.693+0000 7f725d847700 1 --2- 172.21.15.110:0/2956245975 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f7238060550 0x7f7238062a00 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:14.698 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:14.693+0000 7f725d847700 1 --2- 172.21.15.110:0/2956245975 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f725814e2d0 0x7f7258156090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:14.698 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:14.693+0000 7f725d847700 1 --2- 172.21.15.110:0/2956245975 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f725814de60 0x7f7258155b50 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:14.699 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:14.693+0000 7f725d847700 1 --2- 172.21.15.110:0/2956245975 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7244003a60 0x7f7258152b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:14.699 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:14.693+0000 7f725d847700 1 -- 172.21.15.110:0/2956245975 >> 172.21.15.110:0/2956245975 conn(0x7f7258095a60 msgr2=0x7f725806e890 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:30:14.699 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:14.693+0000 7f725d847700 1 -- 172.21.15.110:0/2956245975 shutdown_connections 2023-10-06T03:30:14.699 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:14.693+0000 7f725d847700 1 -- 172.21.15.110:0/2956245975 wait complete. 2023-10-06T03:30:14.710 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-06T03:29:20.703743+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-10-06T03:29:28.307281+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 3, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-06T03:29:32.247251+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 4, 'pool_name': 'cephfs_data', 'create_time': '2023-10-06T03:29:32.880086+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}, {'pool': 5, 'pool_name': 'cephfs_data_ec', 'create_time': '2023-10-06T03:29:36.613779+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '28', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-10-06T03:30:14.710 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-06T03:30:14.824 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:14.821+0000 7fadfb59e700 1 Processor -- start 2023-10-06T03:30:14.824 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:14.821+0000 7fadfb59e700 1 -- start start 2023-10-06T03:30:14.826 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:14.821+0000 7fadfb59e700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fadfc0aef40 0x7fadfc0ce200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:14.826 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:14.821+0000 7fadfb59e700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fadfc0cb1b0 0x7fadfc15cb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:14.826 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:14.821+0000 7fadfb59e700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fadfc15d040 0x7fadfc06b7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:14.826 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:14.821+0000 7fadfb59e700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fadfc15d4c0 con 0x7fadfc15d040 2023-10-06T03:30:14.827 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:14.821+0000 7fadfb59e700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fadfc0cf5f0 con 0x7fadfc0cb1b0 2023-10-06T03:30:14.827 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:14.821+0000 7fadfb59e700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fadfc0ce740 con 0x7fadfc0aef40 2023-10-06T03:30:14.827 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:14.821+0000 7fadfad9d700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fadfc15d040 0x7fadfc06b7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:14.827 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:14.821+0000 7fadfad9d700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fadfc15d040 0x7fadfc06b7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:52206/0 (socket says 172.21.15.110:52206) 2023-10-06T03:30:14.827 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:14.821+0000 7fadfad9d700 1 -- 172.21.15.110:0/970903578 learned_addr learned my addr 172.21.15.110:0/970903578 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:30:14.827 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:14.821+0000 7fadf9d9b700 1 --2- 172.21.15.110:0/970903578 >> v2:172.21.15.192:3300/0 conn(0x7fadfc0cb1b0 0x7fadfc15cb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:14.827 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:14.821+0000 7fadfa59c700 1 --2- 172.21.15.110:0/970903578 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fadfc0aef40 0x7fadfc0ce200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:14.829 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:14.821+0000 7fadfad9d700 1 -- 172.21.15.110:0/970903578 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fadfc0aef40 msgr2=0x7fadfc0ce200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:30:14.829 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:14.821+0000 7fadfad9d700 1 --2- 172.21.15.110:0/970903578 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fadfc0aef40 0x7fadfc0ce200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:14.829 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:14.821+0000 7fadfad9d700 1 -- 172.21.15.110:0/970903578 >> v2:172.21.15.192:3300/0 conn(0x7fadfc0cb1b0 msgr2=0x7fadfc15cb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:30:14.830 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:14.821+0000 7fadfad9d700 1 --2- 172.21.15.110:0/970903578 >> v2:172.21.15.192:3300/0 conn(0x7fadfc0cb1b0 0x7fadfc15cb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:14.830 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:14.821+0000 7fadfad9d700 1 -- 172.21.15.110:0/970903578 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fadfc0ce9e0 con 0x7fadfc15d040 2023-10-06T03:30:14.830 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:14.821+0000 7fadfad9d700 1 --2- 172.21.15.110:0/970903578 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fadfc15d040 0x7fadfc06b7c0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fadfc06ef20 tx=0x7fadf4009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c46707d37eea56 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:30:14.830 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:14.825+0000 7fadf959a700 1 -- 172.21.15.110:0/970903578 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadf400e070 con 0x7fadfc15d040 2023-10-06T03:30:14.830 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:14.825+0000 7fadf959a700 1 -- 172.21.15.110:0/970903578 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fadf400b970 con 0x7fadfc15d040 2023-10-06T03:30:14.830 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:14.825+0000 7fadf959a700 1 -- 172.21.15.110:0/970903578 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadf40092e0 con 0x7fadfc15d040 2023-10-06T03:30:14.830 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:14.825+0000 7fadfb59e700 1 -- 172.21.15.110:0/970903578 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fadfc15d040 msgr2=0x7fadfc06b7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:14.830 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:14.825+0000 7fadfb59e700 1 --2- 172.21.15.110:0/970903578 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fadfc15d040 0x7fadfc06b7c0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fadfc06ef20 tx=0x7fadf4009840 comp rx=0 tx=0).stop 2023-10-06T03:30:14.830 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:14.825+0000 7fadfb59e700 1 -- 172.21.15.110:0/970903578 shutdown_connections 2023-10-06T03:30:14.831 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:14.825+0000 7fadfb59e700 1 --2- 172.21.15.110:0/970903578 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fadfc15d040 0x7fadfc06b7c0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:14.831 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:14.825+0000 7fadfb59e700 1 --2- 172.21.15.110:0/970903578 >> v2:172.21.15.192:3300/0 conn(0x7fadfc0cb1b0 0x7fadfc15cb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:14.831 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:14.825+0000 7fadfb59e700 1 --2- 172.21.15.110:0/970903578 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fadfc0aef40 0x7fadfc0ce200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:14.831 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:14.825+0000 7fadfb59e700 1 -- 172.21.15.110:0/970903578 >> 172.21.15.110:0/970903578 conn(0x7fadfc0cfec0 msgr2=0x7fadfc0d02c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:30:14.831 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:14.825+0000 7fadfb59e700 1 -- 172.21.15.110:0/970903578 shutdown_connections 2023-10-06T03:30:14.831 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:14.825+0000 7fadfb59e700 1 -- 172.21.15.110:0/970903578 wait complete. 2023-10-06T03:30:14.833 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:14.825+0000 7fadfb59e700 1 Processor -- start 2023-10-06T03:30:14.833 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:14.825+0000 7fadfb59e700 1 -- start start 2023-10-06T03:30:14.834 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:14.829+0000 7fadfb59e700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fadfc0aef40 0x7fadfc16be10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:14.834 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:14.829+0000 7fadfb59e700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fadfc0cb1b0 0x7fadfc0c8d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:14.834 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:14.829+0000 7fadfb59e700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fadfc15d040 0x7fadfc0c92a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:14.834 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:14.829+0000 7fadfb59e700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fadfc0c6f70 con 0x7fadfc0aef40 2023-10-06T03:30:14.834 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:14.829+0000 7fadfb59e700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fadfc0c7dd0 con 0x7fadfc0cb1b0 2023-10-06T03:30:14.834 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:14.829+0000 7fadfb59e700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fadfc16c350 con 0x7fadfc15d040 2023-10-06T03:30:14.834 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:14.829+0000 7fadfa59c700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fadfc0aef40 0x7fadfc16be10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:14.834 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:14.829+0000 7fadfa59c700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fadfc0aef40 0x7fadfc16be10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:52222/0 (socket says 172.21.15.110:52222) 2023-10-06T03:30:14.834 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:14.829+0000 7fadfa59c700 1 -- 172.21.15.110:0/2928716579 learned_addr learned my addr 172.21.15.110:0/2928716579 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:30:14.835 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:14.829+0000 7fadf9d9b700 1 --2- 172.21.15.110:0/2928716579 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fadfc0cb1b0 0x7fadfc0c8d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:14.835 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:14.829+0000 7fadfad9d700 1 --2- 172.21.15.110:0/2928716579 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fadfc15d040 0x7fadfc0c92a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:14.835 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:14.829+0000 7fadfa59c700 1 -- 172.21.15.110:0/2928716579 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fadfc15d040 msgr2=0x7fadfc0c92a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:14.835 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:14.829+0000 7fadfa59c700 1 --2- 172.21.15.110:0/2928716579 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fadfc15d040 0x7fadfc0c92a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:14.835 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:14.829+0000 7fadfa59c700 1 -- 172.21.15.110:0/2928716579 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fadfc0cb1b0 msgr2=0x7fadfc0c8d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:14.835 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:14.829+0000 7fadfa59c700 1 --2- 172.21.15.110:0/2928716579 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fadfc0cb1b0 0x7fadfc0c8d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:14.835 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:14.829+0000 7fadfa59c700 1 -- 172.21.15.110:0/2928716579 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fadf4009ae0 con 0x7fadfc0aef40 2023-10-06T03:30:14.835 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:14.829+0000 7fadfa59c700 1 --2- 172.21.15.110:0/2928716579 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fadfc0aef40 0x7fadfc16be10 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fadec00ec80 tx=0x7fadec0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:30:14.835 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:14.829+0000 7fadf27fc700 1 -- 172.21.15.110:0/2928716579 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadec00c850 con 0x7fadfc0aef40 2023-10-06T03:30:14.835 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:14.829+0000 7fadfb59e700 1 -- 172.21.15.110:0/2928716579 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fadfc0c71c0 con 0x7fadfc0aef40 2023-10-06T03:30:14.836 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:14.829+0000 7fadf27fc700 1 -- 172.21.15.110:0/2928716579 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fadec00c9d0 con 0x7fadfc0aef40 2023-10-06T03:30:14.836 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:14.829+0000 7fadf27fc700 1 -- 172.21.15.110:0/2928716579 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadec00cd90 con 0x7fadfc0aef40 2023-10-06T03:30:14.836 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:14.829+0000 7fadfb59e700 1 -- 172.21.15.110:0/2928716579 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fadfc0c7780 con 0x7fadfc0aef40 2023-10-06T03:30:14.836 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:14.829+0000 7fadfb59e700 1 -- 172.21.15.110:0/2928716579 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fadfc070ff0 con 0x7fadfc0aef40 2023-10-06T03:30:14.836 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:14.833+0000 7fadf27fc700 1 -- 172.21.15.110:0/2928716579 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7fadec013680 con 0x7fadfc0aef40 2023-10-06T03:30:14.837 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:14.833+0000 7fadf27fc700 1 --2- 172.21.15.110:0/2928716579 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7faddc060550 0x7faddc062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:14.837 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:14.833+0000 7fadf27fc700 1 -- 172.21.15.110:0/2928716579 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6809+0+0 (secure 0 0 0) 0x7fadec07d720 con 0x7fadfc0aef40 2023-10-06T03:30:14.837 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:14.833+0000 7fadf9d9b700 1 --2- 172.21.15.110:0/2928716579 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7faddc060550 0x7faddc062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:14.838 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:14.833+0000 7fadf9d9b700 1 --2- 172.21.15.110:0/2928716579 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7faddc060550 0x7faddc062a00 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fadfc15bba0 tx=0x7fade40091a0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:30:14.842 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:14.837+0000 7fadf27fc700 1 -- 172.21.15.110:0/2928716579 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fadec04d820 con 0x7fadfc0aef40 2023-10-06T03:30:15.002 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:14.997+0000 7fadfb59e700 1 -- 172.21.15.110:0/2928716579 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fadfc06d8d0 con 0x7fadfc0aef40 2023-10-06T03:30:15.003 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:14.997+0000 7fadf27fc700 1 -- 172.21.15.110:0/2928716579 <== mon.0 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v28) v1 ==== 93+0+10 (secure 0 0 0) 0x7fadec050e40 con 0x7fadfc0aef40 2023-10-06T03:30:15.003 INFO:teuthology.orchestra.run.smithi110.stdout:pg_num: 1 2023-10-06T03:30:15.005 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:15.001+0000 7fadfb59e700 1 -- 172.21.15.110:0/2928716579 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7faddc060550 msgr2=0x7faddc062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:15.005 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:15.001+0000 7fadfb59e700 1 --2- 172.21.15.110:0/2928716579 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7faddc060550 0x7faddc062a00 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fadfc15bba0 tx=0x7fade40091a0 comp rx=0 tx=0).stop 2023-10-06T03:30:15.006 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:15.001+0000 7fadfb59e700 1 -- 172.21.15.110:0/2928716579 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fadfc0aef40 msgr2=0x7fadfc16be10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:15.006 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:15.001+0000 7fadfb59e700 1 --2- 172.21.15.110:0/2928716579 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fadfc0aef40 0x7fadfc16be10 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fadec00ec80 tx=0x7fadec0091a0 comp rx=0 tx=0).stop 2023-10-06T03:30:15.006 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:15.001+0000 7fadfb59e700 1 -- 172.21.15.110:0/2928716579 shutdown_connections 2023-10-06T03:30:15.006 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:15.001+0000 7fadfb59e700 1 --2- 172.21.15.110:0/2928716579 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7faddc060550 0x7faddc062a00 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:15.006 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:15.001+0000 7fadfb59e700 1 --2- 172.21.15.110:0/2928716579 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fadfc15d040 0x7fadfc0c92a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:15.006 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:15.001+0000 7fadfb59e700 1 --2- 172.21.15.110:0/2928716579 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fadfc0cb1b0 0x7fadfc0c8d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:15.006 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:15.001+0000 7fadfb59e700 1 --2- 172.21.15.110:0/2928716579 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fadfc0aef40 0x7fadfc16be10 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:15.006 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:15.001+0000 7fadfb59e700 1 -- 172.21.15.110:0/2928716579 >> 172.21.15.110:0/2928716579 conn(0x7fadfc0cfec0 msgr2=0x7fadfc158930 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:30:15.006 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:15.001+0000 7fadfb59e700 1 -- 172.21.15.110:0/2928716579 shutdown_connections 2023-10-06T03:30:15.006 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:15.001+0000 7fadfb59e700 1 -- 172.21.15.110:0/2928716579 wait complete. 2023-10-06T03:30:15.017 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-10-06T03:30:15.124 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:15.121+0000 7fe3b359e700 1 Processor -- start 2023-10-06T03:30:15.124 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:15.121+0000 7fe3b359e700 1 -- start start 2023-10-06T03:30:15.126 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:15.121+0000 7fe3b359e700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe3b415d2a0 0x7fe3b406cab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:15.126 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:15.121+0000 7fe3b359e700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fe3b40c97b0 0x7fe3b406bc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:15.126 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:15.121+0000 7fe3b359e700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3b4069ef0 con 0x7fe3b406b7c0 2023-10-06T03:30:15.126 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:15.121+0000 7fe3b359e700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe3b406c150 con 0x7fe3b40c97b0 2023-10-06T03:30:15.127 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:15.121+0000 7fe3b359e700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3b406a140 con 0x7fe3b415d2a0 2023-10-06T03:30:15.127 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:15.121+0000 7fe3b1d9b700 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7fe3b406b7c0 0x7fe3b40af1f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:53292/0 (socket says 172.21.15.110:53292) 2023-10-06T03:30:15.127 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:15.121+0000 7fe3b1d9b700 1 -- 172.21.15.110:0/2217300504 learned_addr learned my addr 172.21.15.110:0/2217300504 (peer_addr_for_me v1:172.21.15.110:0/0) 2023-10-06T03:30:15.127 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:15.121+0000 7fe3b259c700 1 --2- 172.21.15.110:0/2217300504 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe3b415d2a0 0x7fe3b406cab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:15.127 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:15.121+0000 7fe3b2d9d700 1 --2- 172.21.15.110:0/2217300504 >> v2:172.21.15.192:3300/0 conn(0x7fe3b40c97b0 0x7fe3b406bc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:15.127 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:15.121+0000 7fe3b159a700 1 -- 172.21.15.110:0/2217300504 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3059350602 0 0) 0x7fe3b4069ef0 con 0x7fe3b406b7c0 2023-10-06T03:30:15.127 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:15.121+0000 7fe3b159a700 1 -- 172.21.15.110:0/2217300504 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe3a0003570 con 0x7fe3b406b7c0 2023-10-06T03:30:15.127 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:15.121+0000 7fe3b259c700 1 -- 172.21.15.110:0/2217300504 >> v2:172.21.15.192:3300/0 conn(0x7fe3b40c97b0 msgr2=0x7fe3b406bc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:30:15.127 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:15.121+0000 7fe3b259c700 1 --2- 172.21.15.110:0/2217300504 >> v2:172.21.15.192:3300/0 conn(0x7fe3b40c97b0 0x7fe3b406bc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:15.127 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:15.121+0000 7fe3b259c700 1 -- 172.21.15.110:0/2217300504 >> v1:172.21.15.110:6789/0 conn(0x7fe3b406b7c0 legacy=0x7fe3b40af1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:15.127 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:15.121+0000 7fe3b259c700 1 -- 172.21.15.110:0/2217300504 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3b406a420 con 0x7fe3b415d2a0 2023-10-06T03:30:15.128 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:15.121+0000 7fe3b259c700 1 --2- 172.21.15.110:0/2217300504 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe3b415d2a0 0x7fe3b406cab0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fe3a40098e0 tx=0x7fe3a40091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6994ff3cc3af7188 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:30:15.129 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:15.121+0000 7fe3b159a700 1 -- 172.21.15.110:0/2217300504 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3a4011630 con 0x7fe3b415d2a0 2023-10-06T03:30:15.129 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:15.121+0000 7fe3b159a700 1 -- 172.21.15.110:0/2217300504 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3a4011790 con 0x7fe3b415d2a0 2023-10-06T03:30:15.129 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:15.121+0000 7fe3b159a700 1 -- 172.21.15.110:0/2217300504 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3a4011b10 con 0x7fe3b415d2a0 2023-10-06T03:30:15.129 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:15.125+0000 7fe3b359e700 1 -- 172.21.15.110:0/2217300504 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe3b415d2a0 msgr2=0x7fe3b406cab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:15.129 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:15.125+0000 7fe3b359e700 1 --2- 172.21.15.110:0/2217300504 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe3b415d2a0 0x7fe3b406cab0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fe3a40098e0 tx=0x7fe3a40091a0 comp rx=0 tx=0).stop 2023-10-06T03:30:15.129 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:15.125+0000 7fe3b359e700 1 -- 172.21.15.110:0/2217300504 shutdown_connections 2023-10-06T03:30:15.129 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:15.125+0000 7fe3b359e700 1 --2- 172.21.15.110:0/2217300504 >> v2:172.21.15.192:3300/0 conn(0x7fe3b40c97b0 0x7fe3b406bc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:15.129 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:15.125+0000 7fe3b359e700 1 --2- 172.21.15.110:0/2217300504 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe3b415d2a0 0x7fe3b406cab0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:15.129 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:15.125+0000 7fe3b359e700 1 -- 172.21.15.110:0/2217300504 >> 172.21.15.110:0/2217300504 conn(0x7fe3b40cbac0 msgr2=0x7fe3b40cbec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:30:15.129 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:15.125+0000 7fe3b359e700 1 -- 172.21.15.110:0/2217300504 shutdown_connections 2023-10-06T03:30:15.129 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:15.125+0000 7fe3b359e700 1 -- 172.21.15.110:0/2217300504 wait complete. 2023-10-06T03:30:15.130 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:15.125+0000 7fe3b359e700 1 Processor -- start 2023-10-06T03:30:15.130 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:15.125+0000 7fe3b359e700 1 -- start start 2023-10-06T03:30:15.133 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:15.129+0000 7fe3b359e700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe3b406b7c0 0x7fe3b4159db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:15.133 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:15.129+0000 7fe3b359e700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe3b40c97b0 0x7fe3b40c19d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:15.133 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:15.129+0000 7fe3b359e700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe3b415d2a0 0x7fe3b40c1f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:15.133 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:15.129+0000 7fe3b359e700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3b40c3150 con 0x7fe3b40c97b0 2023-10-06T03:30:15.133 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:15.129+0000 7fe3b359e700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3b40c6250 con 0x7fe3b406b7c0 2023-10-06T03:30:15.133 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:15.129+0000 7fe3b359e700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3b40c3d80 con 0x7fe3b415d2a0 2023-10-06T03:30:15.133 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:15.129+0000 7fe3b1d9b700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe3b40c97b0 0x7fe3b40c19d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:15.134 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:15.129+0000 7fe3b1d9b700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe3b40c97b0 0x7fe3b40c19d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:52226/0 (socket says 172.21.15.110:52226) 2023-10-06T03:30:15.134 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:15.129+0000 7fe3b1d9b700 1 -- 172.21.15.110:0/3071736466 learned_addr learned my addr 172.21.15.110:0/3071736466 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:30:15.134 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:15.129+0000 7fe3b259c700 1 --2- 172.21.15.110:0/3071736466 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe3b406b7c0 0x7fe3b4159db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:15.134 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:15.129+0000 7fe3b2d9d700 1 --2- 172.21.15.110:0/3071736466 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe3b415d2a0 0x7fe3b40c1f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:15.134 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:15.129+0000 7fe3b1d9b700 1 -- 172.21.15.110:0/3071736466 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe3b415d2a0 msgr2=0x7fe3b40c1f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:15.134 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:15.129+0000 7fe3b1d9b700 1 --2- 172.21.15.110:0/3071736466 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe3b415d2a0 0x7fe3b40c1f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:15.134 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:15.129+0000 7fe3b1d9b700 1 -- 172.21.15.110:0/3071736466 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe3b406b7c0 msgr2=0x7fe3b4159db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:15.134 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:15.129+0000 7fe3b1d9b700 1 --2- 172.21.15.110:0/3071736466 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe3b406b7c0 0x7fe3b4159db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:15.134 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:15.129+0000 7fe3b1d9b700 1 -- 172.21.15.110:0/3071736466 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3a4009420 con 0x7fe3b40c97b0 2023-10-06T03:30:15.134 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:15.129+0000 7fe3b1d9b700 1 --2- 172.21.15.110:0/3071736466 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe3b40c97b0 0x7fe3b40c19d0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fe39c0011e0 tx=0x7fe39c007df0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:30:15.134 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:15.129+0000 7fe3aa7fc700 1 -- 172.21.15.110:0/3071736466 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe39c005b30 con 0x7fe3b40c97b0 2023-10-06T03:30:15.135 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:15.129+0000 7fe3aa7fc700 1 -- 172.21.15.110:0/3071736466 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe39c005c90 con 0x7fe3b40c97b0 2023-10-06T03:30:15.135 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:15.129+0000 7fe3b359e700 1 -- 172.21.15.110:0/3071736466 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3b40c2540 con 0x7fe3b40c97b0 2023-10-06T03:30:15.135 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:15.129+0000 7fe3aa7fc700 1 -- 172.21.15.110:0/3071736466 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe39c0072e0 con 0x7fe3b40c97b0 2023-10-06T03:30:15.135 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:15.129+0000 7fe3b359e700 1 -- 172.21.15.110:0/3071736466 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3b40c2b00 con 0x7fe3b40c97b0 2023-10-06T03:30:15.135 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:15.129+0000 7fe3b359e700 1 -- 172.21.15.110:0/3071736466 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3b40c63c0 con 0x7fe3b40c97b0 2023-10-06T03:30:15.135 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:15.133+0000 7fe3aa7fc700 1 -- 172.21.15.110:0/3071736466 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7fe39c007650 con 0x7fe3b40c97b0 2023-10-06T03:30:15.136 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:15.133+0000 7fe3aa7fc700 1 --2- 172.21.15.110:0/3071736466 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fe394060580 0x7fe394062a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:15.136 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:15.133+0000 7fe3aa7fc700 1 -- 172.21.15.110:0/3071736466 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6809+0+0 (secure 0 0 0) 0x7fe39c07c040 con 0x7fe3b40c97b0 2023-10-06T03:30:15.136 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:15.133+0000 7fe3b259c700 1 --2- 172.21.15.110:0/3071736466 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fe394060580 0x7fe394062a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:15.137 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:15.133+0000 7fe3b259c700 1 --2- 172.21.15.110:0/3071736466 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fe394060580 0x7fe394062a30 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fe3a40091e0 tx=0x7fe3a4017000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:30:15.148 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:15.145+0000 7fe3aa7fc700 1 -- 172.21.15.110:0/3071736466 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fe39c049be0 con 0x7fe3b40c97b0 2023-10-06T03:30:15.308 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:15.305+0000 7fe3b359e700 1 -- 172.21.15.110:0/3071736466 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7fe3b4070ff0 con 0x7fe3b40c97b0 2023-10-06T03:30:15.309 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:15.305+0000 7fe3aa7fc700 1 -- 172.21.15.110:0/3071736466 <== mon.0 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v28) v1 ==== 92+0+10 (secure 0 0 0) 0x7fe39c017020 con 0x7fe3b40c97b0 2023-10-06T03:30:15.309 INFO:teuthology.orchestra.run.smithi110.stdout:pg_num: 8 2023-10-06T03:30:15.311 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:15.305+0000 7fe3b359e700 1 -- 172.21.15.110:0/3071736466 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fe394060580 msgr2=0x7fe394062a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:15.311 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:15.305+0000 7fe3b359e700 1 --2- 172.21.15.110:0/3071736466 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fe394060580 0x7fe394062a30 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fe3a40091e0 tx=0x7fe3a4017000 comp rx=0 tx=0).stop 2023-10-06T03:30:15.312 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:15.305+0000 7fe3b359e700 1 -- 172.21.15.110:0/3071736466 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe3b40c97b0 msgr2=0x7fe3b40c19d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:15.312 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:15.305+0000 7fe3b359e700 1 --2- 172.21.15.110:0/3071736466 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe3b40c97b0 0x7fe3b40c19d0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fe39c0011e0 tx=0x7fe39c007df0 comp rx=0 tx=0).stop 2023-10-06T03:30:15.312 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:15.309+0000 7fe3b359e700 1 -- 172.21.15.110:0/3071736466 shutdown_connections 2023-10-06T03:30:15.312 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:15.309+0000 7fe3b359e700 1 --2- 172.21.15.110:0/3071736466 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fe394060580 0x7fe394062a30 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:15.312 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:15.309+0000 7fe3b359e700 1 --2- 172.21.15.110:0/3071736466 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe3b415d2a0 0x7fe3b40c1f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:15.312 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:15.309+0000 7fe3b359e700 1 --2- 172.21.15.110:0/3071736466 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe3b40c97b0 0x7fe3b40c19d0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:15.312 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:15.309+0000 7fe3b359e700 1 --2- 172.21.15.110:0/3071736466 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe3b406b7c0 0x7fe3b4159db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:15.312 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:15.309+0000 7fe3b359e700 1 -- 172.21.15.110:0/3071736466 >> 172.21.15.110:0/3071736466 conn(0x7fe3b40cbac0 msgr2=0x7fe3b415af80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:30:15.312 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:15.309+0000 7fe3b359e700 1 -- 172.21.15.110:0/3071736466 shutdown_connections 2023-10-06T03:30:15.312 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:15.309+0000 7fe3b359e700 1 -- 172.21.15.110:0/3071736466 wait complete. 2023-10-06T03:30:15.324 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-06T03:30:15.432 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:15.429+0000 7f2ef4ecb700 1 Processor -- start 2023-10-06T03:30:15.432 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:15.429+0000 7f2ef4ecb700 1 -- start start 2023-10-06T03:30:15.434 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:15.429+0000 7f2ef4ecb700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f2ef00ce880 0x7f2ef00b0a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:15.434 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:15.429+0000 7f2ef4ecb700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2ef006bb90 0x7f2ef006c010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:15.434 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:15.429+0000 7f2ef4ecb700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ef00cfa00 con 0x7f2ef006b7c0 2023-10-06T03:30:15.435 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:15.429+0000 7f2ef4ecb700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2ef00cb1c0 con 0x7f2ef00ce880 2023-10-06T03:30:15.435 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:15.429+0000 7f2ef4ecb700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ef00cedf0 con 0x7f2ef006bb90 2023-10-06T03:30:15.435 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:15.429+0000 7f2eeeffd700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f2ef00ce880 0x7f2ef00b0a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:15.435 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:15.429+0000 7f2eeffff700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2ef006bb90 0x7f2ef006c010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:15.435 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:15.429+0000 7f2eeeffd700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f2ef00ce880 0x7f2ef00b0a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.110:36568/0 (socket says 172.21.15.110:36568) 2023-10-06T03:30:15.435 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:15.429+0000 7f2eeeffd700 1 -- 172.21.15.110:0/2641597651 learned_addr learned my addr 172.21.15.110:0/2641597651 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:30:15.435 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:15.429+0000 7f2eef7fe700 1 --1- 172.21.15.110:0/2641597651 >> v1:172.21.15.192:6789/0 conn(0x7f2ef006b7c0 0x7f2ef00ce170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.110:52152/0 (socket says 172.21.15.110:52152) 2023-10-06T03:30:15.436 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:15.429+0000 7f2eeffff700 1 -- 172.21.15.110:0/2641597651 >> v2:172.21.15.192:3300/0 conn(0x7f2ef00ce880 msgr2=0x7f2ef00b0a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:30:15.436 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:15.429+0000 7f2eeffff700 1 --2- 172.21.15.110:0/2641597651 >> v2:172.21.15.192:3300/0 conn(0x7f2ef00ce880 0x7f2ef00b0a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:15.437 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:15.429+0000 7f2eeffff700 1 -- 172.21.15.110:0/2641597651 >> v1:172.21.15.192:6789/0 conn(0x7f2ef006b7c0 legacy=0x7f2ef00ce170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:15.437 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:15.429+0000 7f2eeffff700 1 -- 172.21.15.110:0/2641597651 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ef00cf0d0 con 0x7f2ef006bb90 2023-10-06T03:30:15.437 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:15.433+0000 7f2eeffff700 1 --2- 172.21.15.110:0/2641597651 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2ef006bb90 0x7f2ef006c010 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f2ee8009e60 tx=0x7f2ee8009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=79b6360c6c55ca34 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:30:15.437 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:15.433+0000 7f2eee7fc700 1 -- 172.21.15.110:0/2641597651 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ee800e070 con 0x7f2ef006bb90 2023-10-06T03:30:15.437 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:15.433+0000 7f2eee7fc700 1 -- 172.21.15.110:0/2641597651 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ee800bb60 con 0x7f2ef006bb90 2023-10-06T03:30:15.437 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:15.433+0000 7f2eee7fc700 1 -- 172.21.15.110:0/2641597651 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ee8012430 con 0x7f2ef006bb90 2023-10-06T03:30:15.437 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:15.433+0000 7f2ef4ecb700 1 -- 172.21.15.110:0/2641597651 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2ef006bb90 msgr2=0x7f2ef006c010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:15.437 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:15.433+0000 7f2ef4ecb700 1 --2- 172.21.15.110:0/2641597651 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2ef006bb90 0x7f2ef006c010 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f2ee8009e60 tx=0x7f2ee8009780 comp rx=0 tx=0).stop 2023-10-06T03:30:15.437 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:15.433+0000 7f2ef4ecb700 1 -- 172.21.15.110:0/2641597651 shutdown_connections 2023-10-06T03:30:15.437 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:15.433+0000 7f2ef4ecb700 1 --2- 172.21.15.110:0/2641597651 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2ef006bb90 0x7f2ef006c010 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:15.437 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:15.433+0000 7f2ef4ecb700 1 --2- 172.21.15.110:0/2641597651 >> v2:172.21.15.192:3300/0 conn(0x7f2ef00ce880 0x7f2ef00b0a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:15.438 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:15.433+0000 7f2ef4ecb700 1 -- 172.21.15.110:0/2641597651 >> 172.21.15.110:0/2641597651 conn(0x7f2ef00cfea0 msgr2=0x7f2ef00d02a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:30:15.438 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:15.433+0000 7f2ef4ecb700 1 -- 172.21.15.110:0/2641597651 shutdown_connections 2023-10-06T03:30:15.438 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:15.433+0000 7f2ef4ecb700 1 -- 172.21.15.110:0/2641597651 wait complete. 2023-10-06T03:30:15.438 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:15.433+0000 7f2ef4ecb700 1 Processor -- start 2023-10-06T03:30:15.438 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:15.433+0000 7f2ef4ecb700 1 -- start start 2023-10-06T03:30:15.440 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:15.437+0000 7f2ef4ecb700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2ef006b7c0 0x7f2ef01519a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:15.440 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:15.437+0000 7f2ef4ecb700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2ef006bb90 0x7f2ef015bc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:15.440 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:15.437+0000 7f2ef4ecb700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2ef00ce880 0x7f2ef00c90a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:15.440 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:15.437+0000 7f2ef4ecb700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ef0169730 con 0x7f2ef006bb90 2023-10-06T03:30:15.440 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:15.437+0000 7f2ef4ecb700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ef0168ab0 con 0x7f2ef00ce880 2023-10-06T03:30:15.440 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:15.437+0000 7f2ef4ecb700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ef0168d90 con 0x7f2ef006b7c0 2023-10-06T03:30:15.440 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:15.437+0000 7f2eeeffd700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2ef006bb90 0x7f2ef015bc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:15.440 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:15.437+0000 7f2eeeffd700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2ef006bb90 0x7f2ef015bc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:52242/0 (socket says 172.21.15.110:52242) 2023-10-06T03:30:15.441 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:15.437+0000 7f2eeeffd700 1 -- 172.21.15.110:0/4089368676 learned_addr learned my addr 172.21.15.110:0/4089368676 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:30:15.441 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:15.437+0000 7f2eeffff700 1 --2- 172.21.15.110:0/4089368676 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2ef00ce880 0x7f2ef00c90a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:15.441 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:15.437+0000 7f2eef7fe700 1 --2- 172.21.15.110:0/4089368676 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2ef006b7c0 0x7f2ef01519a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:15.442 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:15.437+0000 7f2eeeffd700 1 -- 172.21.15.110:0/4089368676 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2ef006b7c0 msgr2=0x7f2ef01519a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:15.442 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:15.437+0000 7f2eeeffd700 1 --2- 172.21.15.110:0/4089368676 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2ef006b7c0 0x7f2ef01519a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:15.442 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:15.437+0000 7f2eeeffd700 1 -- 172.21.15.110:0/4089368676 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2ef00ce880 msgr2=0x7f2ef00c90a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:15.442 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:15.437+0000 7f2eeeffd700 1 --2- 172.21.15.110:0/4089368676 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2ef00ce880 0x7f2ef00c90a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:15.442 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:15.437+0000 7f2eeeffd700 1 -- 172.21.15.110:0/4089368676 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ee8009a20 con 0x7f2ef006bb90 2023-10-06T03:30:15.442 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:15.437+0000 7f2eeeffd700 1 --2- 172.21.15.110:0/4089368676 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2ef006bb90 0x7f2ef015bc00 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f2ee400e980 tx=0x7f2ee4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:30:15.442 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:15.437+0000 7f2ed77fe700 1 -- 172.21.15.110:0/4089368676 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ee4013680 con 0x7f2ef006bb90 2023-10-06T03:30:15.442 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:15.437+0000 7f2ef4ecb700 1 -- 172.21.15.110:0/4089368676 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ef0169070 con 0x7f2ef006bb90 2023-10-06T03:30:15.442 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:15.437+0000 7f2ed77fe700 1 -- 172.21.15.110:0/4089368676 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ee40137e0 con 0x7f2ef006bb90 2023-10-06T03:30:15.442 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:15.437+0000 7f2ed77fe700 1 -- 172.21.15.110:0/4089368676 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ee4013b60 con 0x7f2ef006bb90 2023-10-06T03:30:15.443 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:15.437+0000 7f2ef4ecb700 1 -- 172.21.15.110:0/4089368676 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ef0151ee0 con 0x7f2ef006bb90 2023-10-06T03:30:15.443 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:15.437+0000 7f2ef4ecb700 1 -- 172.21.15.110:0/4089368676 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ef0070ff0 con 0x7f2ef006bb90 2023-10-06T03:30:15.443 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:15.441+0000 7f2ed77fe700 1 -- 172.21.15.110:0/4089368676 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7f2ee400c400 con 0x7f2ef006bb90 2023-10-06T03:30:15.444 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:15.441+0000 7f2ed77fe700 1 --2- 172.21.15.110:0/4089368676 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f2edc060550 0x7f2edc062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:15.445 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:15.441+0000 7f2ed77fe700 1 -- 172.21.15.110:0/4089368676 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6809+0+0 (secure 0 0 0) 0x7f2ee407d530 con 0x7f2ef006bb90 2023-10-06T03:30:15.445 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:15.441+0000 7f2eef7fe700 1 --2- 172.21.15.110:0/4089368676 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f2edc060550 0x7f2edc062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:15.445 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:15.441+0000 7f2eef7fe700 1 --2- 172.21.15.110:0/4089368676 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f2edc060550 0x7f2edc062a00 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f2ee0004fd0 tx=0x7f2ee0007eb0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:30:15.450 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:15.445+0000 7f2ed77fe700 1 -- 172.21.15.110:0/4089368676 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f2ee4051160 con 0x7f2ef006bb90 2023-10-06T03:30:15.614 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:15.609+0000 7f2ef4ecb700 1 -- 172.21.15.110:0/4089368676 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f2ef00cf3b0 con 0x7f2ef006bb90 2023-10-06T03:30:15.614 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:15.609+0000 7f2ed77fe700 1 -- 172.21.15.110:0/4089368676 <== mon.0 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v29) v1 ==== 104+0+11 (secure 0 0 0) 0x7f2ee4050cf0 con 0x7f2ef006bb90 2023-10-06T03:30:15.615 INFO:teuthology.orchestra.run.smithi110.stdout:pg_num: 32 2023-10-06T03:30:15.617 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:15.613+0000 7f2ef4ecb700 1 -- 172.21.15.110:0/4089368676 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f2edc060550 msgr2=0x7f2edc062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:15.617 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:15.613+0000 7f2ef4ecb700 1 --2- 172.21.15.110:0/4089368676 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f2edc060550 0x7f2edc062a00 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f2ee0004fd0 tx=0x7f2ee0007eb0 comp rx=0 tx=0).stop 2023-10-06T03:30:15.617 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:15.613+0000 7f2ef4ecb700 1 -- 172.21.15.110:0/4089368676 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2ef006bb90 msgr2=0x7f2ef015bc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:15.618 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:15.613+0000 7f2ef4ecb700 1 --2- 172.21.15.110:0/4089368676 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2ef006bb90 0x7f2ef015bc00 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f2ee400e980 tx=0x7f2ee4009250 comp rx=0 tx=0).stop 2023-10-06T03:30:15.618 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:15.613+0000 7f2ef4ecb700 1 -- 172.21.15.110:0/4089368676 shutdown_connections 2023-10-06T03:30:15.618 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:15.613+0000 7f2ef4ecb700 1 --2- 172.21.15.110:0/4089368676 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f2edc060550 0x7f2edc062a00 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:15.618 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:15.613+0000 7f2ef4ecb700 1 --2- 172.21.15.110:0/4089368676 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2ef00ce880 0x7f2ef00c90a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:15.618 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:15.613+0000 7f2ef4ecb700 1 --2- 172.21.15.110:0/4089368676 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2ef006bb90 0x7f2ef015bc00 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:15.618 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:15.613+0000 7f2ef4ecb700 1 --2- 172.21.15.110:0/4089368676 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2ef006b7c0 0x7f2ef01519a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:15.618 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:15.613+0000 7f2ef4ecb700 1 -- 172.21.15.110:0/4089368676 >> 172.21.15.110:0/4089368676 conn(0x7f2ef00cfea0 msgr2=0x7f2ef00ce170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:30:15.618 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:15.613+0000 7f2ef4ecb700 1 -- 172.21.15.110:0/4089368676 shutdown_connections 2023-10-06T03:30:15.618 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:15.613+0000 7f2ef4ecb700 1 -- 172.21.15.110:0/4089368676 wait complete. 2023-10-06T03:30:15.629 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-06T03:30:15.738 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:15.733+0000 7f1e0498e700 1 Processor -- start 2023-10-06T03:30:15.738 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:15.733+0000 7f1e0498e700 1 -- start start 2023-10-06T03:30:15.740 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:15.737+0000 7f1e0498e700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f1e000aef40 0x7f1e000ce200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:15.740 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:15.737+0000 7f1e0498e700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1e0015d210 0x7f1e000afa60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:15.740 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:15.737+0000 7f1e0498e700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e000cf5f0 con 0x7f1e000cb1b0 2023-10-06T03:30:15.740 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:15.737+0000 7f1e0498e700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1e000ce740 con 0x7f1e000aef40 2023-10-06T03:30:15.740 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:15.737+0000 7f1e0498e700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e000ce9e0 con 0x7f1e0015d210 2023-10-06T03:30:15.740 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:15.737+0000 7f1dff7fe700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f1e000aef40 0x7f1e000ce200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:15.741 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:15.737+0000 7f1dff7fe700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f1e000aef40 0x7f1e000ce200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:52250/0 (socket says 172.21.15.110:52250) 2023-10-06T03:30:15.741 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:15.737+0000 7f1dff7fe700 1 -- 172.21.15.110:0/3751875271 learned_addr learned my addr 172.21.15.110:0/3751875271 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:30:15.741 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:15.737+0000 7f1dfffff700 1 --2- 172.21.15.110:0/3751875271 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1e0015d210 0x7f1e000afa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:15.741 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:15.737+0000 7f1dff7fe700 1 -- 172.21.15.110:0/3751875271 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1e0015d210 msgr2=0x7f1e000afa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:30:15.741 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:15.737+0000 7f1dff7fe700 1 --2- 172.21.15.110:0/3751875271 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1e0015d210 0x7f1e000afa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:15.741 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:15.737+0000 7f1dff7fe700 1 -- 172.21.15.110:0/3751875271 >> v1:172.21.15.192:6789/0 conn(0x7f1e000cb1b0 legacy=0x7f1e0015cb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:15.741 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:15.737+0000 7f1dff7fe700 1 -- 172.21.15.110:0/3751875271 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e000cecc0 con 0x7f1e000aef40 2023-10-06T03:30:15.742 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:15.737+0000 7f1dff7fe700 1 --2- 172.21.15.110:0/3751875271 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1e000aef40 0x7f1e000ce200 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f1df00099b0 tx=0x7f1df0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c19b179cd3ed3fd1 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:30:15.742 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:15.737+0000 7f1dfe7fc700 1 -- 172.21.15.110:0/3751875271 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1df0009ea0 con 0x7f1e000aef40 2023-10-06T03:30:15.742 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:15.737+0000 7f1dfe7fc700 1 -- 172.21.15.110:0/3751875271 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1df000b860 con 0x7f1e000aef40 2023-10-06T03:30:15.742 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:15.737+0000 7f1dfe7fc700 1 -- 172.21.15.110:0/3751875271 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1df000bc20 con 0x7f1e000aef40 2023-10-06T03:30:15.743 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:15.737+0000 7f1e0498e700 1 -- 172.21.15.110:0/3751875271 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1e000aef40 msgr2=0x7f1e000ce200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:15.743 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:15.737+0000 7f1e0498e700 1 --2- 172.21.15.110:0/3751875271 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1e000aef40 0x7f1e000ce200 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f1df00099b0 tx=0x7f1df0009250 comp rx=0 tx=0).stop 2023-10-06T03:30:15.743 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:15.737+0000 7f1e0498e700 1 -- 172.21.15.110:0/3751875271 shutdown_connections 2023-10-06T03:30:15.743 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:15.737+0000 7f1e0498e700 1 --2- 172.21.15.110:0/3751875271 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1e0015d210 0x7f1e000afa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:15.743 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:15.737+0000 7f1e0498e700 1 --2- 172.21.15.110:0/3751875271 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1e000aef40 0x7f1e000ce200 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:15.743 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:15.737+0000 7f1e0498e700 1 -- 172.21.15.110:0/3751875271 >> 172.21.15.110:0/3751875271 conn(0x7f1e000cfec0 msgr2=0x7f1e000d02c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:30:15.743 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:15.737+0000 7f1e0498e700 1 -- 172.21.15.110:0/3751875271 shutdown_connections 2023-10-06T03:30:15.743 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:15.737+0000 7f1e0498e700 1 -- 172.21.15.110:0/3751875271 wait complete. 2023-10-06T03:30:15.744 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:15.741+0000 7f1e0498e700 1 Processor -- start 2023-10-06T03:30:15.744 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:15.741+0000 7f1e0498e700 1 -- start start 2023-10-06T03:30:15.746 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:15.741+0000 7f1e0498e700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1e000aef40 0x7f1e000ae410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:15.746 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:15.741+0000 7f1e0498e700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1e000cb1b0 0x7f1e000c7930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:15.746 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:15.741+0000 7f1e0498e700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1e0015d210 0x7f1e0015a970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:15.746 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:15.741+0000 7f1e0498e700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e0015a4e0 con 0x7f1e0015d210 2023-10-06T03:30:15.746 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:15.741+0000 7f1e0498e700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e001595f0 con 0x7f1e000cb1b0 2023-10-06T03:30:15.746 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:15.741+0000 7f1e0498e700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e001598d0 con 0x7f1e000aef40 2023-10-06T03:30:15.746 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:15.741+0000 7f1dff7fe700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1e000aef40 0x7f1e000ae410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:15.746 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:15.741+0000 7f1dff7fe700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1e000aef40 0x7f1e000ae410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3301/0 says I am v2:172.21.15.110:34538/0 (socket says 172.21.15.110:34538) 2023-10-06T03:30:15.746 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:15.741+0000 7f1dff7fe700 1 -- 172.21.15.110:0/2055380905 learned_addr learned my addr 172.21.15.110:0/2055380905 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:30:15.746 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:15.741+0000 7f1dfffff700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1e0015d210 0x7f1e0015a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:15.747 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:15.741+0000 7f1dfeffd700 1 --2- 172.21.15.110:0/2055380905 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1e000cb1b0 0x7f1e000c7930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:15.748 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:15.741+0000 7f1dfffff700 1 -- 172.21.15.110:0/2055380905 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1e000aef40 msgr2=0x7f1e000ae410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:15.748 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:15.741+0000 7f1dfffff700 1 --2- 172.21.15.110:0/2055380905 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1e000aef40 0x7f1e000ae410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:15.748 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:15.741+0000 7f1dfffff700 1 -- 172.21.15.110:0/2055380905 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1e000cb1b0 msgr2=0x7f1e000c7930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:15.748 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:15.741+0000 7f1dfffff700 1 --2- 172.21.15.110:0/2055380905 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1e000cb1b0 0x7f1e000c7930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:15.748 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:15.741+0000 7f1dfffff700 1 -- 172.21.15.110:0/2055380905 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1df00094f0 con 0x7f1e0015d210 2023-10-06T03:30:15.748 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:15.741+0000 7f1dfffff700 1 --2- 172.21.15.110:0/2055380905 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1e0015d210 0x7f1e0015a970 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f1df800ec90 tx=0x7f1df800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:30:15.748 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:15.745+0000 7f1de77fe700 1 -- 172.21.15.110:0/2055380905 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1df800c890 con 0x7f1e0015d210 2023-10-06T03:30:15.748 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:15.745+0000 7f1de77fe700 1 -- 172.21.15.110:0/2055380905 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1df800f040 con 0x7f1e0015d210 2023-10-06T03:30:15.748 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:15.745+0000 7f1e0498e700 1 -- 172.21.15.110:0/2055380905 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e00159bb0 con 0x7f1e0015d210 2023-10-06T03:30:15.748 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:15.745+0000 7f1de77fe700 1 -- 172.21.15.110:0/2055380905 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1df800cc60 con 0x7f1e0015d210 2023-10-06T03:30:15.748 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:15.745+0000 7f1e0498e700 1 -- 172.21.15.110:0/2055380905 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e0015a170 con 0x7f1e0015d210 2023-10-06T03:30:15.749 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:15.745+0000 7f1e0498e700 1 -- 172.21.15.110:0/2055380905 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e000af3a0 con 0x7f1e0015d210 2023-10-06T03:30:15.749 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:15.745+0000 7f1de77fe700 1 -- 172.21.15.110:0/2055380905 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7f1df8014690 con 0x7f1e0015d210 2023-10-06T03:30:15.750 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:15.745+0000 7f1de77fe700 1 --2- 172.21.15.110:0/2055380905 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f1dec060550 0x7f1dec062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:15.750 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:15.745+0000 7f1dff7fe700 1 --2- 172.21.15.110:0/2055380905 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f1dec060550 0x7f1dec062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:15.750 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:15.745+0000 7f1de77fe700 1 -- 172.21.15.110:0/2055380905 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6867+0+0 (secure 0 0 0) 0x7f1df807d700 con 0x7f1e0015d210 2023-10-06T03:30:15.751 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:15.745+0000 7f1dff7fe700 1 --2- 172.21.15.110:0/2055380905 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f1dec060550 0x7f1dec062a00 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f1df0009470 tx=0x7f1df0017000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:30:15.757 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:15.753+0000 7f1de77fe700 1 -- 172.21.15.110:0/2055380905 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f1e000af3a0 con 0x7f1e0015d210 2023-10-06T03:30:15.918 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:15.913+0000 7f1e0498e700 1 -- 172.21.15.110:0/2055380905 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f1e000af3a0 con 0x7f1e0015d210 2023-10-06T03:30:15.918 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:15.913+0000 7f1de77fe700 1 -- 172.21.15.110:0/2055380905 <== mon.0 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v29) v1 ==== 100+0+11 (secure 0 0 0) 0x7f1df8018020 con 0x7f1e0015d210 2023-10-06T03:30:15.919 INFO:teuthology.orchestra.run.smithi110.stdout:pg_num: 64 2023-10-06T03:30:15.921 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:15.917+0000 7f1e0498e700 1 -- 172.21.15.110:0/2055380905 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f1dec060550 msgr2=0x7f1dec062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:15.921 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:15.917+0000 7f1e0498e700 1 --2- 172.21.15.110:0/2055380905 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f1dec060550 0x7f1dec062a00 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f1df0009470 tx=0x7f1df0017000 comp rx=0 tx=0).stop 2023-10-06T03:30:15.921 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:15.917+0000 7f1e0498e700 1 -- 172.21.15.110:0/2055380905 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1e0015d210 msgr2=0x7f1e0015a970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:15.921 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:15.917+0000 7f1e0498e700 1 --2- 172.21.15.110:0/2055380905 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1e0015d210 0x7f1e0015a970 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f1df800ec90 tx=0x7f1df800b040 comp rx=0 tx=0).stop 2023-10-06T03:30:15.921 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:15.917+0000 7f1e0498e700 1 -- 172.21.15.110:0/2055380905 shutdown_connections 2023-10-06T03:30:15.921 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:15.917+0000 7f1e0498e700 1 --2- 172.21.15.110:0/2055380905 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f1dec060550 0x7f1dec062a00 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:15.921 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:15.917+0000 7f1e0498e700 1 --2- 172.21.15.110:0/2055380905 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1e0015d210 0x7f1e0015a970 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:15.921 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:15.917+0000 7f1e0498e700 1 --2- 172.21.15.110:0/2055380905 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1e000cb1b0 0x7f1e000c7930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:15.922 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:15.917+0000 7f1e0498e700 1 --2- 172.21.15.110:0/2055380905 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1e000aef40 0x7f1e000ae410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:15.922 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:15.917+0000 7f1e0498e700 1 -- 172.21.15.110:0/2055380905 >> 172.21.15.110:0/2055380905 conn(0x7f1e000cfec0 msgr2=0x7f1e000b0430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:30:15.922 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:15.917+0000 7f1e0498e700 1 -- 172.21.15.110:0/2055380905 shutdown_connections 2023-10-06T03:30:15.922 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:15.917+0000 7f1e0498e700 1 -- 172.21.15.110:0/2055380905 wait complete. 2023-10-06T03:30:15.932 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2023-10-06T03:30:16.045 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:16.041+0000 7f646ae72700 1 Processor -- start 2023-10-06T03:30:16.046 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:16.041+0000 7f646ae72700 1 -- start start 2023-10-06T03:30:16.048 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:16.041+0000 7f646ae72700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f64640aef40 0x7f64640ce200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:16.048 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:16.041+0000 7f646ae72700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f64640cb1b0 0x7f646415cb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:16.048 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:16.041+0000 7f646ae72700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f646415d040 0x7f646406b7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:16.048 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:16.041+0000 7f646ae72700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f646415d4c0 con 0x7f646415d040 2023-10-06T03:30:16.049 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:16.041+0000 7f646ae72700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f64640cf5f0 con 0x7f64640cb1b0 2023-10-06T03:30:16.049 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:16.041+0000 7f646ae72700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64640ce740 con 0x7f64640aef40 2023-10-06T03:30:16.049 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:16.045+0000 7f646a671700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f646415d040 0x7f646406b7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:16.049 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:16.045+0000 7f6469e70700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f64640aef40 0x7f64640ce200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:16.049 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:16.045+0000 7f646a671700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f646415d040 0x7f646406b7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:52264/0 (socket says 172.21.15.110:52264) 2023-10-06T03:30:16.049 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:16.045+0000 7f646a671700 1 -- 172.21.15.110:0/3680277686 learned_addr learned my addr 172.21.15.110:0/3680277686 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:30:16.050 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:16.045+0000 7f646966f700 1 --2- 172.21.15.110:0/3680277686 >> v2:172.21.15.192:3300/0 conn(0x7f64640cb1b0 0x7f646415cb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:16.050 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:16.045+0000 7f646a671700 1 -- 172.21.15.110:0/3680277686 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f64640aef40 msgr2=0x7f64640ce200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:30:16.050 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:16.045+0000 7f646a671700 1 --2- 172.21.15.110:0/3680277686 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f64640aef40 0x7f64640ce200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:16.050 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:16.045+0000 7f646a671700 1 -- 172.21.15.110:0/3680277686 >> v2:172.21.15.192:3300/0 conn(0x7f64640cb1b0 msgr2=0x7f646415cb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:30:16.050 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:16.045+0000 7f646a671700 1 --2- 172.21.15.110:0/3680277686 >> v2:172.21.15.192:3300/0 conn(0x7f64640cb1b0 0x7f646415cb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:16.051 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:16.045+0000 7f646a671700 1 -- 172.21.15.110:0/3680277686 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64640ce9e0 con 0x7f646415d040 2023-10-06T03:30:16.052 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:16.045+0000 7f646a671700 1 --2- 172.21.15.110:0/3680277686 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f646415d040 0x7f646406b7c0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f646406ef20 tx=0x7f645c009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ac7066935d14b1f server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:30:16.052 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:16.045+0000 7f6468e6e700 1 -- 172.21.15.110:0/3680277686 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f645c00e070 con 0x7f646415d040 2023-10-06T03:30:16.053 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:16.045+0000 7f6468e6e700 1 -- 172.21.15.110:0/3680277686 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f645c00b970 con 0x7f646415d040 2023-10-06T03:30:16.053 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:16.045+0000 7f6468e6e700 1 -- 172.21.15.110:0/3680277686 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f645c0092e0 con 0x7f646415d040 2023-10-06T03:30:16.053 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:16.045+0000 7f646ae72700 1 -- 172.21.15.110:0/3680277686 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f646415d040 msgr2=0x7f646406b7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:16.053 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:16.045+0000 7f646ae72700 1 --2- 172.21.15.110:0/3680277686 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f646415d040 0x7f646406b7c0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f646406ef20 tx=0x7f645c009840 comp rx=0 tx=0).stop 2023-10-06T03:30:16.053 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:16.045+0000 7f646ae72700 1 -- 172.21.15.110:0/3680277686 shutdown_connections 2023-10-06T03:30:16.053 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:16.045+0000 7f646ae72700 1 --2- 172.21.15.110:0/3680277686 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f646415d040 0x7f646406b7c0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:16.053 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:16.045+0000 7f646ae72700 1 --2- 172.21.15.110:0/3680277686 >> v2:172.21.15.192:3300/0 conn(0x7f64640cb1b0 0x7f646415cb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:16.053 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:16.045+0000 7f646ae72700 1 --2- 172.21.15.110:0/3680277686 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f64640aef40 0x7f64640ce200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:16.054 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:16.045+0000 7f646ae72700 1 -- 172.21.15.110:0/3680277686 >> 172.21.15.110:0/3680277686 conn(0x7f64640cfec0 msgr2=0x7f64640d02c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:30:16.054 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:16.045+0000 7f646ae72700 1 -- 172.21.15.110:0/3680277686 shutdown_connections 2023-10-06T03:30:16.054 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:16.045+0000 7f646ae72700 1 -- 172.21.15.110:0/3680277686 wait complete. 2023-10-06T03:30:16.054 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:16.049+0000 7f646ae72700 1 Processor -- start 2023-10-06T03:30:16.054 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:16.049+0000 7f646ae72700 1 -- start start 2023-10-06T03:30:16.056 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:16.049+0000 7f646ae72700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f64640aef40 0x7f6464169230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:16.056 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:16.049+0000 7f646ae72700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f64640cb1b0 0x7f646414b910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:16.057 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:16.049+0000 7f646ae72700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f646415d040 0x7f64640ca2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:16.057 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:16.049+0000 7f646ae72700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64640c6f70 con 0x7f646415d040 2023-10-06T03:30:16.057 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:16.049+0000 7f646ae72700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64640c7250 con 0x7f64640aef40 2023-10-06T03:30:16.057 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:16.049+0000 7f646ae72700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64640c7530 con 0x7f64640cb1b0 2023-10-06T03:30:16.057 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:16.049+0000 7f6469e70700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f64640aef40 0x7f6464169230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:16.057 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:16.049+0000 7f646966f700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f64640cb1b0 0x7f646414b910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:16.057 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:16.049+0000 7f646a671700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f646415d040 0x7f64640ca2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:16.057 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:16.049+0000 7f6469e70700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f64640aef40 0x7f6464169230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.110:36620/0 (socket says 172.21.15.110:36620) 2023-10-06T03:30:16.058 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:16.049+0000 7f6469e70700 1 -- 172.21.15.110:0/4233323288 learned_addr learned my addr 172.21.15.110:0/4233323288 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:30:16.058 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:16.049+0000 7f646966f700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f64640cb1b0 0x7f646414b910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3301/0 says I am v2:172.21.15.110:34566/0 (socket says 172.21.15.110:34566) 2023-10-06T03:30:16.058 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:16.049+0000 7f646a671700 1 -- 172.21.15.110:0/4233323288 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f64640cb1b0 msgr2=0x7f646414b910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:16.058 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:16.049+0000 7f646a671700 1 --2- 172.21.15.110:0/4233323288 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f64640cb1b0 0x7f646414b910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:16.058 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:16.049+0000 7f646a671700 1 -- 172.21.15.110:0/4233323288 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f64640aef40 msgr2=0x7f6464169230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:16.058 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:16.049+0000 7f646a671700 1 --2- 172.21.15.110:0/4233323288 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f64640aef40 0x7f6464169230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:16.059 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:16.049+0000 7f646a671700 1 -- 172.21.15.110:0/4233323288 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f645c009ae0 con 0x7f646415d040 2023-10-06T03:30:16.059 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:16.049+0000 7f646966f700 1 --2- 172.21.15.110:0/4233323288 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f64640cb1b0 0x7f646414b910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-06T03:30:16.059 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:16.049+0000 7f6469e70700 1 --2- 172.21.15.110:0/4233323288 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f64640aef40 0x7f6464169230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-06T03:30:16.060 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:16.053+0000 7f646a671700 1 --2- 172.21.15.110:0/4233323288 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f646415d040 0x7f64640ca2f0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f645c015000 tx=0x7f645c018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:30:16.060 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:16.053+0000 7f6455ffb700 1 -- 172.21.15.110:0/4233323288 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f645c0092e0 con 0x7f646415d040 2023-10-06T03:30:16.061 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:16.053+0000 7f6455ffb700 1 -- 172.21.15.110:0/4233323288 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f645c007b00 con 0x7f646415d040 2023-10-06T03:30:16.061 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:16.053+0000 7f646ae72700 1 -- 172.21.15.110:0/4233323288 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64640c7810 con 0x7f646415d040 2023-10-06T03:30:16.061 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:16.053+0000 7f6455ffb700 1 -- 172.21.15.110:0/4233323288 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f645c01c410 con 0x7f646415d040 2023-10-06T03:30:16.062 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:16.053+0000 7f646ae72700 1 -- 172.21.15.110:0/4233323288 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f646414be50 con 0x7f646415d040 2023-10-06T03:30:16.063 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:16.053+0000 7f646ae72700 1 -- 172.21.15.110:0/4233323288 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6464070ff0 con 0x7f646415d040 2023-10-06T03:30:16.063 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:16.053+0000 7f6455ffb700 1 -- 172.21.15.110:0/4233323288 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7f645c003050 con 0x7f646415d040 2023-10-06T03:30:16.063 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:16.053+0000 7f6455ffb700 1 --2- 172.21.15.110:0/4233323288 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f6448060550 0x7f6448062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:16.063 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:16.053+0000 7f6455ffb700 1 -- 172.21.15.110:0/4233323288 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6867+0+0 (secure 0 0 0) 0x7f645c07e2e0 con 0x7f646415d040 2023-10-06T03:30:16.064 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:16.053+0000 7f6469e70700 1 --2- 172.21.15.110:0/4233323288 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f6448060550 0x7f6448062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:16.064 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:16.057+0000 7f6469e70700 1 --2- 172.21.15.110:0/4233323288 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f6448060550 0x7f6448062a00 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f6458009fd0 tx=0x7f64580091a0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:30:16.067 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:16.065+0000 7f6455ffb700 1 -- 172.21.15.110:0/4233323288 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f645c04e440 con 0x7f646415d040 2023-10-06T03:30:16.225 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:16.221+0000 7f646ae72700 1 -- 172.21.15.110:0/4233323288 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f646406d8d0 con 0x7f646415d040 2023-10-06T03:30:16.225 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:16.221+0000 7f6455ffb700 1 -- 172.21.15.110:0/4233323288 <== mon.0 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v29) v1 ==== 103+0+11 (secure 0 0 0) 0x7f645c051a60 con 0x7f646415d040 2023-10-06T03:30:16.226 INFO:teuthology.orchestra.run.smithi110.stdout:pg_num: 32 2023-10-06T03:30:16.228 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:16.225+0000 7f646ae72700 1 -- 172.21.15.110:0/4233323288 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f6448060550 msgr2=0x7f6448062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:16.228 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:16.225+0000 7f646ae72700 1 --2- 172.21.15.110:0/4233323288 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f6448060550 0x7f6448062a00 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f6458009fd0 tx=0x7f64580091a0 comp rx=0 tx=0).stop 2023-10-06T03:30:16.228 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:16.225+0000 7f646ae72700 1 -- 172.21.15.110:0/4233323288 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f646415d040 msgr2=0x7f64640ca2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:16.229 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:16.225+0000 7f646ae72700 1 --2- 172.21.15.110:0/4233323288 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f646415d040 0x7f64640ca2f0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f645c015000 tx=0x7f645c018000 comp rx=0 tx=0).stop 2023-10-06T03:30:16.229 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:16.225+0000 7f646ae72700 1 -- 172.21.15.110:0/4233323288 shutdown_connections 2023-10-06T03:30:16.229 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:16.225+0000 7f646ae72700 1 --2- 172.21.15.110:0/4233323288 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f6448060550 0x7f6448062a00 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:16.229 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:16.225+0000 7f646ae72700 1 --2- 172.21.15.110:0/4233323288 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f646415d040 0x7f64640ca2f0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:16.229 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:16.225+0000 7f646ae72700 1 --2- 172.21.15.110:0/4233323288 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f64640cb1b0 0x7f646414b910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:16.229 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:16.225+0000 7f646ae72700 1 --2- 172.21.15.110:0/4233323288 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f64640aef40 0x7f6464169230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:16.229 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:16.225+0000 7f646ae72700 1 -- 172.21.15.110:0/4233323288 >> 172.21.15.110:0/4233323288 conn(0x7f64640cfec0 msgr2=0x7f64640afb40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:30:16.229 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:16.225+0000 7f646ae72700 1 -- 172.21.15.110:0/4233323288 shutdown_connections 2023-10-06T03:30:16.229 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:16.225+0000 7f646ae72700 1 -- 172.21.15.110:0/4233323288 wait complete. 2023-10-06T03:30:16.264 INFO:tasks.cephfs_test_runner:Starting test: test_snap_dir_name (tasks.cephfs.test_snap_schedules.TestSnapSchedulesSnapdir) Test the correctness of snap directory name 2023-10-06T03:30:16.265 INFO:tasks.cephfs_test_runner:test_snap_dir_name (tasks.cephfs.test_snap_schedules.TestSnapSchedulesSnapdir) 2023-10-06T03:30:16.265 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedulesSnapdir.test_snap_dir_name' 2023-10-06T03:30:16.374 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:16.369+0000 7fe755658700 1 Processor -- start 2023-10-06T03:30:16.374 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:16.369+0000 7fe755658700 1 -- start start 2023-10-06T03:30:16.375 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:16.369+0000 7fe755658700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe75015d420 0x7fe7500c7b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:16.375 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:16.369+0000 7fe755658700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fe750069d70 0x7fe7500ca270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:16.375 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:16.369+0000 7fe755658700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe75006aef0 con 0x7fe75015d7f0 2023-10-06T03:30:16.375 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:16.369+0000 7fe755658700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe75006c7d0 con 0x7fe750069d70 2023-10-06T03:30:16.376 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:16.369+0000 7fe755658700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe75006a2e0 con 0x7fe75015d420 2023-10-06T03:30:16.376 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:16.369+0000 7fe747fff700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fe750069d70 0x7fe7500ca270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:16.376 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:16.369+0000 7fe74effd700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe75015d420 0x7fe7500c7b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:16.376 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:16.373+0000 7fe747fff700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fe750069d70 0x7fe7500ca270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.110:36622/0 (socket says 172.21.15.110:36622) 2023-10-06T03:30:16.376 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:16.373+0000 7fe747fff700 1 -- 172.21.15.110:0/1117532239 learned_addr learned my addr 172.21.15.110:0/1117532239 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:30:16.378 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:16.373+0000 7fe747fff700 1 -- 172.21.15.110:0/1117532239 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe75015d420 msgr2=0x7fe7500c7b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:30:16.378 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:16.373+0000 7fe747fff700 1 --2- 172.21.15.110:0/1117532239 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe75015d420 0x7fe7500c7b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:16.378 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:16.373+0000 7fe747fff700 1 -- 172.21.15.110:0/1117532239 >> v1:172.21.15.192:6789/0 conn(0x7fe75015d7f0 legacy=0x7fe750161250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:16.378 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:16.373+0000 7fe747fff700 1 -- 172.21.15.110:0/1117532239 --> v2:172.21.15.192:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe75006a5c0 con 0x7fe750069d70 2023-10-06T03:30:16.379 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:16.373+0000 7fe74e7fc700 1 -- 172.21.15.110:0/1117532239 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2240816271 0 0) 0x7fe75006aef0 con 0x7fe75015d7f0 2023-10-06T03:30:16.379 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:16.373+0000 7fe747fff700 1 --2- 172.21.15.110:0/1117532239 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe750069d70 0x7fe7500ca270 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fe7380099b0 tx=0x7fe738009250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2c94f8ce9e624838 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:30:16.379 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:16.373+0000 7fe74e7fc700 1 -- 172.21.15.110:0/1117532239 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe738009ea0 con 0x7fe750069d70 2023-10-06T03:30:16.379 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:16.373+0000 7fe74e7fc700 1 -- 172.21.15.110:0/1117532239 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe73800b860 con 0x7fe750069d70 2023-10-06T03:30:16.379 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:16.373+0000 7fe74e7fc700 1 -- 172.21.15.110:0/1117532239 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe73800bc00 con 0x7fe750069d70 2023-10-06T03:30:16.379 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:16.373+0000 7fe755658700 1 -- 172.21.15.110:0/1117532239 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe750069d70 msgr2=0x7fe7500ca270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:16.379 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:16.373+0000 7fe755658700 1 --2- 172.21.15.110:0/1117532239 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe750069d70 0x7fe7500ca270 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fe7380099b0 tx=0x7fe738009250 comp rx=0 tx=0).stop 2023-10-06T03:30:16.379 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:16.373+0000 7fe755658700 1 -- 172.21.15.110:0/1117532239 shutdown_connections 2023-10-06T03:30:16.380 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:16.373+0000 7fe755658700 1 --2- 172.21.15.110:0/1117532239 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe750069d70 0x7fe7500ca270 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:16.380 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:16.373+0000 7fe755658700 1 --2- 172.21.15.110:0/1117532239 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe75015d420 0x7fe7500c7b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:16.380 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:16.373+0000 7fe755658700 1 -- 172.21.15.110:0/1117532239 >> 172.21.15.110:0/1117532239 conn(0x7fe7500cbd00 msgr2=0x7fe7500cc100 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:30:16.380 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:16.373+0000 7fe755658700 1 -- 172.21.15.110:0/1117532239 shutdown_connections 2023-10-06T03:30:16.380 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:16.373+0000 7fe755658700 1 -- 172.21.15.110:0/1117532239 wait complete. 2023-10-06T03:30:16.380 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:16.377+0000 7fe755658700 1 Processor -- start 2023-10-06T03:30:16.381 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:16.377+0000 7fe755658700 1 -- start start 2023-10-06T03:30:16.383 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:16.377+0000 7fe755658700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe750069d70 0x7fe7500c58c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:16.383 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:16.377+0000 7fe755658700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe75015d420 0x7fe750148260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:16.383 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:16.377+0000 7fe755658700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe75015d7f0 0x7fe7500aed20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:16.384 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:16.377+0000 7fe755658700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7500c5430 con 0x7fe750069d70 2023-10-06T03:30:16.384 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:16.377+0000 7fe755658700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7500c4820 con 0x7fe75015d7f0 2023-10-06T03:30:16.384 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:16.377+0000 7fe755658700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7500c4b00 con 0x7fe75015d420 2023-10-06T03:30:16.384 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:16.377+0000 7fe74effd700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe750069d70 0x7fe7500c58c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:16.384 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:16.377+0000 7fe74effd700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe750069d70 0x7fe7500c58c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:52286/0 (socket says 172.21.15.110:52286) 2023-10-06T03:30:16.384 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:16.377+0000 7fe74effd700 1 -- 172.21.15.110:0/3773830046 learned_addr learned my addr 172.21.15.110:0/3773830046 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:30:16.385 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:16.377+0000 7fe74f7fe700 1 --2- 172.21.15.110:0/3773830046 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe75015d7f0 0x7fe7500aed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:16.385 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:16.377+0000 7fe747fff700 1 --2- 172.21.15.110:0/3773830046 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe75015d420 0x7fe750148260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:16.385 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:16.377+0000 7fe74effd700 1 -- 172.21.15.110:0/3773830046 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe75015d420 msgr2=0x7fe750148260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:16.385 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:16.377+0000 7fe74effd700 1 --2- 172.21.15.110:0/3773830046 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe75015d420 0x7fe750148260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:16.385 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:16.377+0000 7fe74effd700 1 -- 172.21.15.110:0/3773830046 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe75015d7f0 msgr2=0x7fe7500aed20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:16.385 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:16.377+0000 7fe74effd700 1 --2- 172.21.15.110:0/3773830046 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe75015d7f0 0x7fe7500aed20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:16.385 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:16.377+0000 7fe74effd700 1 -- 172.21.15.110:0/3773830046 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7380094f0 con 0x7fe750069d70 2023-10-06T03:30:16.385 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:16.377+0000 7fe74f7fe700 1 --2- 172.21.15.110:0/3773830046 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe75015d7f0 0x7fe7500aed20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-06T03:30:16.386 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:16.377+0000 7fe74effd700 1 --2- 172.21.15.110:0/3773830046 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe750069d70 0x7fe7500c58c0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fe74000ec80 tx=0x7fe7400091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:30:16.386 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:16.377+0000 7fe746ffd700 1 -- 172.21.15.110:0/3773830046 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe74000ca60 con 0x7fe750069d70 2023-10-06T03:30:16.386 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:16.377+0000 7fe746ffd700 1 -- 172.21.15.110:0/3773830046 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe74000cbe0 con 0x7fe750069d70 2023-10-06T03:30:16.386 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:16.377+0000 7fe755658700 1 -- 172.21.15.110:0/3773830046 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7500c4de0 con 0x7fe750069d70 2023-10-06T03:30:16.386 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:16.377+0000 7fe746ffd700 1 -- 172.21.15.110:0/3773830046 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe740013470 con 0x7fe750069d70 2023-10-06T03:30:16.386 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:16.377+0000 7fe755658700 1 -- 172.21.15.110:0/3773830046 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7500af260 con 0x7fe750069d70 2023-10-06T03:30:16.388 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:16.381+0000 7fe755658700 1 -- 172.21.15.110:0/3773830046 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe75006ff60 con 0x7fe750069d70 2023-10-06T03:30:16.389 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:16.381+0000 7fe746ffd700 1 -- 172.21.15.110:0/3773830046 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7fe740013860 con 0x7fe750069d70 2023-10-06T03:30:16.389 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:16.381+0000 7fe746ffd700 1 --2- 172.21.15.110:0/3773830046 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fe730060550 0x7fe730062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:16.389 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:16.381+0000 7fe747fff700 1 --2- 172.21.15.110:0/3773830046 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fe730060550 0x7fe730062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:16.389 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:16.381+0000 7fe746ffd700 1 -- 172.21.15.110:0/3773830046 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6867+0+0 (secure 0 0 0) 0x7fe74007d070 con 0x7fe750069d70 2023-10-06T03:30:16.390 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:16.385+0000 7fe747fff700 1 --2- 172.21.15.110:0/3773830046 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fe730060550 0x7fe730062a00 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fe7380099b0 tx=0x7fe738017000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:30:16.395 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:16.389+0000 7fe746ffd700 1 -- 172.21.15.110:0/3773830046 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fe74004d1d0 con 0x7fe750069d70 2023-10-06T03:30:16.616 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:16.613+0000 7fe755658700 1 -- 172.21.15.110:0/3773830046 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedulesSnapdir.test_snap_dir_name"]} v 0) v1 -- 0x7fe75006a8a0 con 0x7fe750069d70 2023-10-06T03:30:17.496 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:17.493+0000 7fe746ffd700 1 -- 172.21.15.110:0/3773830046 <== mon.0 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedulesSnapdir.test_snap_dir_name"]}]=0 v61) v1 ==== 158+0+0 (secure 0 0 0) 0x7fe740054e60 con 0x7fe750069d70 2023-10-06T03:30:17.501 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:17.497+0000 7fe755658700 1 -- 172.21.15.110:0/3773830046 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fe730060550 msgr2=0x7fe730062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:17.501 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:17.497+0000 7fe755658700 1 --2- 172.21.15.110:0/3773830046 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fe730060550 0x7fe730062a00 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fe7380099b0 tx=0x7fe738017000 comp rx=0 tx=0).stop 2023-10-06T03:30:17.501 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:17.497+0000 7fe755658700 1 -- 172.21.15.110:0/3773830046 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe750069d70 msgr2=0x7fe7500c58c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:17.502 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:17.497+0000 7fe755658700 1 --2- 172.21.15.110:0/3773830046 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe750069d70 0x7fe7500c58c0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fe74000ec80 tx=0x7fe7400091a0 comp rx=0 tx=0).stop 2023-10-06T03:30:17.502 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:17.497+0000 7fe755658700 1 -- 172.21.15.110:0/3773830046 shutdown_connections 2023-10-06T03:30:17.502 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:17.497+0000 7fe755658700 1 --2- 172.21.15.110:0/3773830046 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fe730060550 0x7fe730062a00 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:17.502 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:17.497+0000 7fe755658700 1 --2- 172.21.15.110:0/3773830046 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe75015d7f0 0x7fe7500aed20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:17.502 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:17.497+0000 7fe755658700 1 --2- 172.21.15.110:0/3773830046 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe75015d420 0x7fe750148260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:17.502 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:17.497+0000 7fe755658700 1 --2- 172.21.15.110:0/3773830046 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe750069d70 0x7fe7500c58c0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:17.502 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:17.497+0000 7fe755658700 1 -- 172.21.15.110:0/3773830046 >> 172.21.15.110:0/3773830046 conn(0x7fe7500cbd00 msgr2=0x7fe7500c8a90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:30:17.502 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:17.497+0000 7fe755658700 1 -- 172.21.15.110:0/3773830046 shutdown_connections 2023-10-06T03:30:17.502 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:17.497+0000 7fe755658700 1 -- 172.21.15.110:0/3773830046 wait complete. 2023-10-06T03:30:17.519 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2023-10-06T03:30:17.641 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:17.637+0000 7fd6799b5700 1 Processor -- start 2023-10-06T03:30:17.641 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:17.637+0000 7fd6799b5700 1 -- start start 2023-10-06T03:30:17.643 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:17.637+0000 7fd6799b5700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fd6740af070 0x7fd6740af440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:17.643 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:17.637+0000 7fd6799b5700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd6740cfa40 0x7fd6740cb220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:17.643 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:17.637+0000 7fd6799b5700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6740cf5b0 con 0x7fd6740ce2a0 2023-10-06T03:30:17.643 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:17.637+0000 7fd6799b5700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd6740ce700 con 0x7fd6740af070 2023-10-06T03:30:17.643 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:17.637+0000 7fd6799b5700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6740ce9a0 con 0x7fd6740cfa40 2023-10-06T03:30:17.644 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:17.637+0000 7fd6737fe700 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7fd6740ce2a0 0x7fd67415cc50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:53296/0 (socket says 172.21.15.110:53296) 2023-10-06T03:30:17.644 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:17.637+0000 7fd6737fe700 1 -- 172.21.15.110:0/2201721387 learned_addr learned my addr 172.21.15.110:0/2201721387 (peer_addr_for_me v1:172.21.15.110:0/0) 2023-10-06T03:30:17.644 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:17.637+0000 7fd672ffd700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fd6740af070 0x7fd6740af440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:17.644 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:17.637+0000 7fd6727fc700 1 --2- 172.21.15.110:0/2201721387 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd6740cfa40 0x7fd6740cb220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:17.644 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:17.637+0000 7fd671ffb700 1 -- 172.21.15.110:0/2201721387 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2850736175 0 0) 0x7fd6740cf5b0 con 0x7fd6740ce2a0 2023-10-06T03:30:17.644 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:17.637+0000 7fd671ffb700 1 -- 172.21.15.110:0/2201721387 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd65c003570 con 0x7fd6740ce2a0 2023-10-06T03:30:17.644 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:17.641+0000 7fd672ffd700 1 -- 172.21.15.110:0/2201721387 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd6740cfa40 msgr2=0x7fd6740cb220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:30:17.644 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:17.641+0000 7fd672ffd700 1 --2- 172.21.15.110:0/2201721387 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd6740cfa40 0x7fd6740cb220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:17.644 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:17.641+0000 7fd672ffd700 1 -- 172.21.15.110:0/2201721387 >> v1:172.21.15.110:6789/0 conn(0x7fd6740ce2a0 legacy=0x7fd67415cc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:17.644 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:17.641+0000 7fd672ffd700 1 -- 172.21.15.110:0/2201721387 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6740cec80 con 0x7fd6740af070 2023-10-06T03:30:17.645 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:17.641+0000 7fd672ffd700 1 --2- 172.21.15.110:0/2201721387 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd6740af070 0x7fd6740af440 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fd6640098e0 tx=0x7fd6640091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc04aa633c8dffe8 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:30:17.646 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:17.641+0000 7fd671ffb700 1 -- 172.21.15.110:0/2201721387 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd664011630 con 0x7fd6740af070 2023-10-06T03:30:17.646 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:17.641+0000 7fd671ffb700 1 -- 172.21.15.110:0/2201721387 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd664011790 con 0x7fd6740af070 2023-10-06T03:30:17.646 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:17.641+0000 7fd671ffb700 1 -- 172.21.15.110:0/2201721387 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd664011b10 con 0x7fd6740af070 2023-10-06T03:30:17.646 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:17.641+0000 7fd6799b5700 1 -- 172.21.15.110:0/2201721387 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd6740af070 msgr2=0x7fd6740af440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:17.646 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:17.641+0000 7fd6799b5700 1 --2- 172.21.15.110:0/2201721387 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd6740af070 0x7fd6740af440 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fd6640098e0 tx=0x7fd6640091a0 comp rx=0 tx=0).stop 2023-10-06T03:30:17.646 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:17.641+0000 7fd6799b5700 1 -- 172.21.15.110:0/2201721387 shutdown_connections 2023-10-06T03:30:17.646 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:17.641+0000 7fd6799b5700 1 --2- 172.21.15.110:0/2201721387 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd6740cfa40 0x7fd6740cb220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:17.646 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:17.641+0000 7fd6799b5700 1 --2- 172.21.15.110:0/2201721387 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd6740af070 0x7fd6740af440 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:17.646 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:17.641+0000 7fd6799b5700 1 -- 172.21.15.110:0/2201721387 >> 172.21.15.110:0/2201721387 conn(0x7fd67406e840 msgr2=0x7fd6740d0010 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:30:17.646 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:17.641+0000 7fd6799b5700 1 -- 172.21.15.110:0/2201721387 shutdown_connections 2023-10-06T03:30:17.647 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:17.641+0000 7fd6799b5700 1 -- 172.21.15.110:0/2201721387 wait complete. 2023-10-06T03:30:17.647 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:17.641+0000 7fd6799b5700 1 Processor -- start 2023-10-06T03:30:17.647 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:17.641+0000 7fd6799b5700 1 -- start start 2023-10-06T03:30:17.649 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:17.645+0000 7fd6799b5700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd6740af070 0x7fd67415c000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:17.649 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:17.645+0000 7fd6799b5700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd6740ce2a0 0x7fd674144930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:17.649 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:17.645+0000 7fd6799b5700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd6740cfa40 0x7fd6740c89b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:17.649 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:17.645+0000 7fd6799b5700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd674159740 con 0x7fd6740cfa40 2023-10-06T03:30:17.649 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:17.645+0000 7fd6799b5700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd67415c540 con 0x7fd6740af070 2023-10-06T03:30:17.649 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:17.645+0000 7fd6799b5700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6740b1410 con 0x7fd6740ce2a0 2023-10-06T03:30:17.649 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:17.645+0000 7fd6737fe700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd6740cfa40 0x7fd6740c89b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:17.649 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:17.645+0000 7fd6727fc700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd6740ce2a0 0x7fd674144930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:17.649 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:17.645+0000 7fd6737fe700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd6740cfa40 0x7fd6740c89b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:52316/0 (socket says 172.21.15.110:52316) 2023-10-06T03:30:17.650 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:17.645+0000 7fd6727fc700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd6740ce2a0 0x7fd674144930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3301/0 says I am v2:172.21.15.110:34608/0 (socket says 172.21.15.110:34608) 2023-10-06T03:30:17.650 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:17.645+0000 7fd6737fe700 1 -- 172.21.15.110:0/2802418477 learned_addr learned my addr 172.21.15.110:0/2802418477 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:30:17.650 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:17.645+0000 7fd672ffd700 1 --2- 172.21.15.110:0/2802418477 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd6740af070 0x7fd67415c000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:17.651 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:17.645+0000 7fd6737fe700 1 -- 172.21.15.110:0/2802418477 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd6740ce2a0 msgr2=0x7fd674144930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:17.651 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:17.645+0000 7fd6737fe700 1 --2- 172.21.15.110:0/2802418477 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd6740ce2a0 0x7fd674144930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:17.651 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:17.645+0000 7fd6737fe700 1 -- 172.21.15.110:0/2802418477 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd6740af070 msgr2=0x7fd67415c000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:17.651 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:17.645+0000 7fd6737fe700 1 --2- 172.21.15.110:0/2802418477 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd6740af070 0x7fd67415c000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:17.651 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:17.645+0000 7fd6737fe700 1 -- 172.21.15.110:0/2802418477 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd664009420 con 0x7fd6740cfa40 2023-10-06T03:30:17.651 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:17.645+0000 7fd6727fc700 1 --2- 172.21.15.110:0/2802418477 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd6740ce2a0 0x7fd674144930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-06T03:30:17.652 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:17.645+0000 7fd6737fe700 1 --2- 172.21.15.110:0/2802418477 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd6740cfa40 0x7fd6740c89b0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fd66c00cc50 tx=0x7fd66c001a60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:30:17.652 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:17.645+0000 7fd65affd700 1 -- 172.21.15.110:0/2802418477 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd66c00a870 con 0x7fd6740cfa40 2023-10-06T03:30:17.652 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:17.645+0000 7fd65affd700 1 -- 172.21.15.110:0/2802418477 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd66c00a9d0 con 0x7fd6740cfa40 2023-10-06T03:30:17.652 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:17.645+0000 7fd6799b5700 1 -- 172.21.15.110:0/2802418477 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6740b16f0 con 0x7fd6740cfa40 2023-10-06T03:30:17.652 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:17.645+0000 7fd65affd700 1 -- 172.21.15.110:0/2802418477 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd66c00ad90 con 0x7fd6740cfa40 2023-10-06T03:30:17.652 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:17.645+0000 7fd6799b5700 1 -- 172.21.15.110:0/2802418477 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6740b1cb0 con 0x7fd6740cfa40 2023-10-06T03:30:17.652 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:17.649+0000 7fd6799b5700 1 -- 172.21.15.110:0/2802418477 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd6741598b0 con 0x7fd6740cfa40 2023-10-06T03:30:17.653 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:17.649+0000 7fd65affd700 1 -- 172.21.15.110:0/2802418477 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7fd66c007300 con 0x7fd6740cfa40 2023-10-06T03:30:17.654 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:17.649+0000 7fd65affd700 1 --2- 172.21.15.110:0/2802418477 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fd660060550 0x7fd660062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:17.654 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:17.649+0000 7fd65affd700 1 -- 172.21.15.110:0/2802418477 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 6867+0+0 (secure 0 0 0) 0x7fd66c07cf00 con 0x7fd6740cfa40 2023-10-06T03:30:17.654 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:17.649+0000 7fd672ffd700 1 --2- 172.21.15.110:0/2802418477 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fd660060550 0x7fd660062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:17.655 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:17.649+0000 7fd672ffd700 1 --2- 172.21.15.110:0/2802418477 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fd660060550 0x7fd660062a00 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fd6640091e0 tx=0x7fd664017000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:30:17.660 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:17.657+0000 7fd65affd700 1 -- 172.21.15.110:0/2802418477 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fd6741598b0 con 0x7fd6740cfa40 2023-10-06T03:30:17.822 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:17.817+0000 7fd6799b5700 1 -- 172.21.15.110:0/2802418477 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7fd674159880 con 0x7fd6740cfa40 2023-10-06T03:30:17.827 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:17.821+0000 7fd65affd700 1 -- 172.21.15.110:0/2802418477 <== mon.0 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v10)=0 v10) v1 ==== 122+0+0 (secure 0 0 0) 0x7fd66c04d0e0 con 0x7fd6740cfa40 2023-10-06T03:30:17.830 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:17.825+0000 7fd6799b5700 1 -- 172.21.15.110:0/2802418477 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fd660060550 msgr2=0x7fd660062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:17.830 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:17.825+0000 7fd6799b5700 1 --2- 172.21.15.110:0/2802418477 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fd660060550 0x7fd660062a00 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fd6640091e0 tx=0x7fd664017000 comp rx=0 tx=0).stop 2023-10-06T03:30:17.830 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:17.825+0000 7fd6799b5700 1 -- 172.21.15.110:0/2802418477 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd6740cfa40 msgr2=0x7fd6740c89b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:17.830 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:17.825+0000 7fd6799b5700 1 --2- 172.21.15.110:0/2802418477 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd6740cfa40 0x7fd6740c89b0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fd66c00cc50 tx=0x7fd66c001a60 comp rx=0 tx=0).stop 2023-10-06T03:30:17.830 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:17.825+0000 7fd6799b5700 1 -- 172.21.15.110:0/2802418477 shutdown_connections 2023-10-06T03:30:17.830 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:17.825+0000 7fd6799b5700 1 --2- 172.21.15.110:0/2802418477 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fd660060550 0x7fd660062a00 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:17.830 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:17.825+0000 7fd6799b5700 1 --2- 172.21.15.110:0/2802418477 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd6740cfa40 0x7fd6740c89b0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:17.831 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:17.825+0000 7fd6799b5700 1 --2- 172.21.15.110:0/2802418477 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd6740ce2a0 0x7fd674144930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:17.831 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:17.825+0000 7fd6799b5700 1 --2- 172.21.15.110:0/2802418477 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd6740af070 0x7fd67415c000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:17.831 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:17.825+0000 7fd6799b5700 1 -- 172.21.15.110:0/2802418477 >> 172.21.15.110:0/2802418477 conn(0x7fd67406e840 msgr2=0x7fd674156c50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:30:17.831 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:17.825+0000 7fd6799b5700 1 -- 172.21.15.110:0/2802418477 shutdown_connections 2023-10-06T03:30:17.831 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:17.825+0000 7fd6799b5700 1 -- 172.21.15.110:0/2802418477 wait complete. 2023-10-06T03:30:17.842 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-10-06T03:30:17.842 DEBUG:teuthology.orchestra.run.smithi110:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-06T03:30:17.860 DEBUG:teuthology.orchestra.run.smithi192:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-06T03:30:17.881 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-10-06T03:30:17.881 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi110.front.sepia.ceph.com... 2023-10-06T03:30:17.881 INFO:teuthology.orchestra.run:Running command with timeout 1800 2023-10-06T03:30:17.882 DEBUG:teuthology.orchestra.run.smithi110:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2023-10-06T03:30:17.896 INFO:teuthology.orchestra.run:waiting for 900 2023-10-06T03:30:17.898 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi110.front.sepia.ceph.com 2023-10-06T03:30:17.898 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-06T03:30:17.898 DEBUG:teuthology.orchestra.run.smithi110:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2023-10-06T03:30:17.946 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi192.front.sepia.ceph.com... 2023-10-06T03:30:17.946 INFO:teuthology.orchestra.run:Running command with timeout 1800 2023-10-06T03:30:17.947 DEBUG:teuthology.orchestra.run.smithi192:> sudo fusermount -u /home/ubuntu/cephtest/mnt.1 2023-10-06T03:30:17.973 INFO:teuthology.orchestra.run:waiting for 900 2023-10-06T03:30:17.973 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi192.front.sepia.ceph.com 2023-10-06T03:30:17.973 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-06T03:30:17.973 DEBUG:teuthology.orchestra.run.smithi192:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2023-10-06T03:30:18.035 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi110.front.sepia.ceph.com... 2023-10-06T03:30:18.036 INFO:teuthology.orchestra.run:Running command with timeout 1800 2023-10-06T03:30:18.036 DEBUG:teuthology.orchestra.run.smithi110:> sudo fusermount -u /home/ubuntu/cephtest/mnt.2 2023-10-06T03:30:18.051 INFO:teuthology.orchestra.run:waiting for 900 2023-10-06T03:30:18.053 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi110.front.sepia.ceph.com 2023-10-06T03:30:18.053 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-06T03:30:18.053 DEBUG:teuthology.orchestra.run.smithi110:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2023-10-06T03:30:18.102 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi192.front.sepia.ceph.com... 2023-10-06T03:30:18.102 INFO:teuthology.orchestra.run:Running command with timeout 1800 2023-10-06T03:30:18.103 DEBUG:teuthology.orchestra.run.smithi192:> sudo fusermount -u /home/ubuntu/cephtest/mnt.3 2023-10-06T03:30:18.119 INFO:teuthology.orchestra.run:waiting for 900 2023-10-06T03:30:18.120 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi192.front.sepia.ceph.com 2023-10-06T03:30:18.120 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-06T03:30:18.120 DEBUG:teuthology.orchestra.run.smithi192:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2023-10-06T03:30:18.169 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2023-10-06T03:30:18.169 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-06T03:30:18.283 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:18.277+0000 7f7e9bfff700 1 Processor -- start 2023-10-06T03:30:18.284 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:18.277+0000 7f7e9bfff700 1 -- start start 2023-10-06T03:30:18.284 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:18.281+0000 7f7e9bfff700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7e9c0c9bb0 0x7f7e9c0ade90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:18.285 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:18.281+0000 7f7e9bfff700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f7e9c0aec00 0x7f7e9c0c6540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:18.285 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:18.281+0000 7f7e9bfff700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e9c06c190 con 0x7f7e9c0adac0 2023-10-06T03:30:18.285 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:18.281+0000 7f7e9bfff700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e9c0cb2b0 con 0x7f7e9c0aec00 2023-10-06T03:30:18.285 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:18.281+0000 7f7e9bfff700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e9c06b860 con 0x7f7e9c0c9bb0 2023-10-06T03:30:18.285 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:18.281+0000 7f7e9a7fc700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7e9c0c9bb0 0x7f7e9c0ade90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:18.285 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:18.281+0000 7f7e9a7fc700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7e9c0c9bb0 0x7f7e9c0ade90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3301/0 says I am v2:172.21.15.110:34612/0 (socket says 172.21.15.110:34612) 2023-10-06T03:30:18.285 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:18.281+0000 7f7e9a7fc700 1 -- 172.21.15.110:0/2769237706 learned_addr learned my addr 172.21.15.110:0/2769237706 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:30:18.285 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:18.281+0000 7f7e9b7fe700 1 --2- 172.21.15.110:0/2769237706 >> v2:172.21.15.192:3300/0 conn(0x7f7e9c0aec00 0x7f7e9c0c6540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:18.285 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:18.281+0000 7f7e9affd700 1 --1- 172.21.15.110:0/2769237706 >> v1:172.21.15.192:6789/0 conn(0x7f7e9c0adac0 0x7f7e9c0caa10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.110:52168/0 (socket says 172.21.15.110:52168) 2023-10-06T03:30:18.286 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:18.281+0000 7f7e9a7fc700 1 -- 172.21.15.110:0/2769237706 >> v2:172.21.15.192:3300/0 conn(0x7f7e9c0aec00 msgr2=0x7f7e9c0c6540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:30:18.286 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:18.281+0000 7f7e9a7fc700 1 --2- 172.21.15.110:0/2769237706 >> v2:172.21.15.192:3300/0 conn(0x7f7e9c0aec00 0x7f7e9c0c6540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:18.287 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:18.281+0000 7f7e9a7fc700 1 -- 172.21.15.110:0/2769237706 >> v1:172.21.15.192:6789/0 conn(0x7f7e9c0adac0 legacy=0x7f7e9c0caa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:18.287 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:18.281+0000 7f7e9a7fc700 1 -- 172.21.15.110:0/2769237706 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e9c06bb40 con 0x7f7e9c0c9bb0 2023-10-06T03:30:18.287 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:18.281+0000 7f7e9a7fc700 1 --2- 172.21.15.110:0/2769237706 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7e9c0c9bb0 0x7f7e9c0ade90 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f7e8c009f20 tx=0x7f7e8c009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7b561e30b85993ec server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:30:18.287 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:18.281+0000 7f7e99ffb700 1 -- 172.21.15.110:0/2769237706 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e8c00e070 con 0x7f7e9c0c9bb0 2023-10-06T03:30:18.287 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:18.281+0000 7f7e99ffb700 1 -- 172.21.15.110:0/2769237706 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e8c00b9d0 con 0x7f7e9c0c9bb0 2023-10-06T03:30:18.287 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:18.281+0000 7f7e99ffb700 1 -- 172.21.15.110:0/2769237706 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e8c0092e0 con 0x7f7e9c0c9bb0 2023-10-06T03:30:18.288 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:18.281+0000 7f7e9bfff700 1 -- 172.21.15.110:0/2769237706 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7e9c0c9bb0 msgr2=0x7f7e9c0ade90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:18.288 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:18.281+0000 7f7e9bfff700 1 --2- 172.21.15.110:0/2769237706 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7e9c0c9bb0 0x7f7e9c0ade90 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f7e8c009f20 tx=0x7f7e8c009840 comp rx=0 tx=0).stop 2023-10-06T03:30:18.288 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:18.281+0000 7f7e9bfff700 1 -- 172.21.15.110:0/2769237706 shutdown_connections 2023-10-06T03:30:18.288 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:18.285+0000 7f7e9bfff700 1 --2- 172.21.15.110:0/2769237706 >> v2:172.21.15.192:3300/0 conn(0x7f7e9c0aec00 0x7f7e9c0c6540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:18.288 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:18.285+0000 7f7e9bfff700 1 --2- 172.21.15.110:0/2769237706 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7e9c0c9bb0 0x7f7e9c0ade90 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:18.288 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:18.285+0000 7f7e9bfff700 1 -- 172.21.15.110:0/2769237706 >> 172.21.15.110:0/2769237706 conn(0x7f7e9c0cc470 msgr2=0x7f7e9c0c8410 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:30:18.289 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:18.285+0000 7f7e9bfff700 1 -- 172.21.15.110:0/2769237706 shutdown_connections 2023-10-06T03:30:18.289 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:18.285+0000 7f7e9bfff700 1 -- 172.21.15.110:0/2769237706 wait complete. 2023-10-06T03:30:18.290 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:18.285+0000 7f7e9bfff700 1 Processor -- start 2023-10-06T03:30:18.290 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:18.285+0000 7f7e9bfff700 1 -- start start 2023-10-06T03:30:18.293 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:18.289+0000 7f7e9bfff700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7e9c0adac0 0x7f7e9c0c1a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:18.293 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:18.289+0000 7f7e9bfff700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7e9c0aec00 0x7f7e9c16e360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:18.293 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:18.289+0000 7f7e9bfff700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7e9c0c9bb0 0x7f7e9c159780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:18.293 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:18.289+0000 7f7e9bfff700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e9c16e8a0 con 0x7f7e9c0c9bb0 2023-10-06T03:30:18.293 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:18.289+0000 7f7e9bfff700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e9c0694a0 con 0x7f7e9c0aec00 2023-10-06T03:30:18.293 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:18.289+0000 7f7e9bfff700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e9c069780 con 0x7f7e9c0adac0 2023-10-06T03:30:18.293 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:18.289+0000 7f7e9b7fe700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7e9c0c9bb0 0x7f7e9c159780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:18.293 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:18.289+0000 7f7e9b7fe700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7e9c0c9bb0 0x7f7e9c159780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:52328/0 (socket says 172.21.15.110:52328) 2023-10-06T03:30:18.293 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:18.289+0000 7f7e9b7fe700 1 -- 172.21.15.110:0/1214518085 learned_addr learned my addr 172.21.15.110:0/1214518085 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:30:18.294 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:18.289+0000 7f7e9a7fc700 1 --2- 172.21.15.110:0/1214518085 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7e9c0aec00 0x7f7e9c16e360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:18.294 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:18.289+0000 7f7e9affd700 1 --2- 172.21.15.110:0/1214518085 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7e9c0adac0 0x7f7e9c0c1a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:18.295 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:18.289+0000 7f7e9b7fe700 1 -- 172.21.15.110:0/1214518085 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7e9c0adac0 msgr2=0x7f7e9c0c1a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:18.295 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:18.289+0000 7f7e9b7fe700 1 --2- 172.21.15.110:0/1214518085 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7e9c0adac0 0x7f7e9c0c1a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:18.295 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:18.289+0000 7f7e9b7fe700 1 -- 172.21.15.110:0/1214518085 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7e9c0aec00 msgr2=0x7f7e9c16e360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:18.295 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:18.289+0000 7f7e9b7fe700 1 --2- 172.21.15.110:0/1214518085 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7e9c0aec00 0x7f7e9c16e360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:18.295 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:18.289+0000 7f7e9b7fe700 1 -- 172.21.15.110:0/1214518085 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e8c009ae0 con 0x7f7e9c0c9bb0 2023-10-06T03:30:18.295 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:18.289+0000 7f7e9b7fe700 1 --2- 172.21.15.110:0/1214518085 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7e9c0c9bb0 0x7f7e9c159780 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f7e9400ea50 tx=0x7f7e940091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:30:18.296 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:18.289+0000 7f7e92ffd700 1 -- 172.21.15.110:0/1214518085 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e940096b0 con 0x7f7e9c0c9bb0 2023-10-06T03:30:18.296 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:18.289+0000 7f7e92ffd700 1 -- 172.21.15.110:0/1214518085 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e9400c850 con 0x7f7e9c0c9bb0 2023-10-06T03:30:18.296 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:18.289+0000 7f7e9bfff700 1 -- 172.21.15.110:0/1214518085 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e9c069a60 con 0x7f7e9c0c9bb0 2023-10-06T03:30:18.296 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:18.289+0000 7f7e92ffd700 1 -- 172.21.15.110:0/1214518085 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e9400cc10 con 0x7f7e9c0c9bb0 2023-10-06T03:30:18.296 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:18.289+0000 7f7e9bfff700 1 -- 172.21.15.110:0/1214518085 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e9c06a020 con 0x7f7e9c0c9bb0 2023-10-06T03:30:18.296 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:18.289+0000 7f7e9bfff700 1 -- 172.21.15.110:0/1214518085 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e9c070ff0 con 0x7f7e9c0c9bb0 2023-10-06T03:30:18.298 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:18.293+0000 7f7e92ffd700 1 -- 172.21.15.110:0/1214518085 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7f7e94013470 con 0x7f7e9c0c9bb0 2023-10-06T03:30:18.299 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:18.293+0000 7f7e92ffd700 1 --2- 172.21.15.110:0/1214518085 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f7e7c060550 0x7f7e7c062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:18.299 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:18.297+0000 7f7e92ffd700 1 -- 172.21.15.110:0/1214518085 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 6867+0+0 (secure 0 0 0) 0x7f7e9407d590 con 0x7f7e9c0c9bb0 2023-10-06T03:30:18.299 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:18.297+0000 7f7e9affd700 1 --2- 172.21.15.110:0/1214518085 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f7e7c060550 0x7f7e7c062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:18.300 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:18.297+0000 7f7e9affd700 1 --2- 172.21.15.110:0/1214518085 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f7e7c060550 0x7f7e7c062a00 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f7e9c0c23a0 tx=0x7f7e84000bc0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:30:18.303 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:18.297+0000 7f7e92ffd700 1 -- 172.21.15.110:0/1214518085 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f7e9404d770 con 0x7f7e9c0c9bb0 2023-10-06T03:30:18.500 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:18.497+0000 7f7e9bfff700 1 -- 172.21.15.110:0/1214518085 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7e9c0c9920 con 0x7f7e9c0c9bb0 2023-10-06T03:30:18.500 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:18.497+0000 7f7e92ffd700 1 -- 172.21.15.110:0/1214518085 <== mon.0 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4602 (secure 0 0 0) 0x7f7e94050d90 con 0x7f7e9c0c9bb0 2023-10-06T03:30:18.501 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-10-06T03:30:18.501 INFO:teuthology.orchestra.run.smithi110.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4279,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6833/1434505700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6832","nonce":1434505700},{"type":"v1","addr":"172.21.15.192:6833","nonce":1434505700}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4281,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/1168370789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6834","nonce":1168370789},{"type":"v1","addr":"172.21.15.110:6835","nonce":1168370789}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4288,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6835/1271810086","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":1271810086},{"type":"v1","addr":"172.21.15.192:6835","nonce":1271810086}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-06T03:29:33.894998+0000","modified":"2023-10-06T03:29:47.307075+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4293},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4293":{"gid":4293,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.110:6837/540320464","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6836","nonce":540320464},{"type":"v1","addr":"172.21.15.110:6837","nonce":540320464}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-06T03:30:18.503 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:18.497+0000 7f7e9bfff700 1 -- 172.21.15.110:0/1214518085 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f7e7c060550 msgr2=0x7f7e7c062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:18.504 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:18.497+0000 7f7e9bfff700 1 --2- 172.21.15.110:0/1214518085 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f7e7c060550 0x7f7e7c062a00 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f7e9c0c23a0 tx=0x7f7e84000bc0 comp rx=0 tx=0).stop 2023-10-06T03:30:18.504 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:18.497+0000 7f7e9bfff700 1 -- 172.21.15.110:0/1214518085 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7e9c0c9bb0 msgr2=0x7f7e9c159780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:18.504 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:18.497+0000 7f7e9bfff700 1 --2- 172.21.15.110:0/1214518085 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7e9c0c9bb0 0x7f7e9c159780 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f7e9400ea50 tx=0x7f7e940091a0 comp rx=0 tx=0).stop 2023-10-06T03:30:18.504 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:18.501+0000 7f7e9bfff700 1 -- 172.21.15.110:0/1214518085 shutdown_connections 2023-10-06T03:30:18.504 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:18.501+0000 7f7e9bfff700 1 --2- 172.21.15.110:0/1214518085 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f7e7c060550 0x7f7e7c062a00 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:18.504 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:18.501+0000 7f7e9bfff700 1 --2- 172.21.15.110:0/1214518085 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7e9c0c9bb0 0x7f7e9c159780 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:18.504 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:18.501+0000 7f7e9bfff700 1 --2- 172.21.15.110:0/1214518085 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7e9c0aec00 0x7f7e9c16e360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:18.504 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:18.501+0000 7f7e9bfff700 1 --2- 172.21.15.110:0/1214518085 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7e9c0adac0 0x7f7e9c0c1a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:18.504 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:18.501+0000 7f7e9bfff700 1 -- 172.21.15.110:0/1214518085 >> 172.21.15.110:0/1214518085 conn(0x7f7e9c0cc470 msgr2=0x7f7e9c15b7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:30:18.504 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:18.501+0000 7f7e9bfff700 1 -- 172.21.15.110:0/1214518085 shutdown_connections 2023-10-06T03:30:18.504 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:18.501+0000 7f7e9bfff700 1 -- 172.21.15.110:0/1214518085 wait complete. 2023-10-06T03:30:18.505 INFO:teuthology.orchestra.run.smithi110.stderr:dumped fsmap epoch 8 2023-10-06T03:30:18.515 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-06T03:30:18.614 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:18.609+0000 7fce69f27700 1 Processor -- start 2023-10-06T03:30:18.614 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:18.609+0000 7fce69f27700 1 -- start start 2023-10-06T03:30:18.616 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:18.613+0000 7fce69f27700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fce64156210 0x7fce640af2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:18.616 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:18.613+0000 7fce69f27700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fce640c7380 0x7fce640c7800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:18.616 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:18.613+0000 7fce69f27700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce640aee40 con 0x7fce64155e40 2023-10-06T03:30:18.616 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:18.613+0000 7fce69f27700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fce640af810 con 0x7fce640c7380 2023-10-06T03:30:18.616 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:18.613+0000 7fce69f27700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce640c9670 con 0x7fce64156210 2023-10-06T03:30:18.616 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:18.613+0000 7fce637fe700 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7fce64155e40 0x7fce640c8f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:53300/0 (socket says 172.21.15.110:53300) 2023-10-06T03:30:18.616 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:18.613+0000 7fce637fe700 1 -- 172.21.15.110:0/3570395518 learned_addr learned my addr 172.21.15.110:0/3570395518 (peer_addr_for_me v1:172.21.15.110:0/0) 2023-10-06T03:30:18.616 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:18.613+0000 7fce63fff700 1 --2- 172.21.15.110:0/3570395518 >> v2:172.21.15.192:3300/0 conn(0x7fce640c7380 0x7fce640c7800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:18.617 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:18.613+0000 7fce62ffd700 1 --2- 172.21.15.110:0/3570395518 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fce64156210 0x7fce640af2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:18.617 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:18.613+0000 7fce627fc700 1 -- 172.21.15.110:0/3570395518 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 823413539 0 0) 0x7fce640aee40 con 0x7fce64155e40 2023-10-06T03:30:18.618 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:18.613+0000 7fce627fc700 1 -- 172.21.15.110:0/3570395518 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fce50003570 con 0x7fce64155e40 2023-10-06T03:30:18.618 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:18.613+0000 7fce63fff700 1 -- 172.21.15.110:0/3570395518 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fce64156210 msgr2=0x7fce640af2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:30:18.618 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:18.613+0000 7fce63fff700 1 --2- 172.21.15.110:0/3570395518 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fce64156210 0x7fce640af2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:18.618 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:18.613+0000 7fce63fff700 1 -- 172.21.15.110:0/3570395518 >> v1:172.21.15.110:6789/0 conn(0x7fce64155e40 legacy=0x7fce640c8f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:18.618 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:18.613+0000 7fce63fff700 1 -- 172.21.15.110:0/3570395518 --> v2:172.21.15.192:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce640ae230 con 0x7fce640c7380 2023-10-06T03:30:18.619 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:18.613+0000 7fce63fff700 1 --2- 172.21.15.110:0/3570395518 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fce640c7380 0x7fce640c7800 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fce540098e0 tx=0x7fce540091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6ce81b5f411b3e6a server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:30:18.620 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:18.617+0000 7fce627fc700 1 -- 172.21.15.110:0/3570395518 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce5400f6b0 con 0x7fce640c7380 2023-10-06T03:30:18.620 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:18.617+0000 7fce627fc700 1 -- 172.21.15.110:0/3570395518 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce5400f810 con 0x7fce640c7380 2023-10-06T03:30:18.620 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:18.617+0000 7fce627fc700 1 -- 172.21.15.110:0/3570395518 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce5400fbb0 con 0x7fce640c7380 2023-10-06T03:30:18.621 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:18.617+0000 7fce69f27700 1 -- 172.21.15.110:0/3570395518 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fce640c7380 msgr2=0x7fce640c7800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:18.621 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:18.617+0000 7fce69f27700 1 --2- 172.21.15.110:0/3570395518 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fce640c7380 0x7fce640c7800 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fce540098e0 tx=0x7fce540091a0 comp rx=0 tx=0).stop 2023-10-06T03:30:18.622 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:18.617+0000 7fce69f27700 1 -- 172.21.15.110:0/3570395518 shutdown_connections 2023-10-06T03:30:18.622 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:18.617+0000 7fce69f27700 1 --2- 172.21.15.110:0/3570395518 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fce640c7380 0x7fce640c7800 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:18.622 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:18.617+0000 7fce69f27700 1 --2- 172.21.15.110:0/3570395518 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fce64156210 0x7fce640af2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:18.622 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:18.617+0000 7fce69f27700 1 -- 172.21.15.110:0/3570395518 >> 172.21.15.110:0/3570395518 conn(0x7fce640cd860 msgr2=0x7fce640c9850 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:30:18.622 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:18.617+0000 7fce69f27700 1 -- 172.21.15.110:0/3570395518 shutdown_connections 2023-10-06T03:30:18.622 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:18.617+0000 7fce69f27700 1 -- 172.21.15.110:0/3570395518 wait complete. 2023-10-06T03:30:18.623 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:18.617+0000 7fce69f27700 1 Processor -- start 2023-10-06T03:30:18.623 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:18.621+0000 7fce69f27700 1 -- start start 2023-10-06T03:30:18.625 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:18.621+0000 7fce69f27700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fce640c7380 0x7fce64158170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:18.625 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:18.621+0000 7fce69f27700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fce64155e40 0x7fce6416c650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:18.625 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:18.621+0000 7fce69f27700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fce64156210 0x7fce6416cb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:18.625 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:18.621+0000 7fce69f27700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce641572e0 con 0x7fce64156210 2023-10-06T03:30:18.626 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:18.621+0000 7fce69f27700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce64159e30 con 0x7fce640c7380 2023-10-06T03:30:18.626 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:18.621+0000 7fce69f27700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce64159220 con 0x7fce64155e40 2023-10-06T03:30:18.626 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:18.621+0000 7fce63fff700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fce64156210 0x7fce6416cb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:18.626 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:18.621+0000 7fce637fe700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fce640c7380 0x7fce64158170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:18.626 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:18.621+0000 7fce63fff700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fce64156210 0x7fce6416cb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:52342/0 (socket says 172.21.15.110:52342) 2023-10-06T03:30:18.626 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:18.621+0000 7fce63fff700 1 -- 172.21.15.110:0/972499360 learned_addr learned my addr 172.21.15.110:0/972499360 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:30:18.626 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:18.621+0000 7fce62ffd700 1 --2- 172.21.15.110:0/972499360 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fce64155e40 0x7fce6416c650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:18.626 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:18.621+0000 7fce63fff700 1 -- 172.21.15.110:0/972499360 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fce64155e40 msgr2=0x7fce6416c650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:18.626 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:18.621+0000 7fce63fff700 1 --2- 172.21.15.110:0/972499360 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fce64155e40 0x7fce6416c650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:18.626 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:18.621+0000 7fce63fff700 1 -- 172.21.15.110:0/972499360 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fce640c7380 msgr2=0x7fce64158170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:18.627 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:18.621+0000 7fce63fff700 1 --2- 172.21.15.110:0/972499360 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fce640c7380 0x7fce64158170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:18.627 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:18.621+0000 7fce63fff700 1 -- 172.21.15.110:0/972499360 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce54009420 con 0x7fce64156210 2023-10-06T03:30:18.627 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:18.621+0000 7fce63fff700 1 --2- 172.21.15.110:0/972499360 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fce64156210 0x7fce6416cb90 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fce64069b90 tx=0x7fce54019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:30:18.628 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:18.621+0000 7fce5b7fe700 1 -- 172.21.15.110:0/972499360 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce5401ae50 con 0x7fce64156210 2023-10-06T03:30:18.628 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:18.621+0000 7fce5b7fe700 1 -- 172.21.15.110:0/972499360 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce54010470 con 0x7fce64156210 2023-10-06T03:30:18.628 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:18.621+0000 7fce69f27700 1 -- 172.21.15.110:0/972499360 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce64159500 con 0x7fce64156210 2023-10-06T03:30:18.628 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:18.621+0000 7fce5b7fe700 1 -- 172.21.15.110:0/972499360 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce54010830 con 0x7fce64156210 2023-10-06T03:30:18.628 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:18.621+0000 7fce69f27700 1 -- 172.21.15.110:0/972499360 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce64159ac0 con 0x7fce64156210 2023-10-06T03:30:18.628 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:18.625+0000 7fce69f27700 1 -- 172.21.15.110:0/972499360 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce480052b0 con 0x7fce64156210 2023-10-06T03:30:18.628 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:18.625+0000 7fce5b7fe700 1 -- 172.21.15.110:0/972499360 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7fce54010c20 con 0x7fce64156210 2023-10-06T03:30:18.629 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:18.625+0000 7fce5b7fe700 1 --2- 172.21.15.110:0/972499360 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fce44060620 0x7fce44062ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:18.629 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:18.625+0000 7fce5b7fe700 1 -- 172.21.15.110:0/972499360 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 6867+0+0 (secure 0 0 0) 0x7fce54014070 con 0x7fce64156210 2023-10-06T03:30:18.629 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:18.625+0000 7fce637fe700 1 --2- 172.21.15.110:0/972499360 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fce44060620 0x7fce44062ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:18.629 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:18.625+0000 7fce637fe700 1 --2- 172.21.15.110:0/972499360 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fce44060620 0x7fce44062ad0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fce64155a40 tx=0x7fce5c009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:30:18.632 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:18.629+0000 7fce5b7fe700 1 -- 172.21.15.110:0/972499360 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fce5404e980 con 0x7fce64156210 2023-10-06T03:30:18.792 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:18.789+0000 7fce69f27700 1 -- 172.21.15.110:0/972499360 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fce48005830 con 0x7fce64156210 2023-10-06T03:30:18.794 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:18.789+0000 7fce5b7fe700 1 -- 172.21.15.110:0/972499360 <== mon.0 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v30) v1 ==== 74+0+18850 (secure 0 0 0) 0x7fce54051fa0 con 0x7fce64156210 2023-10-06T03:30:18.794 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-10-06T03:30:18.795 INFO:teuthology.orchestra.run.smithi110.stdout:{"epoch":30,"fsid":"93b8f6ac-4c92-4b45-b8e4-b59617024aa2","created":"2023-10-06T03:29:10.519583+0000","modified":"2023-10-06T03:30:16.473472+0000","last_up_change":"2023-10-06T03:29:21.960511+0000","last_in_change":"2023-10-06T03:29:12.560436+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-06T03:29:20.703743+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2023-10-06T03:29:28.307281+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2023-10-06T03:29:32.247251+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data","create_time":"2023-10-06T03:29:32.880086+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}}},{"pool":5,"pool_name":"cephfs_data_ec","create_time":"2023-10-06T03:29:36.613779+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"28","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"5866eb6c-2adc-46e4-be7a-e5bcda0f4065","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6824","nonce":33394},{"type":"v1","addr":"172.21.15.110:6825","nonce":33394}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6826","nonce":33394},{"type":"v1","addr":"172.21.15.110:6827","nonce":33394}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6830","nonce":33394},{"type":"v1","addr":"172.21.15.110:6831","nonce":33394}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6828","nonce":33394},{"type":"v1","addr":"172.21.15.110:6829","nonce":33394}]},"public_addr":"172.21.15.110:6825/33394","cluster_addr":"172.21.15.110:6827/33394","heartbeat_back_addr":"172.21.15.110:6831/33394","heartbeat_front_addr":"172.21.15.110:6829/33394","state":["exists","up"]},{"osd":1,"uuid":"a38de118-062b-4a05-9782-2ceddf6ff4df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6800","nonce":33396},{"type":"v1","addr":"172.21.15.110:6801","nonce":33396}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6802","nonce":33396},{"type":"v1","addr":"172.21.15.110:6803","nonce":33396}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6806","nonce":33396},{"type":"v1","addr":"172.21.15.110:6807","nonce":33396}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6804","nonce":33396},{"type":"v1","addr":"172.21.15.110:6805","nonce":33396}]},"public_addr":"172.21.15.110:6801/33396","cluster_addr":"172.21.15.110:6803/33396","heartbeat_back_addr":"172.21.15.110:6807/33396","heartbeat_front_addr":"172.21.15.110:6805/33396","state":["exists","up"]},{"osd":2,"uuid":"56e39be2-3407-4b69-877e-cf16617b06d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6816","nonce":33397},{"type":"v1","addr":"172.21.15.110:6817","nonce":33397}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6818","nonce":33397},{"type":"v1","addr":"172.21.15.110:6819","nonce":33397}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6822","nonce":33397},{"type":"v1","addr":"172.21.15.110:6823","nonce":33397}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6820","nonce":33397},{"type":"v1","addr":"172.21.15.110:6821","nonce":33397}]},"public_addr":"172.21.15.110:6817/33397","cluster_addr":"172.21.15.110:6819/33397","heartbeat_back_addr":"172.21.15.110:6823/33397","heartbeat_front_addr":"172.21.15.110:6821/33397","state":["exists","up"]},{"osd":3,"uuid":"01c82d13-c1d4-4bde-b55e-3e0089b40115","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6808","nonce":33395},{"type":"v1","addr":"172.21.15.110:6809","nonce":33395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6810","nonce":33395},{"type":"v1","addr":"172.21.15.110:6811","nonce":33395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6814","nonce":33395},{"type":"v1","addr":"172.21.15.110:6815","nonce":33395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6812","nonce":33395},{"type":"v1","addr":"172.21.15.110:6813","nonce":33395}]},"public_addr":"172.21.15.110:6809/33395","cluster_addr":"172.21.15.110:6811/33395","heartbeat_back_addr":"172.21.15.110:6815/33395","heartbeat_front_addr":"172.21.15.110:6813/33395","state":["exists","up"]},{"osd":4,"uuid":"86983430-5fa3-4d17-998f-03e2cd843237","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":33683},{"type":"v1","addr":"172.21.15.192:6817","nonce":33683}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":33683},{"type":"v1","addr":"172.21.15.192:6819","nonce":33683}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":33683},{"type":"v1","addr":"172.21.15.192:6823","nonce":33683}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6820","nonce":33683},{"type":"v1","addr":"172.21.15.192:6821","nonce":33683}]},"public_addr":"172.21.15.192:6817/33683","cluster_addr":"172.21.15.192:6819/33683","heartbeat_back_addr":"172.21.15.192:6823/33683","heartbeat_front_addr":"172.21.15.192:6821/33683","state":["exists","up"]},{"osd":5,"uuid":"6945257f-821d-4619-a63f-ebe857eba6f4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":33684},{"type":"v1","addr":"172.21.15.192:6801","nonce":33684}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":33684},{"type":"v1","addr":"172.21.15.192:6803","nonce":33684}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":33684},{"type":"v1","addr":"172.21.15.192:6807","nonce":33684}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6804","nonce":33684},{"type":"v1","addr":"172.21.15.192:6805","nonce":33684}]},"public_addr":"172.21.15.192:6801/33684","cluster_addr":"172.21.15.192:6803/33684","heartbeat_back_addr":"172.21.15.192:6807/33684","heartbeat_front_addr":"172.21.15.192:6805/33684","state":["exists","up"]},{"osd":6,"uuid":"7854d1ae-7060-4780-8f97-73cea5e7a12f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6824","nonce":33686},{"type":"v1","addr":"172.21.15.192:6825","nonce":33686}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":33686},{"type":"v1","addr":"172.21.15.192:6827","nonce":33686}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":33686},{"type":"v1","addr":"172.21.15.192:6831","nonce":33686}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6828","nonce":33686},{"type":"v1","addr":"172.21.15.192:6829","nonce":33686}]},"public_addr":"172.21.15.192:6825/33686","cluster_addr":"172.21.15.192:6827/33686","heartbeat_back_addr":"172.21.15.192:6831/33686","heartbeat_front_addr":"172.21.15.192:6829/33686","state":["exists","up"]},{"osd":7,"uuid":"f1ebdfd5-b859-482b-9f97-c18c4a26f445","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6808","nonce":33685},{"type":"v1","addr":"172.21.15.192:6809","nonce":33685}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":33685},{"type":"v1","addr":"172.21.15.192:6811","nonce":33685}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":33685},{"type":"v1","addr":"172.21.15.192:6815","nonce":33685}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6812","nonce":33685},{"type":"v1","addr":"172.21.15.192:6813","nonce":33685}]},"public_addr":"172.21.15.192:6809/33685","cluster_addr":"172.21.15.192:6811/33685","heartbeat_back_addr":"172.21.15.192:6815/33685","heartbeat_front_addr":"172.21.15.192:6813/33685","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T03:29:18.002048+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T03:29:17.478576+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T03:29:18.034944+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T03:29:17.628513+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T03:29:18.047338+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T03:29:17.772349+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T03:29:17.883958+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T03:29:18.151838+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"4.7","mappings":[{"from":1,"to":2}]},{"pgid":"4.3b","mappings":[{"from":3,"to":2}]}],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-06T03:30:18.799 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:18.793+0000 7fce69f27700 1 -- 172.21.15.110:0/972499360 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fce44060620 msgr2=0x7fce44062ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:18.799 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:18.793+0000 7fce69f27700 1 --2- 172.21.15.110:0/972499360 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fce44060620 0x7fce44062ad0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fce64155a40 tx=0x7fce5c009040 comp rx=0 tx=0).stop 2023-10-06T03:30:18.799 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:18.793+0000 7fce69f27700 1 -- 172.21.15.110:0/972499360 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fce64156210 msgr2=0x7fce6416cb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:18.800 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:18.793+0000 7fce69f27700 1 --2- 172.21.15.110:0/972499360 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fce64156210 0x7fce6416cb90 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fce64069b90 tx=0x7fce54019000 comp rx=0 tx=0).stop 2023-10-06T03:30:18.800 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:18.793+0000 7fce69f27700 1 -- 172.21.15.110:0/972499360 shutdown_connections 2023-10-06T03:30:18.800 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:18.793+0000 7fce69f27700 1 --2- 172.21.15.110:0/972499360 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fce44060620 0x7fce44062ad0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:18.800 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:18.793+0000 7fce69f27700 1 --2- 172.21.15.110:0/972499360 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fce64156210 0x7fce6416cb90 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:18.800 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:18.793+0000 7fce69f27700 1 --2- 172.21.15.110:0/972499360 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fce64155e40 0x7fce6416c650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:18.800 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:18.793+0000 7fce69f27700 1 --2- 172.21.15.110:0/972499360 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fce640c7380 0x7fce64158170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:18.800 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:18.793+0000 7fce69f27700 1 -- 172.21.15.110:0/972499360 >> 172.21.15.110:0/972499360 conn(0x7fce640cd860 msgr2=0x7fce640c8c00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:30:18.800 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:18.793+0000 7fce69f27700 1 -- 172.21.15.110:0/972499360 shutdown_connections 2023-10-06T03:30:18.800 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:18.793+0000 7fce69f27700 1 -- 172.21.15.110:0/972499360 wait complete. 2023-10-06T03:30:18.814 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-06T03:29:20.703743+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-10-06T03:29:28.307281+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 3, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-06T03:29:32.247251+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 4, 'pool_name': 'cephfs_data', 'create_time': '2023-10-06T03:29:32.880086+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}, {'pool': 5, 'pool_name': 'cephfs_data_ec', 'create_time': '2023-10-06T03:29:36.613779+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '28', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-10-06T03:30:18.815 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-06T03:30:18.924 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:18.921+0000 7f4eae1da700 1 Processor -- start 2023-10-06T03:30:18.924 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:18.921+0000 7f4eae1da700 1 -- start start 2023-10-06T03:30:18.925 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:18.921+0000 7f4eae1da700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4ea815d2a0 0x7f4ea806cab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:18.925 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:18.921+0000 7f4eae1da700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ea8069ef0 con 0x7f4ea806b7c0 2023-10-06T03:30:18.925 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:18.921+0000 7f4eae1da700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ea80b0010 con 0x7f4ea80c97b0 2023-10-06T03:30:18.926 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:18.921+0000 7f4eae1da700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ea806a140 con 0x7f4ea815d2a0 2023-10-06T03:30:18.926 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:18.921+0000 7f4eac9d7700 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f4ea806b7c0 0x7f4ea80af1f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:53302/0 (socket says 172.21.15.110:53302) 2023-10-06T03:30:18.926 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:18.921+0000 7f4eac9d7700 1 -- 172.21.15.110:0/613364681 learned_addr learned my addr 172.21.15.110:0/613364681 (peer_addr_for_me v1:172.21.15.110:0/0) 2023-10-06T03:30:18.926 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:18.921+0000 7f4ead1d8700 1 --2- 172.21.15.110:0/613364681 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4ea815d2a0 0x7f4ea806cab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:18.929 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:18.921+0000 7f4e9bfff700 1 -- 172.21.15.110:0/613364681 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1099562733 0 0) 0x7f4ea8069ef0 con 0x7f4ea806b7c0 2023-10-06T03:30:18.929 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:18.921+0000 7f4e9bfff700 1 -- 172.21.15.110:0/613364681 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4e94003570 con 0x7f4ea806b7c0 2023-10-06T03:30:18.929 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:18.921+0000 7f4e9bfff700 1 -- 172.21.15.110:0/613364681 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2994679084 0 0) 0x7f4ea80b0010 con 0x7f4ea80c97b0 2023-10-06T03:30:18.930 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:18.921+0000 7f4e9bfff700 1 -- 172.21.15.110:0/613364681 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4ea8069ef0 con 0x7f4ea80c97b0 2023-10-06T03:30:18.930 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:18.921+0000 7f4e9bfff700 1 -- 172.21.15.110:0/613364681 <== mon.0 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1356848431 0 0) 0x7f4e94003570 con 0x7f4ea806b7c0 2023-10-06T03:30:18.930 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:18.921+0000 7f4e9bfff700 1 -- 172.21.15.110:0/613364681 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4ea815d2a0 msgr2=0x7f4ea806cab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:30:18.930 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:18.921+0000 7f4e9bfff700 1 --2- 172.21.15.110:0/613364681 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4ea815d2a0 0x7f4ea806cab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:18.930 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:18.921+0000 7f4e9bfff700 1 -- 172.21.15.110:0/613364681 >> v1:172.21.15.192:6789/0 conn(0x7f4ea80c97b0 legacy=0x7f4ea80af900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:18.930 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:18.921+0000 7f4e9bfff700 1 -- 172.21.15.110:0/613364681 --> v1:172.21.15.110:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ea806a420 con 0x7f4ea806b7c0 2023-10-06T03:30:18.930 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:18.921+0000 7f4e9bfff700 1 -- 172.21.15.110:0/613364681 <== mon.0 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 118603703 0 0) 0x7f4ea4001470 con 0x7f4ea806b7c0 2023-10-06T03:30:18.930 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:18.921+0000 7f4ead1d8700 1 --2- 172.21.15.110:0/613364681 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4ea815d2a0 0x7f4ea806cab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-06T03:30:18.930 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:18.921+0000 7f4e9bfff700 1 -- 172.21.15.110:0/613364681 >> v1:172.21.15.110:6789/0 conn(0x7f4ea806b7c0 legacy=0x7f4ea80af1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:18.930 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:18.921+0000 7f4e9bfff700 1 --2- 172.21.15.110:0/613364681 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4e94003c20 0x7f4e94004010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:18.931 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:18.921+0000 7f4e9bfff700 1 --2- 172.21.15.110:0/613364681 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4e94004810 0x7f4e94004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:18.931 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:18.921+0000 7f4e9bfff700 1 --2- 172.21.15.110:0/613364681 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4ea815d2a0 0x7f4e940051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:18.931 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:18.921+0000 7f4e9bfff700 1 -- 172.21.15.110:0/613364681 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ea806a140 con 0x7f4e94003c20 2023-10-06T03:30:18.931 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:18.921+0000 7f4e9bfff700 1 -- 172.21.15.110:0/613364681 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e9400bb10 con 0x7f4e94004810 2023-10-06T03:30:18.931 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:18.921+0000 7f4e9bfff700 1 -- 172.21.15.110:0/613364681 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e9400bcd0 con 0x7f4ea815d2a0 2023-10-06T03:30:18.931 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:18.921+0000 7f4ead1d8700 1 --2- 172.21.15.110:0/613364681 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4e94003c20 0x7f4e94004010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:18.931 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:18.921+0000 7f4eac9d7700 1 --2- 172.21.15.110:0/613364681 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4e94004810 0x7f4e94004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:18.931 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:18.921+0000 7f4ead9d9700 1 --2- 172.21.15.110:0/613364681 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4ea815d2a0 0x7f4e940051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:18.931 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:18.925+0000 7f4ead1d8700 1 -- 172.21.15.110:0/613364681 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4ea815d2a0 msgr2=0x7f4e940051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:30:18.931 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:18.925+0000 7f4ead1d8700 1 --2- 172.21.15.110:0/613364681 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4ea815d2a0 0x7f4e940051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:18.931 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:18.925+0000 7f4ead1d8700 1 -- 172.21.15.110:0/613364681 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4e94004810 msgr2=0x7f4e94004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:30:18.931 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:18.925+0000 7f4ead1d8700 1 --2- 172.21.15.110:0/613364681 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4e94004810 0x7f4e94004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:18.932 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:18.925+0000 7f4ead1d8700 1 -- 172.21.15.110:0/613364681 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e9c009420 con 0x7f4e94003c20 2023-10-06T03:30:18.932 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:18.925+0000 7f4ead1d8700 1 --2- 172.21.15.110:0/613364681 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4e94003c20 0x7f4e94004010 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f4e9c009df0 tx=0x7f4e9c0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=26661756d774ad30 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:30:18.932 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:18.925+0000 7f4e9bfff700 1 -- 172.21.15.110:0/613364681 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e9c009740 con 0x7f4e94003c20 2023-10-06T03:30:18.932 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:18.925+0000 7f4e9bfff700 1 -- 172.21.15.110:0/613364681 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e9c00e920 con 0x7f4e94003c20 2023-10-06T03:30:18.932 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:18.925+0000 7f4e9bfff700 1 -- 172.21.15.110:0/613364681 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e9c00ece0 con 0x7f4e94003c20 2023-10-06T03:30:18.932 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:18.925+0000 7f4eae1da700 1 -- 172.21.15.110:0/613364681 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4e94003c20 msgr2=0x7f4e94004010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:18.932 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:18.925+0000 7f4eae1da700 1 --2- 172.21.15.110:0/613364681 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4e94003c20 0x7f4e94004010 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f4e9c009df0 tx=0x7f4e9c0091a0 comp rx=0 tx=0).stop 2023-10-06T03:30:18.932 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:18.925+0000 7f4ead9d9700 1 -- 172.21.15.110:0/613364681 reap_dead start 2023-10-06T03:30:18.932 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:18.925+0000 7f4eae1da700 1 -- 172.21.15.110:0/613364681 shutdown_connections 2023-10-06T03:30:18.932 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:18.925+0000 7f4eae1da700 1 -- 172.21.15.110:0/613364681 >> 172.21.15.110:0/613364681 conn(0x7f4ea80cbac0 msgr2=0x7f4ea80cbec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:30:18.932 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:18.925+0000 7f4eae1da700 1 -- 172.21.15.110:0/613364681 shutdown_connections 2023-10-06T03:30:18.932 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:18.925+0000 7f4eae1da700 1 -- 172.21.15.110:0/613364681 wait complete. 2023-10-06T03:30:18.935 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:18.925+0000 7f4eae1da700 1 Processor -- start 2023-10-06T03:30:18.935 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:18.925+0000 7f4eae1da700 1 -- start start 2023-10-06T03:30:18.935 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:18.929+0000 7f4eae1da700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4e94003c20 0x7f4ea815d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:18.935 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:18.929+0000 7f4eae1da700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4ea815dde0 0x7f4ea81505e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:18.935 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:18.929+0000 7f4eae1da700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4ea8150b20 0x7f4ea815f680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:18.935 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:18.929+0000 7f4eae1da700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ea8150fc0 con 0x7f4ea815dde0 2023-10-06T03:30:18.935 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:18.929+0000 7f4eae1da700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ea8161100 con 0x7f4ea8150b20 2023-10-06T03:30:18.935 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:18.929+0000 7f4eae1da700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ea8161960 con 0x7f4e94003c20 2023-10-06T03:30:18.935 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:18.929+0000 7f4eac9d7700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4ea815dde0 0x7f4ea81505e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:18.935 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:18.929+0000 7f4eac9d7700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4ea815dde0 0x7f4ea81505e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:52352/0 (socket says 172.21.15.110:52352) 2023-10-06T03:30:18.936 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:18.929+0000 7f4eac9d7700 1 -- 172.21.15.110:0/1833962967 learned_addr learned my addr 172.21.15.110:0/1833962967 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:30:18.936 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:18.929+0000 7f4ead9d9700 1 --2- 172.21.15.110:0/1833962967 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4ea8150b20 0x7f4ea815f680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:18.936 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:18.929+0000 7f4ead1d8700 1 --2- 172.21.15.110:0/1833962967 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4e94003c20 0x7f4ea815d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:18.936 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:18.929+0000 7f4eac9d7700 1 -- 172.21.15.110:0/1833962967 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4e94003c20 msgr2=0x7f4ea815d8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:18.936 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:18.929+0000 7f4eac9d7700 1 --2- 172.21.15.110:0/1833962967 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4e94003c20 0x7f4ea815d8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:18.936 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:18.929+0000 7f4eac9d7700 1 -- 172.21.15.110:0/1833962967 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4ea8150b20 msgr2=0x7f4ea815f680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:18.936 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:18.929+0000 7f4eac9d7700 1 --2- 172.21.15.110:0/1833962967 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4ea8150b20 0x7f4ea815f680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:18.936 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:18.929+0000 7f4eac9d7700 1 -- 172.21.15.110:0/1833962967 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ea81603d0 con 0x7f4ea815dde0 2023-10-06T03:30:18.936 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:18.929+0000 7f4ead9d9700 1 --2- 172.21.15.110:0/1833962967 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4ea8150b20 0x7f4ea815f680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-06T03:30:18.936 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:18.929+0000 7f4eac9d7700 1 --2- 172.21.15.110:0/1833962967 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4ea815dde0 0x7f4ea81505e0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f4ea4000fd0 tx=0x7f4ea4001e10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:30:18.936 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:18.929+0000 7f4e997fa700 1 -- 172.21.15.110:0/1833962967 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ea4018030 con 0x7f4ea815dde0 2023-10-06T03:30:18.936 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:18.929+0000 7f4e997fa700 1 -- 172.21.15.110:0/1833962967 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ea400f9a0 con 0x7f4ea815dde0 2023-10-06T03:30:18.937 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:18.929+0000 7f4eae1da700 1 -- 172.21.15.110:0/1833962967 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ea8161310 con 0x7f4ea815dde0 2023-10-06T03:30:18.937 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:18.929+0000 7f4e997fa700 1 -- 172.21.15.110:0/1833962967 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ea400fd40 con 0x7f4ea815dde0 2023-10-06T03:30:18.937 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:18.929+0000 7f4eae1da700 1 -- 172.21.15.110:0/1833962967 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ea815fbc0 con 0x7f4ea815dde0 2023-10-06T03:30:18.938 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:18.929+0000 7f4eae1da700 1 -- 172.21.15.110:0/1833962967 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ea8070ff0 con 0x7f4ea815dde0 2023-10-06T03:30:18.938 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:18.933+0000 7f4e997fa700 1 -- 172.21.15.110:0/1833962967 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7f4ea401b070 con 0x7f4ea815dde0 2023-10-06T03:30:18.938 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:18.933+0000 7f4e997fa700 1 --2- 172.21.15.110:0/1833962967 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f4e8c060550 0x7f4e8c062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:18.938 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:18.933+0000 7f4e997fa700 1 -- 172.21.15.110:0/1833962967 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 6867+0+0 (secure 0 0 0) 0x7f4ea4019070 con 0x7f4ea815dde0 2023-10-06T03:30:18.938 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:18.933+0000 7f4ead1d8700 1 --2- 172.21.15.110:0/1833962967 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f4e8c060550 0x7f4e8c062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:18.938 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:18.933+0000 7f4ead1d8700 1 --2- 172.21.15.110:0/1833962967 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f4e8c060550 0x7f4e8c062a00 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f4e9c0091e0 tx=0x7f4e9c01a000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:30:18.940 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:18.937+0000 7f4e997fa700 1 -- 172.21.15.110:0/1833962967 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f4ea404cd40 con 0x7f4ea815dde0 2023-10-06T03:30:19.098 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:19.093+0000 7f4eae1da700 1 -- 172.21.15.110:0/1833962967 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f4ea8161aa0 con 0x7f4ea815dde0 2023-10-06T03:30:19.099 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:19.097+0000 7f4e997fa700 1 -- 172.21.15.110:0/1833962967 <== mon.0 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v30) v1 ==== 93+0+10 (secure 0 0 0) 0x7f4ea400fea0 con 0x7f4ea815dde0 2023-10-06T03:30:19.099 INFO:teuthology.orchestra.run.smithi110.stdout:pg_num: 1 2023-10-06T03:30:19.103 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:19.097+0000 7f4eae1da700 1 -- 172.21.15.110:0/1833962967 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f4e8c060550 msgr2=0x7f4e8c062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:19.103 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:19.097+0000 7f4eae1da700 1 --2- 172.21.15.110:0/1833962967 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f4e8c060550 0x7f4e8c062a00 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f4e9c0091e0 tx=0x7f4e9c01a000 comp rx=0 tx=0).stop 2023-10-06T03:30:19.103 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:19.097+0000 7f4eae1da700 1 -- 172.21.15.110:0/1833962967 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4ea815dde0 msgr2=0x7f4ea81505e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:19.103 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:19.097+0000 7f4eae1da700 1 --2- 172.21.15.110:0/1833962967 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4ea815dde0 0x7f4ea81505e0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f4ea4000fd0 tx=0x7f4ea4001e10 comp rx=0 tx=0).stop 2023-10-06T03:30:19.103 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:19.097+0000 7f4eae1da700 1 -- 172.21.15.110:0/1833962967 shutdown_connections 2023-10-06T03:30:19.103 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:19.097+0000 7f4eae1da700 1 --2- 172.21.15.110:0/1833962967 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f4e8c060550 0x7f4e8c062a00 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:19.104 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:19.097+0000 7f4eae1da700 1 --2- 172.21.15.110:0/1833962967 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4ea8150b20 0x7f4ea815f680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:19.104 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:19.097+0000 7f4eae1da700 1 --2- 172.21.15.110:0/1833962967 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4ea815dde0 0x7f4ea81505e0 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:19.104 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:19.097+0000 7f4eae1da700 1 --2- 172.21.15.110:0/1833962967 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4e94003c20 0x7f4ea815d8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:19.104 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:19.097+0000 7f4eae1da700 1 -- 172.21.15.110:0/1833962967 >> 172.21.15.110:0/1833962967 conn(0x7f4ea80cbac0 msgr2=0x7f4ea815a790 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:30:19.104 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:19.097+0000 7f4eae1da700 1 -- 172.21.15.110:0/1833962967 shutdown_connections 2023-10-06T03:30:19.104 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:19.097+0000 7f4eae1da700 1 -- 172.21.15.110:0/1833962967 wait complete. 2023-10-06T03:30:19.120 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-10-06T03:30:19.238 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:19.233+0000 7f38c6927700 1 Processor -- start 2023-10-06T03:30:19.238 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:19.233+0000 7f38c6927700 1 -- start start 2023-10-06T03:30:19.241 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:19.237+0000 7f38c6927700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f38c0158e20 0x7f38c01591f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:19.241 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:19.237+0000 7f38c6927700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f38c006c7d0 0x7f38c006cc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:19.241 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:19.237+0000 7f38c6927700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f38c00670a0 0x7f38c0067520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:19.241 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:19.237+0000 7f38c6927700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f38c00c9000 con 0x7f38c0158e20 2023-10-06T03:30:19.241 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:19.237+0000 7f38c6927700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f38c00af7d0 con 0x7f38c00670a0 2023-10-06T03:30:19.241 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:19.237+0000 7f38c6927700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38c006d160 con 0x7f38c006c7d0 2023-10-06T03:30:19.241 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:19.237+0000 7f38c5925700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f38c0158e20 0x7f38c01591f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:19.241 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:19.237+0000 7f38c5925700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f38c0158e20 0x7f38c01591f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:52362/0 (socket says 172.21.15.110:52362) 2023-10-06T03:30:19.242 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:19.237+0000 7f38c5925700 1 -- 172.21.15.110:0/1425204611 learned_addr learned my addr 172.21.15.110:0/1425204611 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:30:19.242 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:19.237+0000 7f38c6126700 1 --2- 172.21.15.110:0/1425204611 >> v2:172.21.15.192:3300/0 conn(0x7f38c00670a0 0x7f38c0067520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:19.242 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:19.237+0000 7f38c5124700 1 --2- 172.21.15.110:0/1425204611 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f38c006c7d0 0x7f38c006cc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:19.242 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:19.237+0000 7f38c5925700 1 -- 172.21.15.110:0/1425204611 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f38c006c7d0 msgr2=0x7f38c006cc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:30:19.242 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:19.237+0000 7f38c5925700 1 --2- 172.21.15.110:0/1425204611 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f38c006c7d0 0x7f38c006cc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:19.242 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:19.237+0000 7f38c5925700 1 -- 172.21.15.110:0/1425204611 >> v2:172.21.15.192:3300/0 conn(0x7f38c00670a0 msgr2=0x7f38c0067520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:30:19.242 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:19.237+0000 7f38c5925700 1 --2- 172.21.15.110:0/1425204611 >> v2:172.21.15.192:3300/0 conn(0x7f38c00670a0 0x7f38c0067520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:19.242 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:19.237+0000 7f38c5925700 1 -- 172.21.15.110:0/1425204611 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38c0067a60 con 0x7f38c0158e20 2023-10-06T03:30:19.242 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:19.237+0000 7f38c6126700 1 --2- 172.21.15.110:0/1425204611 >> v2:172.21.15.192:3300/0 conn(0x7f38c00670a0 0x7f38c0067520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-06T03:30:19.244 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:19.237+0000 7f38c5925700 1 --2- 172.21.15.110:0/1425204611 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f38c0158e20 0x7f38c01591f0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f38c00c8f80 tx=0x7f38b0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c6c62558da667176 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:30:19.244 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:19.237+0000 7f38c4923700 1 -- 172.21.15.110:0/1425204611 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38b000b640 con 0x7f38c0158e20 2023-10-06T03:30:19.244 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:19.237+0000 7f38c4923700 1 -- 172.21.15.110:0/1425204611 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38b000b7a0 con 0x7f38c0158e20 2023-10-06T03:30:19.244 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:19.237+0000 7f38c4923700 1 -- 172.21.15.110:0/1425204611 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38b000bb60 con 0x7f38c0158e20 2023-10-06T03:30:19.244 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:19.237+0000 7f38c6927700 1 -- 172.21.15.110:0/1425204611 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f38c0158e20 msgr2=0x7f38c01591f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:19.244 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:19.237+0000 7f38c6927700 1 --2- 172.21.15.110:0/1425204611 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f38c0158e20 0x7f38c01591f0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f38c00c8f80 tx=0x7f38b0009250 comp rx=0 tx=0).stop 2023-10-06T03:30:19.244 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:19.241+0000 7f38c6927700 1 -- 172.21.15.110:0/1425204611 shutdown_connections 2023-10-06T03:30:19.244 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:19.241+0000 7f38c6927700 1 --2- 172.21.15.110:0/1425204611 >> v2:172.21.15.192:3300/0 conn(0x7f38c00670a0 0x7f38c0067520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:19.244 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:19.241+0000 7f38c6927700 1 --2- 172.21.15.110:0/1425204611 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f38c006c7d0 0x7f38c006cc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:19.244 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:19.241+0000 7f38c6927700 1 --2- 172.21.15.110:0/1425204611 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f38c0158e20 0x7f38c01591f0 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:19.245 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:19.241+0000 7f38c6927700 1 -- 172.21.15.110:0/1425204611 >> 172.21.15.110:0/1425204611 conn(0x7f38c006ab30 msgr2=0x7f38c00adff0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:30:19.245 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:19.241+0000 7f38c6927700 1 -- 172.21.15.110:0/1425204611 shutdown_connections 2023-10-06T03:30:19.245 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:19.241+0000 7f38c6927700 1 -- 172.21.15.110:0/1425204611 wait complete. 2023-10-06T03:30:19.246 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:19.241+0000 7f38c6927700 1 Processor -- start 2023-10-06T03:30:19.246 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:19.241+0000 7f38c6927700 1 -- start start 2023-10-06T03:30:19.249 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:19.245+0000 7f38c6927700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f38c00670a0 0x7f38c00a88d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:19.249 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:19.245+0000 7f38c6927700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f38c006c7d0 0x7f38c00a8e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:19.249 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:19.245+0000 7f38c6927700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f38c0158e20 0x7f38c01691e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:19.250 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:19.245+0000 7f38c6927700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38c00aa080 con 0x7f38c0158e20 2023-10-06T03:30:19.250 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:19.245+0000 7f38c6927700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38c00acac0 con 0x7f38c00670a0 2023-10-06T03:30:19.250 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:19.245+0000 7f38c6927700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38c00a9470 con 0x7f38c006c7d0 2023-10-06T03:30:19.250 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:19.245+0000 7f38c6126700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f38c0158e20 0x7f38c01691e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:19.250 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:19.245+0000 7f38c5925700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f38c00670a0 0x7f38c00a88d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:19.250 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:19.245+0000 7f38c6126700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f38c0158e20 0x7f38c01691e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:52372/0 (socket says 172.21.15.110:52372) 2023-10-06T03:30:19.250 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:19.245+0000 7f38c5925700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f38c00670a0 0x7f38c00a88d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.110:36716/0 (socket says 172.21.15.110:36716) 2023-10-06T03:30:19.250 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:19.245+0000 7f38c6126700 1 -- 172.21.15.110:0/4293388354 learned_addr learned my addr 172.21.15.110:0/4293388354 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:30:19.250 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:19.245+0000 7f38c5124700 1 --2- 172.21.15.110:0/4293388354 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f38c006c7d0 0x7f38c00a8e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:19.250 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:19.245+0000 7f38c6126700 1 -- 172.21.15.110:0/4293388354 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f38c006c7d0 msgr2=0x7f38c00a8e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:19.250 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:19.245+0000 7f38c6126700 1 --2- 172.21.15.110:0/4293388354 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f38c006c7d0 0x7f38c00a8e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:19.250 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:19.245+0000 7f38c6126700 1 -- 172.21.15.110:0/4293388354 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f38c00670a0 msgr2=0x7f38c00a88d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:19.251 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:19.245+0000 7f38c6126700 1 --2- 172.21.15.110:0/4293388354 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f38c00670a0 0x7f38c00a88d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:19.251 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:19.245+0000 7f38c6126700 1 -- 172.21.15.110:0/4293388354 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38b8009a20 con 0x7f38c0158e20 2023-10-06T03:30:19.251 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:19.245+0000 7f38c6126700 1 --2- 172.21.15.110:0/4293388354 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f38c0158e20 0x7f38c01691e0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f38b8009fc0 tx=0x7f38b800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:30:19.251 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:19.245+0000 7f38b5ffb700 1 -- 172.21.15.110:0/4293388354 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38b800cbd0 con 0x7f38c0158e20 2023-10-06T03:30:19.251 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:19.245+0000 7f38c6927700 1 -- 172.21.15.110:0/4293388354 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38b00094f0 con 0x7f38c0158e20 2023-10-06T03:30:19.251 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:19.245+0000 7f38b5ffb700 1 -- 172.21.15.110:0/4293388354 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38b800f040 con 0x7f38c0158e20 2023-10-06T03:30:19.251 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:19.245+0000 7f38b5ffb700 1 -- 172.21.15.110:0/4293388354 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38b8014480 con 0x7f38c0158e20 2023-10-06T03:30:19.251 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:19.245+0000 7f38c6927700 1 -- 172.21.15.110:0/4293388354 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38c00a9a30 con 0x7f38c0158e20 2023-10-06T03:30:19.252 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:19.245+0000 7f38c6927700 1 -- 172.21.15.110:0/4293388354 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f38c006d2d0 con 0x7f38c0158e20 2023-10-06T03:30:19.252 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:19.249+0000 7f38b5ffb700 1 -- 172.21.15.110:0/4293388354 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7f38b8003020 con 0x7f38c0158e20 2023-10-06T03:30:19.252 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:19.249+0000 7f38b5ffb700 1 --2- 172.21.15.110:0/4293388354 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f38a4060550 0x7f38a4062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:19.252 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:19.249+0000 7f38b5ffb700 1 -- 172.21.15.110:0/4293388354 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 6867+0+0 (secure 0 0 0) 0x7f38b807d560 con 0x7f38c0158e20 2023-10-06T03:30:19.252 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:19.249+0000 7f38c5925700 1 --2- 172.21.15.110:0/4293388354 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f38a4060550 0x7f38a4062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:19.253 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:19.249+0000 7f38c5925700 1 --2- 172.21.15.110:0/4293388354 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f38a4060550 0x7f38a4062a00 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f38c00c8f80 tx=0x7f38b0017000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:30:19.261 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:19.257+0000 7f38b5ffb700 1 -- 172.21.15.110:0/4293388354 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f38b804d6c0 con 0x7f38c0158e20 2023-10-06T03:30:19.421 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:19.417+0000 7f38c6927700 1 -- 172.21.15.110:0/4293388354 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f38c0070ff0 con 0x7f38c0158e20 2023-10-06T03:30:19.422 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:19.417+0000 7f38b5ffb700 1 -- 172.21.15.110:0/4293388354 <== mon.0 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v30) v1 ==== 92+0+10 (secure 0 0 0) 0x7f38b8050ce0 con 0x7f38c0158e20 2023-10-06T03:30:19.422 INFO:teuthology.orchestra.run.smithi110.stdout:pg_num: 8 2023-10-06T03:30:19.426 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:19.421+0000 7f38c6927700 1 -- 172.21.15.110:0/4293388354 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f38a4060550 msgr2=0x7f38a4062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:19.426 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:19.421+0000 7f38c6927700 1 --2- 172.21.15.110:0/4293388354 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f38a4060550 0x7f38a4062a00 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f38c00c8f80 tx=0x7f38b0017000 comp rx=0 tx=0).stop 2023-10-06T03:30:19.426 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:19.421+0000 7f38c6927700 1 -- 172.21.15.110:0/4293388354 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f38c0158e20 msgr2=0x7f38c01691e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:19.427 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:19.421+0000 7f38c6927700 1 --2- 172.21.15.110:0/4293388354 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f38c0158e20 0x7f38c01691e0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f38b8009fc0 tx=0x7f38b800b040 comp rx=0 tx=0).stop 2023-10-06T03:30:19.427 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:19.421+0000 7f38c6927700 1 -- 172.21.15.110:0/4293388354 shutdown_connections 2023-10-06T03:30:19.427 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:19.421+0000 7f38c6927700 1 --2- 172.21.15.110:0/4293388354 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f38a4060550 0x7f38a4062a00 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:19.427 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:19.421+0000 7f38c6927700 1 --2- 172.21.15.110:0/4293388354 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f38c0158e20 0x7f38c01691e0 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:19.427 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:19.421+0000 7f38c6927700 1 --2- 172.21.15.110:0/4293388354 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f38c006c7d0 0x7f38c00a8e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:19.427 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:19.421+0000 7f38c6927700 1 --2- 172.21.15.110:0/4293388354 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f38c00670a0 0x7f38c00a88d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:19.427 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:19.421+0000 7f38c6927700 1 -- 172.21.15.110:0/4293388354 >> 172.21.15.110:0/4293388354 conn(0x7f38c006ab30 msgr2=0x7f38c0155c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:30:19.427 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:19.421+0000 7f38c6927700 1 -- 172.21.15.110:0/4293388354 shutdown_connections 2023-10-06T03:30:19.427 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:19.421+0000 7f38c6927700 1 -- 172.21.15.110:0/4293388354 wait complete. 2023-10-06T03:30:19.442 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-06T03:30:19.548 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:19.545+0000 7fda3e503700 1 Processor -- start 2023-10-06T03:30:19.548 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:19.545+0000 7fda3e503700 1 -- start start 2023-10-06T03:30:19.550 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:19.545+0000 7fda3e503700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fda380b1d40 0x7fda380b2110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:19.550 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:19.545+0000 7fda3e503700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fda38155fe0 0x7fda380b00c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:19.550 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:19.545+0000 7fda3e503700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda380caa70 con 0x7fda380afc70 2023-10-06T03:30:19.550 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:19.545+0000 7fda3e503700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fda380b0600 con 0x7fda380b1d40 2023-10-06T03:30:19.550 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:19.545+0000 7fda3e503700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda380ca740 con 0x7fda38155fe0 2023-10-06T03:30:19.550 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:19.545+0000 7fda3d501700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fda380b1d40 0x7fda380b2110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:19.551 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:19.545+0000 7fda3cd00700 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7fda380afc70 0x7fda381558d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:53310/0 (socket says 172.21.15.110:53310) 2023-10-06T03:30:19.551 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:19.545+0000 7fda3cd00700 1 -- 172.21.15.110:0/1498560113 learned_addr learned my addr 172.21.15.110:0/1498560113 (peer_addr_for_me v1:172.21.15.110:0/0) 2023-10-06T03:30:19.551 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:19.545+0000 7fda3dd02700 1 --2- 172.21.15.110:0/1498560113 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fda38155fe0 0x7fda380b00c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:19.552 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:19.545+0000 7fda3d501700 1 -- 172.21.15.110:0/1498560113 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fda38155fe0 msgr2=0x7fda380b00c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:30:19.552 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:19.545+0000 7fda3d501700 1 --2- 172.21.15.110:0/1498560113 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fda38155fe0 0x7fda380b00c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:19.552 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:19.545+0000 7fda2ffff700 1 -- 172.21.15.110:0/1498560113 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1799914310 0 0) 0x7fda380caa70 con 0x7fda380afc70 2023-10-06T03:30:19.553 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:19.545+0000 7fda3d501700 1 -- 172.21.15.110:0/1498560113 >> v1:172.21.15.110:6789/0 conn(0x7fda380afc70 legacy=0x7fda381558d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:19.553 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:19.545+0000 7fda3d501700 1 -- 172.21.15.110:0/1498560113 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda38159cc0 con 0x7fda380b1d40 2023-10-06T03:30:19.553 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:19.545+0000 7fda3dd02700 1 --2- 172.21.15.110:0/1498560113 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fda38155fe0 0x7fda380b00c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-06T03:30:19.553 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:19.545+0000 7fda3d501700 1 --2- 172.21.15.110:0/1498560113 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fda380b1d40 0x7fda380b2110 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fda340098e0 tx=0x7fda340091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2abde04504a6d216 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:30:19.553 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:19.549+0000 7fda2ffff700 1 -- 172.21.15.110:0/1498560113 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda34009e80 con 0x7fda380b1d40 2023-10-06T03:30:19.553 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:19.549+0000 7fda2ffff700 1 -- 172.21.15.110:0/1498560113 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda3400b860 con 0x7fda380b1d40 2023-10-06T03:30:19.553 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:19.549+0000 7fda2ffff700 1 -- 172.21.15.110:0/1498560113 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda3400bbe0 con 0x7fda380b1d40 2023-10-06T03:30:19.553 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:19.549+0000 7fda3e503700 1 -- 172.21.15.110:0/1498560113 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fda380b1d40 msgr2=0x7fda380b2110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:19.553 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:19.549+0000 7fda3e503700 1 --2- 172.21.15.110:0/1498560113 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fda380b1d40 0x7fda380b2110 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fda340098e0 tx=0x7fda340091a0 comp rx=0 tx=0).stop 2023-10-06T03:30:19.553 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:19.549+0000 7fda3e503700 1 -- 172.21.15.110:0/1498560113 shutdown_connections 2023-10-06T03:30:19.553 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:19.549+0000 7fda3e503700 1 --2- 172.21.15.110:0/1498560113 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fda38155fe0 0x7fda380b00c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:19.553 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:19.549+0000 7fda3e503700 1 --2- 172.21.15.110:0/1498560113 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fda380b1d40 0x7fda380b2110 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:19.554 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:19.549+0000 7fda3e503700 1 -- 172.21.15.110:0/1498560113 >> 172.21.15.110:0/1498560113 conn(0x7fda38068470 msgr2=0x7fda38068870 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:30:19.554 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:19.549+0000 7fda3e503700 1 -- 172.21.15.110:0/1498560113 shutdown_connections 2023-10-06T03:30:19.554 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:19.549+0000 7fda3e503700 1 -- 172.21.15.110:0/1498560113 wait complete. 2023-10-06T03:30:19.554 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:19.549+0000 7fda3e503700 1 Processor -- start 2023-10-06T03:30:19.554 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:19.549+0000 7fda3e503700 1 -- start start 2023-10-06T03:30:19.556 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:19.553+0000 7fda3e503700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fda380afc70 0x7fda3815b990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:19.557 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:19.553+0000 7fda3e503700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fda380b1d40 0x7fda381511b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:19.557 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:19.553+0000 7fda3e503700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fda38155fe0 0x7fda3815c800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:19.557 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:19.553+0000 7fda3e503700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda3815cd40 con 0x7fda38155fe0 2023-10-06T03:30:19.557 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:19.553+0000 7fda3e503700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda380c4a80 con 0x7fda380afc70 2023-10-06T03:30:19.557 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:19.553+0000 7fda3e503700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda380c45f0 con 0x7fda380b1d40 2023-10-06T03:30:19.557 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:19.553+0000 7fda3dd02700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fda38155fe0 0x7fda3815c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:19.557 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:19.553+0000 7fda3dd02700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fda38155fe0 0x7fda3815c800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:52386/0 (socket says 172.21.15.110:52386) 2023-10-06T03:30:19.557 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:19.553+0000 7fda3dd02700 1 -- 172.21.15.110:0/1600249927 learned_addr learned my addr 172.21.15.110:0/1600249927 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:30:19.557 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:19.553+0000 7fda3cd00700 1 --2- 172.21.15.110:0/1600249927 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fda380b1d40 0x7fda381511b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:19.557 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:19.553+0000 7fda3d501700 1 --2- 172.21.15.110:0/1600249927 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fda380afc70 0x7fda3815b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:19.558 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:19.553+0000 7fda3dd02700 1 -- 172.21.15.110:0/1600249927 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fda380b1d40 msgr2=0x7fda381511b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:19.558 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:19.553+0000 7fda3dd02700 1 --2- 172.21.15.110:0/1600249927 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fda380b1d40 0x7fda381511b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:19.558 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:19.553+0000 7fda3dd02700 1 -- 172.21.15.110:0/1600249927 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fda380afc70 msgr2=0x7fda3815b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:19.558 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:19.553+0000 7fda3dd02700 1 --2- 172.21.15.110:0/1600249927 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fda380afc70 0x7fda3815b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:19.558 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:19.553+0000 7fda3dd02700 1 -- 172.21.15.110:0/1600249927 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda34009420 con 0x7fda38155fe0 2023-10-06T03:30:19.558 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:19.553+0000 7fda3dd02700 1 --2- 172.21.15.110:0/1600249927 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fda38155fe0 0x7fda3815c800 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fda30009fc0 tx=0x7fda300099b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:30:19.558 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:19.553+0000 7fda2d7fa700 1 -- 172.21.15.110:0/1600249927 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda3000dd40 con 0x7fda38155fe0 2023-10-06T03:30:19.558 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:19.553+0000 7fda2d7fa700 1 -- 172.21.15.110:0/1600249927 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda3000f040 con 0x7fda38155fe0 2023-10-06T03:30:19.558 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:19.553+0000 7fda3e503700 1 -- 172.21.15.110:0/1600249927 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda380c39e0 con 0x7fda38155fe0 2023-10-06T03:30:19.558 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:19.553+0000 7fda2d7fa700 1 -- 172.21.15.110:0/1600249927 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda30015690 con 0x7fda38155fe0 2023-10-06T03:30:19.559 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:19.553+0000 7fda3e503700 1 -- 172.21.15.110:0/1600249927 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda380c3fa0 con 0x7fda38155fe0 2023-10-06T03:30:19.559 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:19.553+0000 7fda3e503700 1 -- 172.21.15.110:0/1600249927 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda38070ff0 con 0x7fda38155fe0 2023-10-06T03:30:19.559 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:19.553+0000 7fda2d7fa700 1 -- 172.21.15.110:0/1600249927 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7fda3001e3b0 con 0x7fda38155fe0 2023-10-06T03:30:19.559 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:19.553+0000 7fda2d7fa700 1 --2- 172.21.15.110:0/1600249927 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fda1c060550 0x7fda1c062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:19.560 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:19.553+0000 7fda2d7fa700 1 -- 172.21.15.110:0/1600249927 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 6867+0+0 (secure 0 0 0) 0x7fda3007d1b0 con 0x7fda38155fe0 2023-10-06T03:30:19.560 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:19.557+0000 7fda3d501700 1 --2- 172.21.15.110:0/1600249927 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fda1c060550 0x7fda1c062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:19.560 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:19.557+0000 7fda3d501700 1 --2- 172.21.15.110:0/1600249927 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fda1c060550 0x7fda1c062a00 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fda380cd590 tx=0x7fda34017000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:30:19.570 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:19.565+0000 7fda2d7fa700 1 -- 172.21.15.110:0/1600249927 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fda3004d310 con 0x7fda38155fe0 2023-10-06T03:30:19.732 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:19.729+0000 7fda3e503700 1 -- 172.21.15.110:0/1600249927 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fda380b07e0 con 0x7fda38155fe0 2023-10-06T03:30:19.732 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:19.729+0000 7fda2d7fa700 1 -- 172.21.15.110:0/1600249927 <== mon.0 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v30) v1 ==== 104+0+11 (secure 0 0 0) 0x7fda30050930 con 0x7fda38155fe0 2023-10-06T03:30:19.733 INFO:teuthology.orchestra.run.smithi110.stdout:pg_num: 32 2023-10-06T03:30:19.735 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:19.729+0000 7fda3e503700 1 -- 172.21.15.110:0/1600249927 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fda1c060550 msgr2=0x7fda1c062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:19.735 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:19.729+0000 7fda3e503700 1 --2- 172.21.15.110:0/1600249927 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fda1c060550 0x7fda1c062a00 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fda380cd590 tx=0x7fda34017000 comp rx=0 tx=0).stop 2023-10-06T03:30:19.736 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:19.729+0000 7fda3e503700 1 -- 172.21.15.110:0/1600249927 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fda38155fe0 msgr2=0x7fda3815c800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:19.736 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:19.729+0000 7fda3e503700 1 --2- 172.21.15.110:0/1600249927 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fda38155fe0 0x7fda3815c800 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fda30009fc0 tx=0x7fda300099b0 comp rx=0 tx=0).stop 2023-10-06T03:30:19.736 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:19.729+0000 7fda3e503700 1 -- 172.21.15.110:0/1600249927 shutdown_connections 2023-10-06T03:30:19.736 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:19.729+0000 7fda3e503700 1 --2- 172.21.15.110:0/1600249927 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fda1c060550 0x7fda1c062a00 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:19.736 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:19.729+0000 7fda3e503700 1 --2- 172.21.15.110:0/1600249927 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fda38155fe0 0x7fda3815c800 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:19.736 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:19.729+0000 7fda3e503700 1 --2- 172.21.15.110:0/1600249927 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fda380b1d40 0x7fda381511b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:19.736 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:19.729+0000 7fda3e503700 1 --2- 172.21.15.110:0/1600249927 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fda380afc70 0x7fda3815b990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:19.736 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:19.733+0000 7fda3e503700 1 -- 172.21.15.110:0/1600249927 >> 172.21.15.110:0/1600249927 conn(0x7fda38068470 msgr2=0x7fda3806d440 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:30:19.736 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:19.733+0000 7fda3e503700 1 -- 172.21.15.110:0/1600249927 shutdown_connections 2023-10-06T03:30:19.736 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:19.733+0000 7fda3e503700 1 -- 172.21.15.110:0/1600249927 wait complete. 2023-10-06T03:30:19.748 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-06T03:30:19.857 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:19.853+0000 7f2c1d813700 1 Processor -- start 2023-10-06T03:30:19.858 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:19.853+0000 7f2c1d813700 1 -- start start 2023-10-06T03:30:19.860 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:19.853+0000 7f2c1d813700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f2c18158e70 0x7f2c1815d620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:19.860 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:19.853+0000 7f2c1d813700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f2c180caf40 0x7f2c1806d500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:19.861 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:19.853+0000 7f2c1d813700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c1806d070 con 0x7f2c180ad940 2023-10-06T03:30:19.861 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:19.853+0000 7f2c1d813700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2c1815db60 con 0x7f2c18158e70 2023-10-06T03:30:19.861 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:19.853+0000 7f2c1d813700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2c1806cd20 con 0x7f2c180caf40 2023-10-06T03:30:19.861 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:19.853+0000 7f2c16ffd700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f2c18158e70 0x7f2c1815d620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:19.861 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:19.853+0000 7f2c16ffd700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f2c18158e70 0x7f2c1815d620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:52392/0 (socket says 172.21.15.110:52392) 2023-10-06T03:30:19.861 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:19.853+0000 7f2c16ffd700 1 -- 172.21.15.110:0/184488167 learned_addr learned my addr 172.21.15.110:0/184488167 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:30:19.862 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:19.853+0000 7f2c177fe700 1 --2- 172.21.15.110:0/184488167 >> v2:172.21.15.192:3300/0 conn(0x7f2c180caf40 0x7f2c1806d500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:19.862 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:19.857+0000 7f2c16ffd700 1 -- 172.21.15.110:0/184488167 >> v2:172.21.15.192:3300/0 conn(0x7f2c180caf40 msgr2=0x7f2c1806d500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:30:19.862 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:19.857+0000 7f2c16ffd700 1 --2- 172.21.15.110:0/184488167 >> v2:172.21.15.192:3300/0 conn(0x7f2c180caf40 0x7f2c1806d500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:19.862 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:19.857+0000 7f2c16ffd700 1 -- 172.21.15.110:0/184488167 >> v1:172.21.15.192:6789/0 conn(0x7f2c180ad940 legacy=0x7f2c180ca830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:19.862 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:19.857+0000 7f2c16ffd700 1 -- 172.21.15.110:0/184488167 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c180c8500 con 0x7f2c18158e70 2023-10-06T03:30:19.862 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:19.857+0000 7f2c16ffd700 1 --2- 172.21.15.110:0/184488167 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2c18158e70 0x7f2c1815d620 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f2c1806ea90 tx=0x7f2c0c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=38114ba30eb502ed server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:30:19.863 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:19.857+0000 7f2c15ffb700 1 -- 172.21.15.110:0/184488167 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c0c009e70 con 0x7f2c18158e70 2023-10-06T03:30:19.863 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:19.857+0000 7f2c15ffb700 1 -- 172.21.15.110:0/184488167 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c0c00b860 con 0x7f2c18158e70 2023-10-06T03:30:19.863 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:19.857+0000 7f2c15ffb700 1 -- 172.21.15.110:0/184488167 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c0c00bc20 con 0x7f2c18158e70 2023-10-06T03:30:19.868 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:19.857+0000 7f2c1d813700 1 -- 172.21.15.110:0/184488167 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2c18158e70 msgr2=0x7f2c1815d620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:19.868 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:19.857+0000 7f2c1d813700 1 --2- 172.21.15.110:0/184488167 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2c18158e70 0x7f2c1815d620 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f2c1806ea90 tx=0x7f2c0c009250 comp rx=0 tx=0).stop 2023-10-06T03:30:19.868 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:19.857+0000 7f2c1d813700 1 -- 172.21.15.110:0/184488167 shutdown_connections 2023-10-06T03:30:19.868 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:19.857+0000 7f2c1d813700 1 --2- 172.21.15.110:0/184488167 >> v2:172.21.15.192:3300/0 conn(0x7f2c180caf40 0x7f2c1806d500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:19.868 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:19.857+0000 7f2c1d813700 1 --2- 172.21.15.110:0/184488167 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2c18158e70 0x7f2c1815d620 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:19.869 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:19.857+0000 7f2c1d813700 1 -- 172.21.15.110:0/184488167 >> 172.21.15.110:0/184488167 conn(0x7f2c1806b7c0 msgr2=0x7f2c1806bbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:30:19.869 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:19.857+0000 7f2c1d813700 1 -- 172.21.15.110:0/184488167 shutdown_connections 2023-10-06T03:30:19.869 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:19.857+0000 7f2c1d813700 1 -- 172.21.15.110:0/184488167 wait complete. 2023-10-06T03:30:19.869 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:19.861+0000 7f2c1d813700 1 Processor -- start 2023-10-06T03:30:19.869 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:19.861+0000 7f2c1d813700 1 -- start start 2023-10-06T03:30:19.869 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:19.861+0000 7f2c1d813700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2c180ad940 0x7f2c180b2c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:19.870 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:19.861+0000 7f2c1d813700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2c180caf40 0x7f2c180c5800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:19.870 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:19.861+0000 7f2c1d813700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2c181563b0 0x7f2c1815f110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:19.870 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:19.861+0000 7f2c1d813700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c180c5d40 con 0x7f2c180ad940 2023-10-06T03:30:19.870 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:19.861+0000 7f2c1d813700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c180c6b40 con 0x7f2c181563b0 2023-10-06T03:30:19.870 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:19.861+0000 7f2c1d813700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c180c5f30 con 0x7f2c180caf40 2023-10-06T03:30:19.870 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:19.861+0000 7f2c16ffd700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2c180ad940 0x7f2c180b2c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:19.871 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:19.861+0000 7f2c16ffd700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2c180ad940 0x7f2c180b2c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:52406/0 (socket says 172.21.15.110:52406) 2023-10-06T03:30:19.871 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:19.861+0000 7f2c16ffd700 1 -- 172.21.15.110:0/716345064 learned_addr learned my addr 172.21.15.110:0/716345064 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:30:19.871 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:19.861+0000 7f2c167fc700 1 --2- 172.21.15.110:0/716345064 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2c180caf40 0x7f2c180c5800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:19.871 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:19.861+0000 7f2c177fe700 1 --2- 172.21.15.110:0/716345064 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2c181563b0 0x7f2c1815f110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:19.871 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:19.861+0000 7f2c16ffd700 1 -- 172.21.15.110:0/716345064 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2c180caf40 msgr2=0x7f2c180c5800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:19.871 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:19.861+0000 7f2c16ffd700 1 --2- 172.21.15.110:0/716345064 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2c180caf40 0x7f2c180c5800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:19.872 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:19.861+0000 7f2c16ffd700 1 -- 172.21.15.110:0/716345064 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2c181563b0 msgr2=0x7f2c1815f110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:19.872 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:19.861+0000 7f2c16ffd700 1 --2- 172.21.15.110:0/716345064 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2c181563b0 0x7f2c1815f110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:19.872 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:19.861+0000 7f2c16ffd700 1 -- 172.21.15.110:0/716345064 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c0c0094f0 con 0x7f2c180ad940 2023-10-06T03:30:19.872 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:19.861+0000 7f2c16ffd700 1 --2- 172.21.15.110:0/716345064 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2c180ad940 0x7f2c180b2c30 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f2c1806b260 tx=0x7f2c0c017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:30:19.872 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:19.861+0000 7f2bfeffd700 1 -- 172.21.15.110:0/716345064 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c0c01f070 con 0x7f2c180ad940 2023-10-06T03:30:19.872 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:19.861+0000 7f2bfeffd700 1 -- 172.21.15.110:0/716345064 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c0c002f30 con 0x7f2c180ad940 2023-10-06T03:30:19.872 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:19.861+0000 7f2c1d813700 1 -- 172.21.15.110:0/716345064 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c180c6210 con 0x7f2c180ad940 2023-10-06T03:30:19.873 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:19.861+0000 7f2bfeffd700 1 -- 172.21.15.110:0/716345064 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c0c000bc0 con 0x7f2c180ad940 2023-10-06T03:30:19.873 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:19.861+0000 7f2c1d813700 1 -- 172.21.15.110:0/716345064 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c180c67d0 con 0x7f2c180ad940 2023-10-06T03:30:19.873 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:19.865+0000 7f2c1d813700 1 -- 172.21.15.110:0/716345064 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c180b31c0 con 0x7f2c180ad940 2023-10-06T03:30:19.874 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:19.865+0000 7f2bfeffd700 1 -- 172.21.15.110:0/716345064 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7f2c0c023070 con 0x7f2c180ad940 2023-10-06T03:30:19.875 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:19.865+0000 7f2bfeffd700 1 --2- 172.21.15.110:0/716345064 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f2c04060580 0x7f2c04062a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:19.875 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:19.865+0000 7f2bfeffd700 1 -- 172.21.15.110:0/716345064 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 6867+0+0 (secure 0 0 0) 0x7f2c0c07de30 con 0x7f2c180ad940 2023-10-06T03:30:19.875 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:19.865+0000 7f2c167fc700 1 --2- 172.21.15.110:0/716345064 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f2c04060580 0x7f2c04062a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:19.875 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:19.865+0000 7f2c167fc700 1 --2- 172.21.15.110:0/716345064 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f2c04060580 0x7f2c04062a30 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f2c180c4d00 tx=0x7f2c08007ec0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:30:19.878 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:19.873+0000 7f2bfeffd700 1 -- 172.21.15.110:0/716345064 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f2c0c04df90 con 0x7f2c180ad940 2023-10-06T03:30:20.036 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:20.033+0000 7f2c1d813700 1 -- 172.21.15.110:0/716345064 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f2c1806ff60 con 0x7f2c180ad940 2023-10-06T03:30:20.037 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:20.033+0000 7f2bfeffd700 1 -- 172.21.15.110:0/716345064 <== mon.0 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v30) v1 ==== 100+0+11 (secure 0 0 0) 0x7f2c0c02a020 con 0x7f2c180ad940 2023-10-06T03:30:20.037 INFO:teuthology.orchestra.run.smithi110.stdout:pg_num: 64 2023-10-06T03:30:20.040 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:20.033+0000 7f2c1d813700 1 -- 172.21.15.110:0/716345064 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f2c04060580 msgr2=0x7f2c04062a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:20.040 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:20.033+0000 7f2c1d813700 1 --2- 172.21.15.110:0/716345064 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f2c04060580 0x7f2c04062a30 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f2c180c4d00 tx=0x7f2c08007ec0 comp rx=0 tx=0).stop 2023-10-06T03:30:20.040 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:20.033+0000 7f2c1d813700 1 -- 172.21.15.110:0/716345064 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2c180ad940 msgr2=0x7f2c180b2c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:20.040 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:20.033+0000 7f2c1d813700 1 --2- 172.21.15.110:0/716345064 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2c180ad940 0x7f2c180b2c30 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f2c1806b260 tx=0x7f2c0c017000 comp rx=0 tx=0).stop 2023-10-06T03:30:20.040 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:20.037+0000 7f2c1d813700 1 -- 172.21.15.110:0/716345064 shutdown_connections 2023-10-06T03:30:20.040 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:20.037+0000 7f2c1d813700 1 --2- 172.21.15.110:0/716345064 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f2c04060580 0x7f2c04062a30 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:20.040 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:20.037+0000 7f2c1d813700 1 --2- 172.21.15.110:0/716345064 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2c181563b0 0x7f2c1815f110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:20.041 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:20.037+0000 7f2c1d813700 1 --2- 172.21.15.110:0/716345064 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f2c180caf40 0x7f2c180c5800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:20.041 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:20.037+0000 7f2c1d813700 1 --2- 172.21.15.110:0/716345064 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2c180ad940 0x7f2c180b2c30 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:20.041 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:20.037+0000 7f2c1d813700 1 -- 172.21.15.110:0/716345064 >> 172.21.15.110:0/716345064 conn(0x7f2c1806b7c0 msgr2=0x7f2c1806d500 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:30:20.041 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:20.037+0000 7f2c1d813700 1 -- 172.21.15.110:0/716345064 shutdown_connections 2023-10-06T03:30:20.041 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:20.037+0000 7f2c1d813700 1 -- 172.21.15.110:0/716345064 wait complete. 2023-10-06T03:30:20.051 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2023-10-06T03:30:20.169 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:20.165+0000 7fb07353f700 1 Processor -- start 2023-10-06T03:30:20.170 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:20.165+0000 7fb07353f700 1 -- start start 2023-10-06T03:30:20.171 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:20.165+0000 7fb07353f700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fb06c0aef40 0x7fb06c0ce200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:20.172 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:20.165+0000 7fb07353f700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fb06c15d210 0x7fb06c0afa60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:20.172 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:20.165+0000 7fb07353f700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb06c0cf5f0 con 0x7fb06c0cb1b0 2023-10-06T03:30:20.172 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:20.165+0000 7fb07353f700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb06c0ce740 con 0x7fb06c15d210 2023-10-06T03:30:20.172 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:20.165+0000 7fb07353f700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb06c0ce9e0 con 0x7fb06c0aef40 2023-10-06T03:30:20.172 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:20.165+0000 7fb072d3e700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fb06c15d210 0x7fb06c0afa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:20.172 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:20.165+0000 7fb072d3e700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fb06c15d210 0x7fb06c0afa60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:52414/0 (socket says 172.21.15.110:52414) 2023-10-06T03:30:20.172 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:20.165+0000 7fb072d3e700 1 -- 172.21.15.110:0/1477917904 learned_addr learned my addr 172.21.15.110:0/1477917904 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:30:20.172 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:20.165+0000 7fb07253d700 1 --2- 172.21.15.110:0/1477917904 >> v2:172.21.15.192:3300/0 conn(0x7fb06c0aef40 0x7fb06c0ce200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:20.172 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:20.169+0000 7fb072d3e700 1 -- 172.21.15.110:0/1477917904 >> v2:172.21.15.192:3300/0 conn(0x7fb06c0aef40 msgr2=0x7fb06c0ce200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:30:20.173 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:20.169+0000 7fb072d3e700 1 --2- 172.21.15.110:0/1477917904 >> v2:172.21.15.192:3300/0 conn(0x7fb06c0aef40 0x7fb06c0ce200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:20.173 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:20.169+0000 7fb072d3e700 1 -- 172.21.15.110:0/1477917904 >> v1:172.21.15.192:6789/0 conn(0x7fb06c0cb1b0 legacy=0x7fb06c15cb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:20.173 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:20.169+0000 7fb072d3e700 1 -- 172.21.15.110:0/1477917904 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb06c0cecc0 con 0x7fb06c15d210 2023-10-06T03:30:20.173 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:20.169+0000 7fb072d3e700 1 --2- 172.21.15.110:0/1477917904 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb06c15d210 0x7fb06c0afa60 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fb06c06daf0 tx=0x7fb064009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=693202b67fd9e067 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:30:20.173 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:20.169+0000 7fb07153b700 1 -- 172.21.15.110:0/1477917904 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb06400e070 con 0x7fb06c15d210 2023-10-06T03:30:20.173 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:20.169+0000 7fb07153b700 1 -- 172.21.15.110:0/1477917904 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb06400ba50 con 0x7fb06c15d210 2023-10-06T03:30:20.173 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:20.169+0000 7fb07153b700 1 -- 172.21.15.110:0/1477917904 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0640092e0 con 0x7fb06c15d210 2023-10-06T03:30:20.174 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:20.169+0000 7fb07353f700 1 -- 172.21.15.110:0/1477917904 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb06c15d210 msgr2=0x7fb06c0afa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:20.174 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:20.169+0000 7fb07353f700 1 --2- 172.21.15.110:0/1477917904 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb06c15d210 0x7fb06c0afa60 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fb06c06daf0 tx=0x7fb064009840 comp rx=0 tx=0).stop 2023-10-06T03:30:20.174 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:20.169+0000 7fb07353f700 1 -- 172.21.15.110:0/1477917904 shutdown_connections 2023-10-06T03:30:20.174 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:20.169+0000 7fb07353f700 1 --2- 172.21.15.110:0/1477917904 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb06c15d210 0x7fb06c0afa60 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:20.174 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:20.169+0000 7fb07353f700 1 --2- 172.21.15.110:0/1477917904 >> v2:172.21.15.192:3300/0 conn(0x7fb06c0aef40 0x7fb06c0ce200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:20.175 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:20.169+0000 7fb07353f700 1 -- 172.21.15.110:0/1477917904 >> 172.21.15.110:0/1477917904 conn(0x7fb06c0cfec0 msgr2=0x7fb06c0d02c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:30:20.175 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:20.169+0000 7fb07353f700 1 -- 172.21.15.110:0/1477917904 shutdown_connections 2023-10-06T03:30:20.175 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:20.169+0000 7fb07353f700 1 -- 172.21.15.110:0/1477917904 wait complete. 2023-10-06T03:30:20.175 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:20.173+0000 7fb07353f700 1 Processor -- start 2023-10-06T03:30:20.175 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:20.173+0000 7fb07353f700 1 -- start start 2023-10-06T03:30:20.179 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:20.173+0000 7fb07353f700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb06c0aef40 0x7fb06c0b5210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:20.179 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:20.173+0000 7fb07353f700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb06c0cb1b0 0x7fb06c0c5220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:20.179 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:20.173+0000 7fb07353f700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb06c15d210 0x7fb06c0c5760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:20.179 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:20.173+0000 7fb07353f700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb06c0ca590 con 0x7fb06c15d210 2023-10-06T03:30:20.179 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:20.173+0000 7fb07353f700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb06c0c7b70 con 0x7fb06c0aef40 2023-10-06T03:30:20.179 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:20.173+0000 7fb07353f700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb06c0c7e50 con 0x7fb06c0cb1b0 2023-10-06T03:30:20.179 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:20.173+0000 7fb072d3e700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb06c15d210 0x7fb06c0c5760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:20.179 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:20.173+0000 7fb072d3e700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb06c15d210 0x7fb06c0c5760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:52416/0 (socket says 172.21.15.110:52416) 2023-10-06T03:30:20.179 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:20.173+0000 7fb072d3e700 1 -- 172.21.15.110:0/1638341530 learned_addr learned my addr 172.21.15.110:0/1638341530 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:30:20.179 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:20.173+0000 7fb07253d700 1 --2- 172.21.15.110:0/1638341530 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb06c0aef40 0x7fb06c0b5210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:20.180 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:20.173+0000 7fb071d3c700 1 --2- 172.21.15.110:0/1638341530 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb06c0cb1b0 0x7fb06c0c5220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:20.180 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:20.173+0000 7fb072d3e700 1 -- 172.21.15.110:0/1638341530 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb06c0cb1b0 msgr2=0x7fb06c0c5220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:20.180 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:20.173+0000 7fb072d3e700 1 --2- 172.21.15.110:0/1638341530 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb06c0cb1b0 0x7fb06c0c5220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:20.180 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:20.173+0000 7fb072d3e700 1 -- 172.21.15.110:0/1638341530 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb06c0aef40 msgr2=0x7fb06c0b5210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:20.180 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:20.173+0000 7fb072d3e700 1 --2- 172.21.15.110:0/1638341530 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb06c0aef40 0x7fb06c0b5210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:20.180 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:20.173+0000 7fb072d3e700 1 -- 172.21.15.110:0/1638341530 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb064009ae0 con 0x7fb06c15d210 2023-10-06T03:30:20.180 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:20.173+0000 7fb072d3e700 1 --2- 172.21.15.110:0/1638341530 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb06c15d210 0x7fb06c0c5760 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fb06c088890 tx=0x7fb064018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:30:20.180 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:20.173+0000 7fb0627fc700 1 -- 172.21.15.110:0/1638341530 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb064007000 con 0x7fb06c15d210 2023-10-06T03:30:20.180 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:20.173+0000 7fb0627fc700 1 -- 172.21.15.110:0/1638341530 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb06400be80 con 0x7fb06c15d210 2023-10-06T03:30:20.180 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:20.173+0000 7fb07353f700 1 -- 172.21.15.110:0/1638341530 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb06c0ca220 con 0x7fb06c15d210 2023-10-06T03:30:20.180 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:20.173+0000 7fb0627fc700 1 -- 172.21.15.110:0/1638341530 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb06401c410 con 0x7fb06c15d210 2023-10-06T03:30:20.181 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:20.173+0000 7fb07353f700 1 -- 172.21.15.110:0/1638341530 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb06c0c7790 con 0x7fb06c15d210 2023-10-06T03:30:20.181 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:20.177+0000 7fb07353f700 1 -- 172.21.15.110:0/1638341530 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb06c0c8060 con 0x7fb06c15d210 2023-10-06T03:30:20.181 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:20.177+0000 7fb0627fc700 1 -- 172.21.15.110:0/1638341530 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7fb064003050 con 0x7fb06c15d210 2023-10-06T03:30:20.182 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:20.177+0000 7fb0627fc700 1 --2- 172.21.15.110:0/1638341530 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fb050060550 0x7fb050062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:20.182 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:20.177+0000 7fb0627fc700 1 -- 172.21.15.110:0/1638341530 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 6867+0+0 (secure 0 0 0) 0x7fb06407e660 con 0x7fb06c15d210 2023-10-06T03:30:20.182 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:20.177+0000 7fb07253d700 1 --2- 172.21.15.110:0/1638341530 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fb050060550 0x7fb050062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:20.182 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:20.177+0000 7fb07253d700 1 --2- 172.21.15.110:0/1638341530 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fb050060550 0x7fb050062a00 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fb05c006fd0 tx=0x7fb05c008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:30:20.188 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:20.185+0000 7fb0627fc700 1 -- 172.21.15.110:0/1638341530 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fb06404e720 con 0x7fb06c15d210 2023-10-06T03:30:20.350 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:20.345+0000 7fb07353f700 1 -- 172.21.15.110:0/1638341530 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7fb06c070ff0 con 0x7fb06c15d210 2023-10-06T03:30:20.350 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:20.345+0000 7fb0627fc700 1 -- 172.21.15.110:0/1638341530 <== mon.0 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v30) v1 ==== 103+0+11 (secure 0 0 0) 0x7fb064051d40 con 0x7fb06c15d210 2023-10-06T03:30:20.351 INFO:teuthology.orchestra.run.smithi110.stdout:pg_num: 32 2023-10-06T03:30:20.353 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:20.349+0000 7fb07353f700 1 -- 172.21.15.110:0/1638341530 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fb050060550 msgr2=0x7fb050062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:20.353 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:20.349+0000 7fb07353f700 1 --2- 172.21.15.110:0/1638341530 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fb050060550 0x7fb050062a00 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fb05c006fd0 tx=0x7fb05c008040 comp rx=0 tx=0).stop 2023-10-06T03:30:20.353 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:20.349+0000 7fb07353f700 1 -- 172.21.15.110:0/1638341530 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb06c15d210 msgr2=0x7fb06c0c5760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:20.354 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:20.349+0000 7fb07353f700 1 --2- 172.21.15.110:0/1638341530 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb06c15d210 0x7fb06c0c5760 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fb06c088890 tx=0x7fb064018000 comp rx=0 tx=0).stop 2023-10-06T03:30:20.354 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:20.349+0000 7fb07353f700 1 -- 172.21.15.110:0/1638341530 shutdown_connections 2023-10-06T03:30:20.354 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:20.349+0000 7fb07353f700 1 --2- 172.21.15.110:0/1638341530 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fb050060550 0x7fb050062a00 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:20.354 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:20.349+0000 7fb07353f700 1 --2- 172.21.15.110:0/1638341530 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb06c15d210 0x7fb06c0c5760 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:20.354 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:20.349+0000 7fb07353f700 1 --2- 172.21.15.110:0/1638341530 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb06c0cb1b0 0x7fb06c0c5220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:20.354 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:20.349+0000 7fb07353f700 1 --2- 172.21.15.110:0/1638341530 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb06c0aef40 0x7fb06c0b5210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:20.354 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:20.349+0000 7fb07353f700 1 -- 172.21.15.110:0/1638341530 >> 172.21.15.110:0/1638341530 conn(0x7fb06c0cfec0 msgr2=0x7fb06c158990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:30:20.354 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:20.349+0000 7fb07353f700 1 -- 172.21.15.110:0/1638341530 shutdown_connections 2023-10-06T03:30:20.354 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:20.349+0000 7fb07353f700 1 -- 172.21.15.110:0/1638341530 wait complete. 2023-10-06T03:30:20.365 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-06T03:30:20.469 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:20.465+0000 7f8c81f8e700 1 Processor -- start 2023-10-06T03:30:20.469 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:20.465+0000 7f8c81f8e700 1 -- start start 2023-10-06T03:30:20.471 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:20.465+0000 7f8c81f8e700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f8c7c0ca020 0x7f8c7c1592c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:20.471 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:20.465+0000 7f8c81f8e700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f8c7c06f340 0x7f8c7c0adcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:20.471 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:20.465+0000 7f8c81f8e700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c7c0ca500 con 0x7f8c7c155430 2023-10-06T03:30:20.472 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:20.465+0000 7f8c81f8e700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8c7c0ca7e0 con 0x7f8c7c06f340 2023-10-06T03:30:20.472 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:20.465+0000 7f8c81f8e700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8c7c0caac0 con 0x7f8c7c0ca020 2023-10-06T03:30:20.472 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:20.465+0000 7f8c7bfff700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f8c7c06f340 0x7f8c7c0adcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:20.472 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:20.465+0000 7f8c7bfff700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f8c7c06f340 0x7f8c7c0adcd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:52418/0 (socket says 172.21.15.110:52418) 2023-10-06T03:30:20.472 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:20.465+0000 7f8c7bfff700 1 -- 172.21.15.110:0/4230756812 learned_addr learned my addr 172.21.15.110:0/4230756812 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:30:20.472 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:20.465+0000 7f8c7bfff700 1 -- 172.21.15.110:0/4230756812 >> v2:172.21.15.192:3300/0 conn(0x7f8c7c0ca020 msgr2=0x7f8c7c1592c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:30:20.472 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:20.465+0000 7f8c80f8c700 1 --2- 172.21.15.110:0/4230756812 >> v2:172.21.15.192:3300/0 conn(0x7f8c7c0ca020 0x7f8c7c1592c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:20.472 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:20.465+0000 7f8c7b7fe700 1 -- 172.21.15.110:0/4230756812 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 798707652 0 0) 0x7f8c7c0ca500 con 0x7f8c7c155430 2023-10-06T03:30:20.472 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:20.465+0000 7f8c7bfff700 1 --2- 172.21.15.110:0/4230756812 >> v2:172.21.15.192:3300/0 conn(0x7f8c7c0ca020 0x7f8c7c1592c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:20.472 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:20.465+0000 7f8c7bfff700 1 -- 172.21.15.110:0/4230756812 >> v1:172.21.15.110:6789/0 conn(0x7f8c7c155430 legacy=0x7f8c7c1558b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:20.472 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:20.465+0000 7f8c7bfff700 1 -- 172.21.15.110:0/4230756812 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c7c0cada0 con 0x7f8c7c06f340 2023-10-06T03:30:20.473 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:20.465+0000 7f8c80f8c700 1 --2- 172.21.15.110:0/4230756812 >> v2:172.21.15.192:3300/0 conn(0x7f8c7c0ca020 0x7f8c7c1592c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-06T03:30:20.473 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:20.465+0000 7f8c7bfff700 1 --2- 172.21.15.110:0/4230756812 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8c7c06f340 0x7f8c7c0adcd0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f8c7c0c9db0 tx=0x7f8c6c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb94c593fa1f3a18 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:30:20.473 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:20.465+0000 7f8c7b7fe700 1 -- 172.21.15.110:0/4230756812 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c6c00b860 con 0x7f8c7c06f340 2023-10-06T03:30:20.473 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:20.465+0000 7f8c7b7fe700 1 -- 172.21.15.110:0/4230756812 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c6c00b9c0 con 0x7f8c7c06f340 2023-10-06T03:30:20.473 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:20.465+0000 7f8c7b7fe700 1 -- 172.21.15.110:0/4230756812 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c6c00bd60 con 0x7f8c7c06f340 2023-10-06T03:30:20.474 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:20.469+0000 7f8c81f8e700 1 -- 172.21.15.110:0/4230756812 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8c7c06f340 msgr2=0x7f8c7c0adcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:20.474 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:20.469+0000 7f8c81f8e700 1 --2- 172.21.15.110:0/4230756812 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8c7c06f340 0x7f8c7c0adcd0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f8c7c0c9db0 tx=0x7f8c6c009250 comp rx=0 tx=0).stop 2023-10-06T03:30:20.475 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:20.469+0000 7f8c81f8e700 1 -- 172.21.15.110:0/4230756812 shutdown_connections 2023-10-06T03:30:20.475 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:20.469+0000 7f8c81f8e700 1 --2- 172.21.15.110:0/4230756812 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8c7c06f340 0x7f8c7c0adcd0 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:20.475 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:20.469+0000 7f8c81f8e700 1 --2- 172.21.15.110:0/4230756812 >> v2:172.21.15.192:3300/0 conn(0x7f8c7c0ca020 0x7f8c7c1592c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:20.475 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:20.469+0000 7f8c81f8e700 1 -- 172.21.15.110:0/4230756812 >> 172.21.15.110:0/4230756812 conn(0x7f8c7c0cc470 msgr2=0x7f8c7c0c8410 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:30:20.475 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:20.469+0000 7f8c81f8e700 1 -- 172.21.15.110:0/4230756812 shutdown_connections 2023-10-06T03:30:20.476 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:20.469+0000 7f8c81f8e700 1 -- 172.21.15.110:0/4230756812 wait complete. 2023-10-06T03:30:20.480 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:20.473+0000 7f8c81f8e700 1 Processor -- start 2023-10-06T03:30:20.480 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:20.473+0000 7f8c81f8e700 1 -- start start 2023-10-06T03:30:20.480 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:20.473+0000 7f8c81f8e700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8c7c06f340 0x7f8c7c151c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:20.480 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:20.473+0000 7f8c81f8e700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8c7c0ca020 0x7f8c7c06c3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:20.480 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:20.473+0000 7f8c81f8e700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8c7c155430 0x7f8c7c06c930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:20.481 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:20.473+0000 7f8c81f8e700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c7c151a10 con 0x7f8c7c06f340 2023-10-06T03:30:20.481 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:20.473+0000 7f8c81f8e700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c7c0c1470 con 0x7f8c7c155430 2023-10-06T03:30:20.481 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:20.473+0000 7f8c81f8e700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c7c0c1900 con 0x7f8c7c0ca020 2023-10-06T03:30:20.481 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:20.473+0000 7f8c80f8c700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8c7c06f340 0x7f8c7c151c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:20.481 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:20.473+0000 7f8c80f8c700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8c7c06f340 0x7f8c7c151c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:52428/0 (socket says 172.21.15.110:52428) 2023-10-06T03:30:20.481 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:20.473+0000 7f8c80f8c700 1 -- 172.21.15.110:0/1607247802 learned_addr learned my addr 172.21.15.110:0/1607247802 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:30:20.482 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:20.473+0000 7f8c8178d700 1 --2- 172.21.15.110:0/1607247802 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8c7c155430 0x7f8c7c06c930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:20.482 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:20.473+0000 7f8c7bfff700 1 --2- 172.21.15.110:0/1607247802 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8c7c0ca020 0x7f8c7c06c3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:20.482 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:20.473+0000 7f8c80f8c700 1 -- 172.21.15.110:0/1607247802 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8c7c0ca020 msgr2=0x7f8c7c06c3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:20.482 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:20.473+0000 7f8c80f8c700 1 --2- 172.21.15.110:0/1607247802 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8c7c0ca020 0x7f8c7c06c3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:20.482 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:20.473+0000 7f8c80f8c700 1 -- 172.21.15.110:0/1607247802 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8c7c155430 msgr2=0x7f8c7c06c930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:20.482 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:20.473+0000 7f8c80f8c700 1 --2- 172.21.15.110:0/1607247802 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8c7c155430 0x7f8c7c06c930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:20.482 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:20.473+0000 7f8c80f8c700 1 -- 172.21.15.110:0/1607247802 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c6c0094f0 con 0x7f8c7c06f340 2023-10-06T03:30:20.482 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:20.473+0000 7f8c7bfff700 1 --2- 172.21.15.110:0/1607247802 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8c7c0ca020 0x7f8c7c06c3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-06T03:30:20.482 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:20.473+0000 7f8c80f8c700 1 --2- 172.21.15.110:0/1607247802 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8c7c06f340 0x7f8c7c151c20 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f8c6400c860 tx=0x7f8c64008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:30:20.483 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:20.473+0000 7f8c78ff9700 1 -- 172.21.15.110:0/1607247802 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c64011600 con 0x7f8c7c06f340 2023-10-06T03:30:20.483 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:20.473+0000 7f8c78ff9700 1 -- 172.21.15.110:0/1607247802 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c64011760 con 0x7f8c7c06f340 2023-10-06T03:30:20.483 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:20.473+0000 7f8c81f8e700 1 -- 172.21.15.110:0/1607247802 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c7c0c0860 con 0x7f8c7c06f340 2023-10-06T03:30:20.483 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:20.473+0000 7f8c78ff9700 1 -- 172.21.15.110:0/1607247802 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c64011b20 con 0x7f8c7c06f340 2023-10-06T03:30:20.483 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:20.473+0000 7f8c81f8e700 1 -- 172.21.15.110:0/1607247802 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c7c0c0e20 con 0x7f8c7c06f340 2023-10-06T03:30:20.483 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:20.477+0000 7f8c81f8e700 1 -- 172.21.15.110:0/1607247802 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c7c070ff0 con 0x7f8c7c06f340 2023-10-06T03:30:20.484 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:20.477+0000 7f8c78ff9700 1 -- 172.21.15.110:0/1607247802 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7f8c640193b0 con 0x7f8c7c06f340 2023-10-06T03:30:20.484 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:20.477+0000 7f8c78ff9700 1 --2- 172.21.15.110:0/1607247802 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f8c5c060550 0x7f8c5c062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:20.484 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:20.477+0000 7f8c78ff9700 1 -- 172.21.15.110:0/1607247802 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 6867+0+0 (secure 0 0 0) 0x7f8c6407cc30 con 0x7f8c7c06f340 2023-10-06T03:30:20.484 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:20.477+0000 7f8c7bfff700 1 --2- 172.21.15.110:0/1607247802 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f8c5c060550 0x7f8c5c062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:20.484 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:20.477+0000 7f8c7bfff700 1 --2- 172.21.15.110:0/1607247802 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f8c5c060550 0x7f8c5c062a00 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f8c7c0c9db0 tx=0x7f8c6c017000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:30:20.487 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:20.485+0000 7f8c78ff9700 1 -- 172.21.15.110:0/1607247802 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f8c6404cd90 con 0x7f8c7c06f340 2023-10-06T03:30:20.682 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:20.677+0000 7f8c81f8e700 1 -- 172.21.15.110:0/1607247802 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8c7c0cb080 con 0x7f8c7c06f340 2023-10-06T03:30:20.683 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:20.681+0000 7f8c78ff9700 1 -- 172.21.15.110:0/1607247802 <== mon.0 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4602 (secure 0 0 0) 0x7f8c64015090 con 0x7f8c7c06f340 2023-10-06T03:30:20.684 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-10-06T03:30:20.684 INFO:teuthology.orchestra.run.smithi110.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4279,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6833/1434505700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6832","nonce":1434505700},{"type":"v1","addr":"172.21.15.192:6833","nonce":1434505700}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4281,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/1168370789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6834","nonce":1168370789},{"type":"v1","addr":"172.21.15.110:6835","nonce":1168370789}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4288,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6835/1271810086","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":1271810086},{"type":"v1","addr":"172.21.15.192:6835","nonce":1271810086}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-06T03:29:33.894998+0000","modified":"2023-10-06T03:29:47.307075+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4293},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4293":{"gid":4293,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.110:6837/540320464","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6836","nonce":540320464},{"type":"v1","addr":"172.21.15.110:6837","nonce":540320464}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-06T03:30:20.688 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:20.685+0000 7f8c81f8e700 1 -- 172.21.15.110:0/1607247802 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f8c5c060550 msgr2=0x7f8c5c062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:20.688 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:20.685+0000 7f8c81f8e700 1 --2- 172.21.15.110:0/1607247802 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f8c5c060550 0x7f8c5c062a00 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f8c7c0c9db0 tx=0x7f8c6c017000 comp rx=0 tx=0).stop 2023-10-06T03:30:20.689 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:20.685+0000 7f8c81f8e700 1 -- 172.21.15.110:0/1607247802 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8c7c06f340 msgr2=0x7f8c7c151c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:20.689 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:20.685+0000 7f8c81f8e700 1 --2- 172.21.15.110:0/1607247802 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8c7c06f340 0x7f8c7c151c20 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f8c6400c860 tx=0x7f8c64008040 comp rx=0 tx=0).stop 2023-10-06T03:30:20.689 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:20.685+0000 7f8c81f8e700 1 -- 172.21.15.110:0/1607247802 shutdown_connections 2023-10-06T03:30:20.689 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:20.685+0000 7f8c81f8e700 1 --2- 172.21.15.110:0/1607247802 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f8c5c060550 0x7f8c5c062a00 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:20.689 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:20.685+0000 7f8c81f8e700 1 --2- 172.21.15.110:0/1607247802 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8c7c155430 0x7f8c7c06c930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:20.690 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:20.685+0000 7f8c81f8e700 1 --2- 172.21.15.110:0/1607247802 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8c7c0ca020 0x7f8c7c06c3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:20.690 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:20.685+0000 7f8c81f8e700 1 --2- 172.21.15.110:0/1607247802 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8c7c06f340 0x7f8c7c151c20 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:20.690 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:20.685+0000 7f8c81f8e700 1 -- 172.21.15.110:0/1607247802 >> 172.21.15.110:0/1607247802 conn(0x7f8c7c0cc470 msgr2=0x7f8c7c0693e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:30:20.690 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:20.685+0000 7f8c81f8e700 1 -- 172.21.15.110:0/1607247802 shutdown_connections 2023-10-06T03:30:20.690 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:20.685+0000 7f8c81f8e700 1 -- 172.21.15.110:0/1607247802 wait complete. 2023-10-06T03:30:20.691 INFO:teuthology.orchestra.run.smithi110.stderr:dumped fsmap epoch 8 2023-10-06T03:30:20.705 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-06T03:30:20.813 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:20.805+0000 7f4e49864700 1 Processor -- start 2023-10-06T03:30:20.813 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:20.805+0000 7f4e49864700 1 -- start start 2023-10-06T03:30:20.813 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:20.805+0000 7f4e49864700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f4e4415d420 0x7f4e440c7b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:20.814 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:20.805+0000 7f4e49864700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f4e44069d70 0x7f4e440ca270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:20.814 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:20.805+0000 7f4e49864700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4e4415d7f0 0x7f4e440ca7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:20.814 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:20.805+0000 7f4e49864700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4e4406aef0 con 0x7f4e4415d420 2023-10-06T03:30:20.815 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:20.805+0000 7f4e49864700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4e4406a2e0 con 0x7f4e44069d70 2023-10-06T03:30:20.815 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:20.805+0000 7f4e49864700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e4406a5c0 con 0x7f4e4415d7f0 2023-10-06T03:30:20.815 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:20.805+0000 7f4e42ffd700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f4e4415d420 0x7f4e440c7b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:20.816 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:20.805+0000 7f4e42ffd700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f4e4415d420 0x7f4e440c7b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:52432/0 (socket says 172.21.15.110:52432) 2023-10-06T03:30:20.816 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:20.805+0000 7f4e42ffd700 1 -- 172.21.15.110:0/905454317 learned_addr learned my addr 172.21.15.110:0/905454317 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:30:20.816 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:20.805+0000 7f4e42ffd700 1 -- 172.21.15.110:0/905454317 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4e4415d7f0 msgr2=0x7f4e440ca7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:30:20.817 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:20.805+0000 7f4e427fc700 1 --2- 172.21.15.110:0/905454317 >> v2:172.21.15.192:3300/0 conn(0x7f4e44069d70 0x7f4e440ca270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:20.817 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:20.805+0000 7f4e437fe700 1 --2- 172.21.15.110:0/905454317 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4e4415d7f0 0x7f4e440ca7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:20.817 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:20.805+0000 7f4e42ffd700 1 --2- 172.21.15.110:0/905454317 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4e4415d7f0 0x7f4e440ca7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:20.818 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:20.805+0000 7f4e42ffd700 1 -- 172.21.15.110:0/905454317 >> v2:172.21.15.192:3300/0 conn(0x7f4e44069d70 msgr2=0x7f4e440ca270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:30:20.818 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:20.805+0000 7f4e42ffd700 1 --2- 172.21.15.110:0/905454317 >> v2:172.21.15.192:3300/0 conn(0x7f4e44069d70 0x7f4e440ca270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:20.819 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:20.805+0000 7f4e42ffd700 1 -- 172.21.15.110:0/905454317 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e4406a8a0 con 0x7f4e4415d420 2023-10-06T03:30:20.819 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:20.805+0000 7f4e427fc700 1 --2- 172.21.15.110:0/905454317 >> v2:172.21.15.192:3300/0 conn(0x7f4e44069d70 0x7f4e440ca270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-06T03:30:20.820 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:20.805+0000 7f4e437fe700 1 --2- 172.21.15.110:0/905454317 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4e4415d7f0 0x7f4e440ca7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-06T03:30:20.820 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:20.805+0000 7f4e42ffd700 1 --2- 172.21.15.110:0/905454317 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4e4415d420 0x7f4e440c7b90 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f4e44067ed0 tx=0x7f4e2c009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2fbf7c2fc9e72a27 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:30:20.821 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:20.805+0000 7f4e41ffb700 1 -- 172.21.15.110:0/905454317 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e2c00e070 con 0x7f4e4415d420 2023-10-06T03:30:20.822 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:20.805+0000 7f4e41ffb700 1 -- 172.21.15.110:0/905454317 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e2c00ba50 con 0x7f4e4415d420 2023-10-06T03:30:20.822 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:20.805+0000 7f4e41ffb700 1 -- 172.21.15.110:0/905454317 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e2c0092e0 con 0x7f4e4415d420 2023-10-06T03:30:20.822 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:20.805+0000 7f4e49864700 1 -- 172.21.15.110:0/905454317 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4e4415d420 msgr2=0x7f4e440c7b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:20.822 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:20.805+0000 7f4e49864700 1 --2- 172.21.15.110:0/905454317 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4e4415d420 0x7f4e440c7b90 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f4e44067ed0 tx=0x7f4e2c009840 comp rx=0 tx=0).stop 2023-10-06T03:30:20.823 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:20.809+0000 7f4e49864700 1 -- 172.21.15.110:0/905454317 shutdown_connections 2023-10-06T03:30:20.823 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:20.809+0000 7f4e49864700 1 --2- 172.21.15.110:0/905454317 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4e4415d7f0 0x7f4e440ca7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:20.823 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:20.809+0000 7f4e49864700 1 --2- 172.21.15.110:0/905454317 >> v2:172.21.15.192:3300/0 conn(0x7f4e44069d70 0x7f4e440ca270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:20.823 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:20.809+0000 7f4e49864700 1 --2- 172.21.15.110:0/905454317 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4e4415d420 0x7f4e440c7b90 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:20.823 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:20.809+0000 7f4e49864700 1 -- 172.21.15.110:0/905454317 >> 172.21.15.110:0/905454317 conn(0x7f4e440cbd00 msgr2=0x7f4e440cc100 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:30:20.823 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:20.809+0000 7f4e49864700 1 -- 172.21.15.110:0/905454317 shutdown_connections 2023-10-06T03:30:20.823 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:20.809+0000 7f4e49864700 1 -- 172.21.15.110:0/905454317 wait complete. 2023-10-06T03:30:20.826 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:20.809+0000 7f4e49864700 1 Processor -- start 2023-10-06T03:30:20.826 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:20.809+0000 7f4e49864700 1 -- start start 2023-10-06T03:30:20.826 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:20.813+0000 7f4e49864700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4e44069d70 0x7f4e4414f160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:20.826 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:20.813+0000 7f4e49864700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4e4415d420 0x7f4e44162310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:20.826 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:20.813+0000 7f4e49864700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4e4415d7f0 0x7f4e44163d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:20.826 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:20.813+0000 7f4e49864700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e44163890 con 0x7f4e4415d420 2023-10-06T03:30:20.826 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:20.813+0000 7f4e49864700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e44162c80 con 0x7f4e4415d7f0 2023-10-06T03:30:20.826 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:20.813+0000 7f4e49864700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e44162f60 con 0x7f4e44069d70 2023-10-06T03:30:20.827 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:20.813+0000 7f4e427fc700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4e4415d420 0x7f4e44162310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:20.827 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:20.813+0000 7f4e427fc700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4e4415d420 0x7f4e44162310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:52444/0 (socket says 172.21.15.110:52444) 2023-10-06T03:30:20.827 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:20.813+0000 7f4e427fc700 1 -- 172.21.15.110:0/1279017469 learned_addr learned my addr 172.21.15.110:0/1279017469 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:30:20.827 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:20.813+0000 7f4e42ffd700 1 --2- 172.21.15.110:0/1279017469 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4e44069d70 0x7f4e4414f160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:20.827 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:20.813+0000 7f4e437fe700 1 --2- 172.21.15.110:0/1279017469 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4e4415d7f0 0x7f4e44163d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:20.827 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:20.813+0000 7f4e427fc700 1 -- 172.21.15.110:0/1279017469 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4e44069d70 msgr2=0x7f4e4414f160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:20.827 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:20.813+0000 7f4e427fc700 1 --2- 172.21.15.110:0/1279017469 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4e44069d70 0x7f4e4414f160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:20.827 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:20.813+0000 7f4e427fc700 1 -- 172.21.15.110:0/1279017469 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4e4415d7f0 msgr2=0x7f4e44163d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:20.827 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:20.813+0000 7f4e427fc700 1 --2- 172.21.15.110:0/1279017469 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4e4415d7f0 0x7f4e44163d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:20.827 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:20.813+0000 7f4e427fc700 1 -- 172.21.15.110:0/1279017469 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e2c009ae0 con 0x7f4e4415d420 2023-10-06T03:30:20.827 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:20.813+0000 7f4e427fc700 1 --2- 172.21.15.110:0/1279017469 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4e4415d420 0x7f4e44162310 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f4e3400c880 tx=0x7f4e34008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:30:20.828 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:20.813+0000 7f4e3affd700 1 -- 172.21.15.110:0/1279017469 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e34011600 con 0x7f4e4415d420 2023-10-06T03:30:20.828 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:20.813+0000 7f4e3affd700 1 -- 172.21.15.110:0/1279017469 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e34011760 con 0x7f4e4415d420 2023-10-06T03:30:20.828 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:20.813+0000 7f4e49864700 1 -- 172.21.15.110:0/1279017469 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e44163240 con 0x7f4e4415d420 2023-10-06T03:30:20.828 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:20.813+0000 7f4e3affd700 1 -- 172.21.15.110:0/1279017469 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e34011ae0 con 0x7f4e4415d420 2023-10-06T03:30:20.828 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:20.813+0000 7f4e49864700 1 -- 172.21.15.110:0/1279017469 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e4414f6a0 con 0x7f4e4415d420 2023-10-06T03:30:20.828 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:20.817+0000 7f4e3affd700 1 -- 172.21.15.110:0/1279017469 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7f4e34010350 con 0x7f4e4415d420 2023-10-06T03:30:20.828 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:20.817+0000 7f4e49864700 1 -- 172.21.15.110:0/1279017469 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e4406ff60 con 0x7f4e4415d420 2023-10-06T03:30:20.828 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:20.817+0000 7f4e3affd700 1 --2- 172.21.15.110:0/1279017469 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f4e28060550 0x7f4e28062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:20.828 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:20.817+0000 7f4e3affd700 1 -- 172.21.15.110:0/1279017469 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 6867+0+0 (secure 0 0 0) 0x7f4e3407cbc0 con 0x7f4e4415d420 2023-10-06T03:30:20.828 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:20.817+0000 7f4e42ffd700 1 --2- 172.21.15.110:0/1279017469 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f4e28060550 0x7f4e28062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:20.828 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:20.817+0000 7f4e42ffd700 1 --2- 172.21.15.110:0/1279017469 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f4e28060550 0x7f4e28062a00 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f4e2c009880 tx=0x7f4e2c018000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:30:20.829 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:20.825+0000 7f4e3affd700 1 -- 172.21.15.110:0/1279017469 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f4e3404cd20 con 0x7f4e4415d420 2023-10-06T03:30:20.982 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:20.977+0000 7f4e49864700 1 -- 172.21.15.110:0/1279017469 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4e440cc270 con 0x7f4e4415d420 2023-10-06T03:30:20.983 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:20.981+0000 7f4e3affd700 1 -- 172.21.15.110:0/1279017469 <== mon.0 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v30) v1 ==== 74+0+18850 (secure 0 0 0) 0x7f4e34050340 con 0x7f4e4415d420 2023-10-06T03:30:20.984 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-10-06T03:30:20.984 INFO:teuthology.orchestra.run.smithi110.stdout:{"epoch":30,"fsid":"93b8f6ac-4c92-4b45-b8e4-b59617024aa2","created":"2023-10-06T03:29:10.519583+0000","modified":"2023-10-06T03:30:16.473472+0000","last_up_change":"2023-10-06T03:29:21.960511+0000","last_in_change":"2023-10-06T03:29:12.560436+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-06T03:29:20.703743+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2023-10-06T03:29:28.307281+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2023-10-06T03:29:32.247251+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data","create_time":"2023-10-06T03:29:32.880086+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}}},{"pool":5,"pool_name":"cephfs_data_ec","create_time":"2023-10-06T03:29:36.613779+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"28","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"5866eb6c-2adc-46e4-be7a-e5bcda0f4065","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6824","nonce":33394},{"type":"v1","addr":"172.21.15.110:6825","nonce":33394}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6826","nonce":33394},{"type":"v1","addr":"172.21.15.110:6827","nonce":33394}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6830","nonce":33394},{"type":"v1","addr":"172.21.15.110:6831","nonce":33394}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6828","nonce":33394},{"type":"v1","addr":"172.21.15.110:6829","nonce":33394}]},"public_addr":"172.21.15.110:6825/33394","cluster_addr":"172.21.15.110:6827/33394","heartbeat_back_addr":"172.21.15.110:6831/33394","heartbeat_front_addr":"172.21.15.110:6829/33394","state":["exists","up"]},{"osd":1,"uuid":"a38de118-062b-4a05-9782-2ceddf6ff4df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6800","nonce":33396},{"type":"v1","addr":"172.21.15.110:6801","nonce":33396}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6802","nonce":33396},{"type":"v1","addr":"172.21.15.110:6803","nonce":33396}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6806","nonce":33396},{"type":"v1","addr":"172.21.15.110:6807","nonce":33396}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6804","nonce":33396},{"type":"v1","addr":"172.21.15.110:6805","nonce":33396}]},"public_addr":"172.21.15.110:6801/33396","cluster_addr":"172.21.15.110:6803/33396","heartbeat_back_addr":"172.21.15.110:6807/33396","heartbeat_front_addr":"172.21.15.110:6805/33396","state":["exists","up"]},{"osd":2,"uuid":"56e39be2-3407-4b69-877e-cf16617b06d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6816","nonce":33397},{"type":"v1","addr":"172.21.15.110:6817","nonce":33397}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6818","nonce":33397},{"type":"v1","addr":"172.21.15.110:6819","nonce":33397}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6822","nonce":33397},{"type":"v1","addr":"172.21.15.110:6823","nonce":33397}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6820","nonce":33397},{"type":"v1","addr":"172.21.15.110:6821","nonce":33397}]},"public_addr":"172.21.15.110:6817/33397","cluster_addr":"172.21.15.110:6819/33397","heartbeat_back_addr":"172.21.15.110:6823/33397","heartbeat_front_addr":"172.21.15.110:6821/33397","state":["exists","up"]},{"osd":3,"uuid":"01c82d13-c1d4-4bde-b55e-3e0089b40115","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6808","nonce":33395},{"type":"v1","addr":"172.21.15.110:6809","nonce":33395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6810","nonce":33395},{"type":"v1","addr":"172.21.15.110:6811","nonce":33395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6814","nonce":33395},{"type":"v1","addr":"172.21.15.110:6815","nonce":33395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6812","nonce":33395},{"type":"v1","addr":"172.21.15.110:6813","nonce":33395}]},"public_addr":"172.21.15.110:6809/33395","cluster_addr":"172.21.15.110:6811/33395","heartbeat_back_addr":"172.21.15.110:6815/33395","heartbeat_front_addr":"172.21.15.110:6813/33395","state":["exists","up"]},{"osd":4,"uuid":"86983430-5fa3-4d17-998f-03e2cd843237","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":33683},{"type":"v1","addr":"172.21.15.192:6817","nonce":33683}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":33683},{"type":"v1","addr":"172.21.15.192:6819","nonce":33683}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":33683},{"type":"v1","addr":"172.21.15.192:6823","nonce":33683}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6820","nonce":33683},{"type":"v1","addr":"172.21.15.192:6821","nonce":33683}]},"public_addr":"172.21.15.192:6817/33683","cluster_addr":"172.21.15.192:6819/33683","heartbeat_back_addr":"172.21.15.192:6823/33683","heartbeat_front_addr":"172.21.15.192:6821/33683","state":["exists","up"]},{"osd":5,"uuid":"6945257f-821d-4619-a63f-ebe857eba6f4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":33684},{"type":"v1","addr":"172.21.15.192:6801","nonce":33684}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":33684},{"type":"v1","addr":"172.21.15.192:6803","nonce":33684}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":33684},{"type":"v1","addr":"172.21.15.192:6807","nonce":33684}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6804","nonce":33684},{"type":"v1","addr":"172.21.15.192:6805","nonce":33684}]},"public_addr":"172.21.15.192:6801/33684","cluster_addr":"172.21.15.192:6803/33684","heartbeat_back_addr":"172.21.15.192:6807/33684","heartbeat_front_addr":"172.21.15.192:6805/33684","state":["exists","up"]},{"osd":6,"uuid":"7854d1ae-7060-4780-8f97-73cea5e7a12f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6824","nonce":33686},{"type":"v1","addr":"172.21.15.192:6825","nonce":33686}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":33686},{"type":"v1","addr":"172.21.15.192:6827","nonce":33686}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":33686},{"type":"v1","addr":"172.21.15.192:6831","nonce":33686}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6828","nonce":33686},{"type":"v1","addr":"172.21.15.192:6829","nonce":33686}]},"public_addr":"172.21.15.192:6825/33686","cluster_addr":"172.21.15.192:6827/33686","heartbeat_back_addr":"172.21.15.192:6831/33686","heartbeat_front_addr":"172.21.15.192:6829/33686","state":["exists","up"]},{"osd":7,"uuid":"f1ebdfd5-b859-482b-9f97-c18c4a26f445","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6808","nonce":33685},{"type":"v1","addr":"172.21.15.192:6809","nonce":33685}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":33685},{"type":"v1","addr":"172.21.15.192:6811","nonce":33685}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":33685},{"type":"v1","addr":"172.21.15.192:6815","nonce":33685}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6812","nonce":33685},{"type":"v1","addr":"172.21.15.192:6813","nonce":33685}]},"public_addr":"172.21.15.192:6809/33685","cluster_addr":"172.21.15.192:6811/33685","heartbeat_back_addr":"172.21.15.192:6815/33685","heartbeat_front_addr":"172.21.15.192:6813/33685","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T03:29:18.002048+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T03:29:17.478576+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T03:29:18.034944+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T03:29:17.628513+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T03:29:18.047338+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T03:29:17.772349+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T03:29:17.883958+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T03:29:18.151838+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"4.7","mappings":[{"from":1,"to":2}]},{"pgid":"4.3b","mappings":[{"from":3,"to":2}]}],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-06T03:30:20.986 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:20.981+0000 7f4e49864700 1 -- 172.21.15.110:0/1279017469 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f4e28060550 msgr2=0x7f4e28062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:20.986 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:20.981+0000 7f4e49864700 1 --2- 172.21.15.110:0/1279017469 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f4e28060550 0x7f4e28062a00 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f4e2c009880 tx=0x7f4e2c018000 comp rx=0 tx=0).stop 2023-10-06T03:30:20.986 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:20.981+0000 7f4e49864700 1 -- 172.21.15.110:0/1279017469 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4e4415d420 msgr2=0x7f4e44162310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:20.986 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:20.981+0000 7f4e49864700 1 --2- 172.21.15.110:0/1279017469 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4e4415d420 0x7f4e44162310 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f4e3400c880 tx=0x7f4e34008040 comp rx=0 tx=0).stop 2023-10-06T03:30:20.986 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:20.981+0000 7f4e49864700 1 -- 172.21.15.110:0/1279017469 shutdown_connections 2023-10-06T03:30:20.987 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:20.981+0000 7f4e49864700 1 --2- 172.21.15.110:0/1279017469 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f4e28060550 0x7f4e28062a00 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:20.987 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:20.981+0000 7f4e49864700 1 --2- 172.21.15.110:0/1279017469 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4e4415d7f0 0x7f4e44163d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:20.987 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:20.981+0000 7f4e49864700 1 --2- 172.21.15.110:0/1279017469 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4e4415d420 0x7f4e44162310 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:20.987 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:20.981+0000 7f4e49864700 1 --2- 172.21.15.110:0/1279017469 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4e44069d70 0x7f4e4414f160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:20.987 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:20.981+0000 7f4e49864700 1 -- 172.21.15.110:0/1279017469 >> 172.21.15.110:0/1279017469 conn(0x7f4e440cbd00 msgr2=0x7f4e440c1720 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:30:20.987 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:20.981+0000 7f4e49864700 1 -- 172.21.15.110:0/1279017469 shutdown_connections 2023-10-06T03:30:20.987 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:20.981+0000 7f4e49864700 1 -- 172.21.15.110:0/1279017469 wait complete. 2023-10-06T03:30:20.998 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2023-10-06T03:30:20.998 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-06T03:30:21.106 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:21.101+0000 7ff347435700 1 Processor -- start 2023-10-06T03:30:21.106 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:21.101+0000 7ff347435700 1 -- start start 2023-10-06T03:30:21.107 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:21.101+0000 7ff347435700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7ff3400ca520 0x7ff3400aec00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:21.107 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:21.101+0000 7ff347435700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7ff340158990 0x7ff340158e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:21.107 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:21.101+0000 7ff347435700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3400caf80 con 0x7ff34015cbf0 2023-10-06T03:30:21.107 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:21.101+0000 7ff347435700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff3400c27c0 con 0x7ff340158990 2023-10-06T03:30:21.107 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:21.101+0000 7ff347435700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff34006bf40 con 0x7ff3400ca520 2023-10-06T03:30:21.107 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:21.101+0000 7ff346c34700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7ff340158990 0x7ff340158e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:21.107 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:21.101+0000 7ff346c34700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7ff340158990 0x7ff340158e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:40006/0 (socket says 172.21.15.110:40006) 2023-10-06T03:30:21.107 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:21.101+0000 7ff346c34700 1 -- 172.21.15.110:0/2411907410 learned_addr learned my addr 172.21.15.110:0/2411907410 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:30:21.107 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:21.101+0000 7ff345c32700 1 --2- 172.21.15.110:0/2411907410 >> v2:172.21.15.192:3300/0 conn(0x7ff3400ca520 0x7ff3400aec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:21.109 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:21.105+0000 7ff346c34700 1 -- 172.21.15.110:0/2411907410 >> v2:172.21.15.192:3300/0 conn(0x7ff3400ca520 msgr2=0x7ff3400aec00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:30:21.109 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:21.105+0000 7ff346c34700 1 --2- 172.21.15.110:0/2411907410 >> v2:172.21.15.192:3300/0 conn(0x7ff3400ca520 0x7ff3400aec00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:21.109 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:21.105+0000 7ff346c34700 1 -- 172.21.15.110:0/2411907410 >> v1:172.21.15.110:6789/0 conn(0x7ff34015cbf0 legacy=0x7ff34015cfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:21.109 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:21.105+0000 7ff346c34700 1 -- 172.21.15.110:0/2411907410 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3400af140 con 0x7ff340158990 2023-10-06T03:30:21.109 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:21.105+0000 7ff346c34700 1 --2- 172.21.15.110:0/2411907410 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff340158990 0x7ff340158e10 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7ff3400b1860 tx=0x7ff330009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=153b40d5754e55a5 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:30:21.109 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:21.105+0000 7ff345431700 1 -- 172.21.15.110:0/2411907410 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff330009e70 con 0x7ff340158990 2023-10-06T03:30:21.109 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:21.105+0000 7ff345431700 1 -- 172.21.15.110:0/2411907410 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff33000b860 con 0x7ff340158990 2023-10-06T03:30:21.110 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:21.105+0000 7ff345431700 1 -- 172.21.15.110:0/2411907410 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff33000bc00 con 0x7ff340158990 2023-10-06T03:30:21.110 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:21.105+0000 7ff347435700 1 -- 172.21.15.110:0/2411907410 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff340158990 msgr2=0x7ff340158e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:21.110 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:21.105+0000 7ff347435700 1 --2- 172.21.15.110:0/2411907410 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff340158990 0x7ff340158e10 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7ff3400b1860 tx=0x7ff330009250 comp rx=0 tx=0).stop 2023-10-06T03:30:21.110 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:21.105+0000 7ff347435700 1 -- 172.21.15.110:0/2411907410 shutdown_connections 2023-10-06T03:30:21.110 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:21.105+0000 7ff347435700 1 --2- 172.21.15.110:0/2411907410 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff340158990 0x7ff340158e10 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:21.110 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:21.105+0000 7ff347435700 1 --2- 172.21.15.110:0/2411907410 >> v2:172.21.15.192:3300/0 conn(0x7ff3400ca520 0x7ff3400aec00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:21.110 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:21.105+0000 7ff347435700 1 -- 172.21.15.110:0/2411907410 >> 172.21.15.110:0/2411907410 conn(0x7ff3400cc470 msgr2=0x7ff3400c8410 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:30:21.110 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:21.105+0000 7ff347435700 1 -- 172.21.15.110:0/2411907410 shutdown_connections 2023-10-06T03:30:21.110 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:21.105+0000 7ff347435700 1 -- 172.21.15.110:0/2411907410 wait complete. 2023-10-06T03:30:21.111 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:21.105+0000 7ff347435700 1 Processor -- start 2023-10-06T03:30:21.111 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:21.105+0000 7ff347435700 1 -- start start 2023-10-06T03:30:21.113 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:21.109+0000 7ff347435700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff3400ca520 0x7ff340144720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:21.113 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:21.109+0000 7ff347435700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff340158990 0x7ff34015a4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:21.113 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:21.109+0000 7ff347435700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff34015cbf0 0x7ff34015aa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:21.113 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:21.109+0000 7ff347435700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff34015bc50 con 0x7ff34015cbf0 2023-10-06T03:30:21.113 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:21.109+0000 7ff347435700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3401564e0 con 0x7ff340158990 2023-10-06T03:30:21.114 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:21.109+0000 7ff347435700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff34015b040 con 0x7ff3400ca520 2023-10-06T03:30:21.114 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:21.109+0000 7ff346c34700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff34015cbf0 0x7ff34015aa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:21.114 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:21.109+0000 7ff346c34700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff34015cbf0 0x7ff34015aa00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:40010/0 (socket says 172.21.15.110:40010) 2023-10-06T03:30:21.114 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:21.109+0000 7ff346c34700 1 -- 172.21.15.110:0/255578308 learned_addr learned my addr 172.21.15.110:0/255578308 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:30:21.114 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:21.109+0000 7ff345c32700 1 --2- 172.21.15.110:0/255578308 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff340158990 0x7ff34015a4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:21.114 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:21.109+0000 7ff346433700 1 --2- 172.21.15.110:0/255578308 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff3400ca520 0x7ff340144720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:21.114 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:21.109+0000 7ff346c34700 1 -- 172.21.15.110:0/255578308 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff3400ca520 msgr2=0x7ff340144720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:21.114 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:21.109+0000 7ff346c34700 1 --2- 172.21.15.110:0/255578308 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff3400ca520 0x7ff340144720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:21.114 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:21.109+0000 7ff346c34700 1 -- 172.21.15.110:0/255578308 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff340158990 msgr2=0x7ff34015a4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:21.114 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:21.109+0000 7ff346c34700 1 --2- 172.21.15.110:0/255578308 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff340158990 0x7ff34015a4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:21.115 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:21.109+0000 7ff346c34700 1 -- 172.21.15.110:0/255578308 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3300094f0 con 0x7ff34015cbf0 2023-10-06T03:30:21.115 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:21.109+0000 7ff346c34700 1 --2- 172.21.15.110:0/255578308 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff34015cbf0 0x7ff34015aa00 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7ff34006f3c0 tx=0x7ff330017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:30:21.115 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:21.109+0000 7ff33e7fc700 1 -- 172.21.15.110:0/255578308 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff33000baa0 con 0x7ff34015cbf0 2023-10-06T03:30:21.115 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:21.109+0000 7ff33e7fc700 1 -- 172.21.15.110:0/255578308 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff330025420 con 0x7ff34015cbf0 2023-10-06T03:30:21.115 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:21.109+0000 7ff347435700 1 -- 172.21.15.110:0/255578308 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff34015b320 con 0x7ff34015cbf0 2023-10-06T03:30:21.115 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:21.109+0000 7ff33e7fc700 1 -- 172.21.15.110:0/255578308 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3300257e0 con 0x7ff34015cbf0 2023-10-06T03:30:21.115 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:21.109+0000 7ff347435700 1 -- 172.21.15.110:0/255578308 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff34015b8e0 con 0x7ff34015cbf0 2023-10-06T03:30:21.115 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:21.109+0000 7ff347435700 1 -- 172.21.15.110:0/255578308 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff34015be00 con 0x7ff34015cbf0 2023-10-06T03:30:21.116 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:21.113+0000 7ff33e7fc700 1 -- 172.21.15.110:0/255578308 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7ff330023070 con 0x7ff34015cbf0 2023-10-06T03:30:21.117 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:21.113+0000 7ff33e7fc700 1 --2- 172.21.15.110:0/255578308 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7ff32c060550 0x7ff32c062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:21.117 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:21.113+0000 7ff33e7fc700 1 -- 172.21.15.110:0/255578308 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 6867+0+0 (secure 0 0 0) 0x7ff33007e2b0 con 0x7ff34015cbf0 2023-10-06T03:30:21.117 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:21.113+0000 7ff346433700 1 --2- 172.21.15.110:0/255578308 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7ff32c060550 0x7ff32c062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:21.118 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:21.113+0000 7ff346433700 1 --2- 172.21.15.110:0/255578308 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7ff32c060550 0x7ff32c062a00 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7ff34016bf30 tx=0x7ff334009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:30:21.121 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:21.117+0000 7ff33e7fc700 1 -- 172.21.15.110:0/255578308 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7ff33004e3f0 con 0x7ff34015cbf0 2023-10-06T03:30:21.316 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:21.313+0000 7ff347435700 1 -- 172.21.15.110:0/255578308 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff340070ff0 con 0x7ff34015cbf0 2023-10-06T03:30:21.317 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:21.313+0000 7ff33e7fc700 1 -- 172.21.15.110:0/255578308 <== mon.0 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4602 (secure 0 0 0) 0x7ff330051990 con 0x7ff34015cbf0 2023-10-06T03:30:21.317 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-10-06T03:30:21.317 INFO:teuthology.orchestra.run.smithi110.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4279,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6833/1434505700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6832","nonce":1434505700},{"type":"v1","addr":"172.21.15.192:6833","nonce":1434505700}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4281,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/1168370789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6834","nonce":1168370789},{"type":"v1","addr":"172.21.15.110:6835","nonce":1168370789}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4288,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6835/1271810086","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":1271810086},{"type":"v1","addr":"172.21.15.192:6835","nonce":1271810086}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-06T03:29:33.894998+0000","modified":"2023-10-06T03:29:47.307075+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4293},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4293":{"gid":4293,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.110:6837/540320464","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6836","nonce":540320464},{"type":"v1","addr":"172.21.15.110:6837","nonce":540320464}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-06T03:30:21.321 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:21.317+0000 7ff347435700 1 -- 172.21.15.110:0/255578308 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7ff32c060550 msgr2=0x7ff32c062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:21.321 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:21.317+0000 7ff347435700 1 --2- 172.21.15.110:0/255578308 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7ff32c060550 0x7ff32c062a00 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7ff34016bf30 tx=0x7ff334009040 comp rx=0 tx=0).stop 2023-10-06T03:30:21.321 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:21.317+0000 7ff347435700 1 -- 172.21.15.110:0/255578308 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff34015cbf0 msgr2=0x7ff34015aa00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:21.322 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:21.317+0000 7ff347435700 1 --2- 172.21.15.110:0/255578308 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff34015cbf0 0x7ff34015aa00 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7ff34006f3c0 tx=0x7ff330017000 comp rx=0 tx=0).stop 2023-10-06T03:30:21.322 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:21.317+0000 7ff347435700 1 -- 172.21.15.110:0/255578308 shutdown_connections 2023-10-06T03:30:21.322 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:21.317+0000 7ff347435700 1 --2- 172.21.15.110:0/255578308 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7ff32c060550 0x7ff32c062a00 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:21.322 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:21.317+0000 7ff347435700 1 --2- 172.21.15.110:0/255578308 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff34015cbf0 0x7ff34015aa00 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:21.322 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:21.317+0000 7ff347435700 1 --2- 172.21.15.110:0/255578308 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff340158990 0x7ff34015a4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:21.322 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:21.317+0000 7ff347435700 1 --2- 172.21.15.110:0/255578308 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff3400ca520 0x7ff340144720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:21.322 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:21.317+0000 7ff347435700 1 -- 172.21.15.110:0/255578308 >> 172.21.15.110:0/255578308 conn(0x7ff3400cc470 msgr2=0x7ff3400c5f00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:30:21.322 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:21.317+0000 7ff347435700 1 -- 172.21.15.110:0/255578308 shutdown_connections 2023-10-06T03:30:21.322 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:21.317+0000 7ff347435700 1 -- 172.21.15.110:0/255578308 wait complete. 2023-10-06T03:30:21.323 INFO:teuthology.orchestra.run.smithi110.stderr:dumped fsmap epoch 8 2023-10-06T03:30:21.337 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-06T03:30:21.445 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:21.441+0000 7f10f759e700 1 Processor -- start 2023-10-06T03:30:21.446 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:21.441+0000 7f10f759e700 1 -- start start 2023-10-06T03:30:21.448 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:21.441+0000 7f10f759e700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f10f81559d0 0x7f10f806cc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:21.448 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:21.441+0000 7f10f759e700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f10f8155da0 0x7f10f8159770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:21.448 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:21.441+0000 7f10f759e700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f10f8159cb0 0x7f10f815a0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:21.448 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:21.441+0000 7f10f759e700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f10f80ae3f0 con 0x7f10f8159cb0 2023-10-06T03:30:21.448 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:21.441+0000 7f10f759e700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f10f815a620 con 0x7f10f81559d0 2023-10-06T03:30:21.448 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:21.441+0000 7f10f759e700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10f80caad0 con 0x7f10f8155da0 2023-10-06T03:30:21.449 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:21.441+0000 7f10f6d9d700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f10f8159cb0 0x7f10f815a0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:21.449 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:21.441+0000 7f10f6d9d700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f10f8159cb0 0x7f10f815a0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:40018/0 (socket says 172.21.15.110:40018) 2023-10-06T03:30:21.449 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:21.441+0000 7f10f6d9d700 1 -- 172.21.15.110:0/3532758319 learned_addr learned my addr 172.21.15.110:0/3532758319 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:30:21.449 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:21.441+0000 7f10f5d9b700 1 --2- 172.21.15.110:0/3532758319 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f10f8155da0 0x7f10f8159770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:21.449 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:21.441+0000 7f10f659c700 1 --2- 172.21.15.110:0/3532758319 >> v2:172.21.15.192:3300/0 conn(0x7f10f81559d0 0x7f10f806cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:21.449 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:21.441+0000 7f10f6d9d700 1 -- 172.21.15.110:0/3532758319 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f10f8155da0 msgr2=0x7f10f8159770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:30:21.449 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:21.441+0000 7f10f6d9d700 1 --2- 172.21.15.110:0/3532758319 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f10f8155da0 0x7f10f8159770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:21.449 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:21.441+0000 7f10f6d9d700 1 -- 172.21.15.110:0/3532758319 >> v2:172.21.15.192:3300/0 conn(0x7f10f81559d0 msgr2=0x7f10f806cc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:30:21.449 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:21.441+0000 7f10f6d9d700 1 --2- 172.21.15.110:0/3532758319 >> v2:172.21.15.192:3300/0 conn(0x7f10f81559d0 0x7f10f806cc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:21.449 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:21.441+0000 7f10f6d9d700 1 -- 172.21.15.110:0/3532758319 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10f80b1060 con 0x7f10f8159cb0 2023-10-06T03:30:21.449 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:21.441+0000 7f10f659c700 1 --2- 172.21.15.110:0/3532758319 >> v2:172.21.15.192:3300/0 conn(0x7f10f81559d0 0x7f10f806cc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-06T03:30:21.450 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:21.441+0000 7f10f6d9d700 1 --2- 172.21.15.110:0/3532758319 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f10f8159cb0 0x7f10f815a0e0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f10f815a8a0 tx=0x7f10f0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=261d3cb87a86a13a server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:30:21.450 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:21.445+0000 7f10f559a700 1 -- 172.21.15.110:0/3532758319 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10f0009e70 con 0x7f10f8159cb0 2023-10-06T03:30:21.450 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:21.445+0000 7f10f559a700 1 -- 172.21.15.110:0/3532758319 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10f000b860 con 0x7f10f8159cb0 2023-10-06T03:30:21.450 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:21.445+0000 7f10f559a700 1 -- 172.21.15.110:0/3532758319 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10f000bc00 con 0x7f10f8159cb0 2023-10-06T03:30:21.451 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:21.445+0000 7f10f759e700 1 -- 172.21.15.110:0/3532758319 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f10f8159cb0 msgr2=0x7f10f815a0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:21.451 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:21.445+0000 7f10f759e700 1 --2- 172.21.15.110:0/3532758319 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f10f8159cb0 0x7f10f815a0e0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f10f815a8a0 tx=0x7f10f0009250 comp rx=0 tx=0).stop 2023-10-06T03:30:21.451 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:21.445+0000 7f10f759e700 1 -- 172.21.15.110:0/3532758319 shutdown_connections 2023-10-06T03:30:21.451 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:21.445+0000 7f10f759e700 1 --2- 172.21.15.110:0/3532758319 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f10f8159cb0 0x7f10f815a0e0 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:21.451 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:21.445+0000 7f10f759e700 1 --2- 172.21.15.110:0/3532758319 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f10f8155da0 0x7f10f8159770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:21.451 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:21.445+0000 7f10f759e700 1 --2- 172.21.15.110:0/3532758319 >> v2:172.21.15.192:3300/0 conn(0x7f10f81559d0 0x7f10f806cc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:21.451 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:21.445+0000 7f10f759e700 1 -- 172.21.15.110:0/3532758319 >> 172.21.15.110:0/3532758319 conn(0x7f10f80cc470 msgr2=0x7f10f80c8410 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:30:21.452 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:21.445+0000 7f10f759e700 1 -- 172.21.15.110:0/3532758319 shutdown_connections 2023-10-06T03:30:21.452 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:21.445+0000 7f10f759e700 1 -- 172.21.15.110:0/3532758319 wait complete. 2023-10-06T03:30:21.452 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:21.449+0000 7f10f759e700 1 Processor -- start 2023-10-06T03:30:21.452 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:21.449+0000 7f10f759e700 1 -- start start 2023-10-06T03:30:21.454 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:21.449+0000 7f10f759e700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f10f81559d0 0x7f10f8150140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:21.454 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:21.449+0000 7f10f759e700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f10f8155da0 0x7f10f80c47d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:21.454 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:21.449+0000 7f10f759e700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f10f8159cb0 0x7f10f814e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:21.454 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:21.449+0000 7f10f759e700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10f80c4d10 con 0x7f10f81559d0 2023-10-06T03:30:21.454 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:21.449+0000 7f10f759e700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10f80c4f40 con 0x7f10f8159cb0 2023-10-06T03:30:21.454 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:21.449+0000 7f10f759e700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10f80c5220 con 0x7f10f8155da0 2023-10-06T03:30:21.454 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:21.449+0000 7f10f659c700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f10f81559d0 0x7f10f8150140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:21.454 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:21.449+0000 7f10f659c700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f10f81559d0 0x7f10f8150140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:40030/0 (socket says 172.21.15.110:40030) 2023-10-06T03:30:21.455 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:21.449+0000 7f10f659c700 1 -- 172.21.15.110:0/1236825424 learned_addr learned my addr 172.21.15.110:0/1236825424 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:30:21.455 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:21.449+0000 7f10f5d9b700 1 --2- 172.21.15.110:0/1236825424 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f10f8155da0 0x7f10f80c47d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:21.455 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:21.449+0000 7f10f6d9d700 1 --2- 172.21.15.110:0/1236825424 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f10f8159cb0 0x7f10f814e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:21.456 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:21.449+0000 7f10f659c700 1 -- 172.21.15.110:0/1236825424 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f10f8155da0 msgr2=0x7f10f80c47d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:21.456 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:21.449+0000 7f10f659c700 1 --2- 172.21.15.110:0/1236825424 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f10f8155da0 0x7f10f80c47d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:21.456 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:21.449+0000 7f10f659c700 1 -- 172.21.15.110:0/1236825424 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f10f8159cb0 msgr2=0x7f10f814e9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:21.456 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:21.449+0000 7f10f659c700 1 --2- 172.21.15.110:0/1236825424 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f10f8159cb0 0x7f10f814e9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:21.456 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:21.449+0000 7f10f659c700 1 -- 172.21.15.110:0/1236825424 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10f00094f0 con 0x7f10f81559d0 2023-10-06T03:30:21.456 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:21.449+0000 7f10f5d9b700 1 --2- 172.21.15.110:0/1236825424 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f10f8155da0 0x7f10f80c47d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-06T03:30:21.456 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:21.449+0000 7f10f6d9d700 1 --2- 172.21.15.110:0/1236825424 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f10f8159cb0 0x7f10f814e9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-06T03:30:21.457 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:21.449+0000 7f10f659c700 1 --2- 172.21.15.110:0/1236825424 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f10f81559d0 0x7f10f8150140 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f10ec00e8d0 tx=0x7f10ec008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:30:21.457 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:21.449+0000 7f10e67fc700 1 -- 172.21.15.110:0/1236825424 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10ec00ee70 con 0x7f10f81559d0 2023-10-06T03:30:21.457 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:21.449+0000 7f10e67fc700 1 -- 172.21.15.110:0/1236825424 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10ec00c5e0 con 0x7f10f81559d0 2023-10-06T03:30:21.457 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:21.449+0000 7f10f759e700 1 -- 172.21.15.110:0/1236825424 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10f80c5500 con 0x7f10f81559d0 2023-10-06T03:30:21.457 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:21.449+0000 7f10e67fc700 1 -- 172.21.15.110:0/1236825424 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10ec00c980 con 0x7f10f81559d0 2023-10-06T03:30:21.457 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:21.449+0000 7f10f759e700 1 -- 172.21.15.110:0/1236825424 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10f80c5ac0 con 0x7f10f81559d0 2023-10-06T03:30:21.457 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:21.453+0000 7f10f759e700 1 -- 172.21.15.110:0/1236825424 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f10f8070ff0 con 0x7f10f81559d0 2023-10-06T03:30:21.458 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:21.453+0000 7f10e67fc700 1 -- 172.21.15.110:0/1236825424 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7f10ec025480 con 0x7f10f81559d0 2023-10-06T03:30:21.458 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:21.453+0000 7f10e67fc700 1 --2- 172.21.15.110:0/1236825424 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f10d8060550 0x7f10d8062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:21.458 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:21.453+0000 7f10e67fc700 1 -- 172.21.15.110:0/1236825424 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 6867+0+0 (secure 0 0 0) 0x7f10ec07cf20 con 0x7f10f81559d0 2023-10-06T03:30:21.459 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:21.453+0000 7f10f5d9b700 1 --2- 172.21.15.110:0/1236825424 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f10d8060550 0x7f10d8062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:21.459 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:21.457+0000 7f10f5d9b700 1 --2- 172.21.15.110:0/1236825424 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f10d8060550 0x7f10d8062a00 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f10f8155430 tx=0x7f10e800b040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:30:21.464 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:21.461+0000 7f10e67fc700 1 -- 172.21.15.110:0/1236825424 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f10ec050b90 con 0x7f10f81559d0 2023-10-06T03:30:21.665 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:21.661+0000 7f10f759e700 1 -- 172.21.15.110:0/1236825424 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f10f80ca1a0 con 0x7f10f81559d0 2023-10-06T03:30:21.665 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:21.661+0000 7f10e67fc700 1 -- 172.21.15.110:0/1236825424 <== mon.0 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4602 (secure 0 0 0) 0x7f10ec050720 con 0x7f10f81559d0 2023-10-06T03:30:21.665 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-10-06T03:30:21.666 INFO:teuthology.orchestra.run.smithi110.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4279,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6833/1434505700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6832","nonce":1434505700},{"type":"v1","addr":"172.21.15.192:6833","nonce":1434505700}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4281,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/1168370789","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6834","nonce":1168370789},{"type":"v1","addr":"172.21.15.110:6835","nonce":1168370789}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4288,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6835/1271810086","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":1271810086},{"type":"v1","addr":"172.21.15.192:6835","nonce":1271810086}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-06T03:29:33.894998+0000","modified":"2023-10-06T03:29:47.307075+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4293},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4293":{"gid":4293,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.110:6837/540320464","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6836","nonce":540320464},{"type":"v1","addr":"172.21.15.110:6837","nonce":540320464}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-06T03:30:21.668 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:21.665+0000 7f10f759e700 1 -- 172.21.15.110:0/1236825424 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f10d8060550 msgr2=0x7f10d8062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:21.668 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:21.665+0000 7f10f759e700 1 --2- 172.21.15.110:0/1236825424 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f10d8060550 0x7f10d8062a00 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f10f8155430 tx=0x7f10e800b040 comp rx=0 tx=0).stop 2023-10-06T03:30:21.668 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:21.665+0000 7f10f759e700 1 -- 172.21.15.110:0/1236825424 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f10f81559d0 msgr2=0x7f10f8150140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:21.668 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:21.665+0000 7f10f759e700 1 --2- 172.21.15.110:0/1236825424 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f10f81559d0 0x7f10f8150140 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f10ec00e8d0 tx=0x7f10ec008040 comp rx=0 tx=0).stop 2023-10-06T03:30:21.668 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:21.665+0000 7f10f759e700 1 -- 172.21.15.110:0/1236825424 shutdown_connections 2023-10-06T03:30:21.668 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:21.665+0000 7f10f759e700 1 --2- 172.21.15.110:0/1236825424 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f10d8060550 0x7f10d8062a00 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:21.668 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:21.665+0000 7f10f759e700 1 --2- 172.21.15.110:0/1236825424 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f10f8159cb0 0x7f10f814e9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:21.668 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:21.665+0000 7f10f759e700 1 --2- 172.21.15.110:0/1236825424 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f10f8155da0 0x7f10f80c47d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:21.668 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:21.665+0000 7f10f759e700 1 --2- 172.21.15.110:0/1236825424 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f10f81559d0 0x7f10f8150140 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:21.669 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:21.665+0000 7f10f759e700 1 -- 172.21.15.110:0/1236825424 >> 172.21.15.110:0/1236825424 conn(0x7f10f80cc470 msgr2=0x7f10f806ab30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:30:21.669 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:21.665+0000 7f10f759e700 1 -- 172.21.15.110:0/1236825424 shutdown_connections 2023-10-06T03:30:21.669 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:21.665+0000 7f10f759e700 1 -- 172.21.15.110:0/1236825424 wait complete. 2023-10-06T03:30:21.669 INFO:teuthology.orchestra.run.smithi110.stderr:dumped fsmap epoch 8 2023-10-06T03:30:21.679 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-06T03:30:21.784 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:21.781+0000 7fb9ff37f700 1 Processor -- start 2023-10-06T03:30:21.784 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:21.781+0000 7fb9ff37f700 1 -- start start 2023-10-06T03:30:21.786 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:21.781+0000 7fb9ff37f700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fb9f81559d0 0x7fb9f806cc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:21.786 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:21.781+0000 7fb9ff37f700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb9f8159e80 0x7fb9f80c5f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:21.786 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:21.781+0000 7fb9ff37f700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9f80caad0 con 0x7fb9f8155da0 2023-10-06T03:30:21.787 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:21.781+0000 7fb9ff37f700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb9f80ae3f0 con 0x7fb9f81559d0 2023-10-06T03:30:21.787 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:21.781+0000 7fb9ff37f700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9f80b1060 con 0x7fb9f8159e80 2023-10-06T03:30:21.787 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:21.781+0000 7fb9fdb7c700 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7fb9f8155da0 0x7fb9f8159770 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:56522/0 (socket says 172.21.15.110:56522) 2023-10-06T03:30:21.787 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:21.781+0000 7fb9fdb7c700 1 -- 172.21.15.110:0/2386542040 learned_addr learned my addr 172.21.15.110:0/2386542040 (peer_addr_for_me v1:172.21.15.110:0/0) 2023-10-06T03:30:21.787 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:21.781+0000 7fb9fe37d700 1 --2- 172.21.15.110:0/2386542040 >> v2:172.21.15.192:3300/0 conn(0x7fb9f81559d0 0x7fb9f806cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:21.787 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:21.781+0000 7fb9feb7e700 1 --2- 172.21.15.110:0/2386542040 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb9f8159e80 0x7fb9f80c5f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:21.787 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:21.781+0000 7fb9fd37b700 1 -- 172.21.15.110:0/2386542040 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 878843238 0 0) 0x7fb9f80caad0 con 0x7fb9f8155da0 2023-10-06T03:30:21.787 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:21.781+0000 7fb9fd37b700 1 -- 172.21.15.110:0/2386542040 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb9e0003570 con 0x7fb9f8155da0 2023-10-06T03:30:21.787 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:21.781+0000 7fb9fe37d700 1 -- 172.21.15.110:0/2386542040 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb9f8159e80 msgr2=0x7fb9f80c5f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:30:21.787 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:21.781+0000 7fb9fe37d700 1 --2- 172.21.15.110:0/2386542040 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb9f8159e80 0x7fb9f80c5f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:21.788 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:21.781+0000 7fb9fe37d700 1 -- 172.21.15.110:0/2386542040 >> v1:172.21.15.110:6789/0 conn(0x7fb9f8155da0 legacy=0x7fb9f8159770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:21.788 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:21.781+0000 7fb9fe37d700 1 -- 172.21.15.110:0/2386542040 --> v2:172.21.15.192:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9f80ca1a0 con 0x7fb9f81559d0 2023-10-06T03:30:21.788 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:21.781+0000 7fb9fe37d700 1 --2- 172.21.15.110:0/2386542040 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb9f81559d0 0x7fb9f806cc90 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fb9ec0098e0 tx=0x7fb9ec0091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=49d079848473401d server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:30:21.789 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:21.781+0000 7fb9fd37b700 1 -- 172.21.15.110:0/2386542040 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9ec011630 con 0x7fb9f81559d0 2023-10-06T03:30:21.789 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:21.781+0000 7fb9fd37b700 1 -- 172.21.15.110:0/2386542040 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9ec011790 con 0x7fb9f81559d0 2023-10-06T03:30:21.789 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:21.781+0000 7fb9fd37b700 1 -- 172.21.15.110:0/2386542040 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9ec011b10 con 0x7fb9f81559d0 2023-10-06T03:30:21.789 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:21.785+0000 7fb9ff37f700 1 -- 172.21.15.110:0/2386542040 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb9f81559d0 msgr2=0x7fb9f806cc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:21.789 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:21.785+0000 7fb9ff37f700 1 --2- 172.21.15.110:0/2386542040 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb9f81559d0 0x7fb9f806cc90 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fb9ec0098e0 tx=0x7fb9ec0091a0 comp rx=0 tx=0).stop 2023-10-06T03:30:21.790 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:21.785+0000 7fb9ff37f700 1 -- 172.21.15.110:0/2386542040 shutdown_connections 2023-10-06T03:30:21.790 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:21.785+0000 7fb9ff37f700 1 --2- 172.21.15.110:0/2386542040 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb9f8159e80 0x7fb9f80c5f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:21.790 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:21.785+0000 7fb9ff37f700 1 --2- 172.21.15.110:0/2386542040 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb9f81559d0 0x7fb9f806cc90 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:21.790 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:21.785+0000 7fb9ff37f700 1 -- 172.21.15.110:0/2386542040 >> 172.21.15.110:0/2386542040 conn(0x7fb9f80cc470 msgr2=0x7fb9f80c8410 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:30:21.790 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:21.785+0000 7fb9ff37f700 1 -- 172.21.15.110:0/2386542040 shutdown_connections 2023-10-06T03:30:21.790 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:21.785+0000 7fb9ff37f700 1 -- 172.21.15.110:0/2386542040 wait complete. 2023-10-06T03:30:21.791 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:21.789+0000 7fb9ff37f700 1 Processor -- start 2023-10-06T03:30:21.791 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:21.789+0000 7fb9ff37f700 1 -- start start 2023-10-06T03:30:21.795 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:21.789+0000 7fb9ff37f700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb9f81559d0 0x7fb9f8153ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:21.795 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:21.789+0000 7fb9ff37f700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb9f8155da0 0x7fb9f8158a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:21.795 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:21.789+0000 7fb9ff37f700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb9f8159e80 0x7fb9f8156c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:21.795 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:21.789+0000 7fb9ff37f700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9f8142930 con 0x7fb9f8159e80 2023-10-06T03:30:21.795 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:21.789+0000 7fb9ff37f700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9f81585a0 con 0x7fb9f8155da0 2023-10-06T03:30:21.795 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:21.789+0000 7fb9ff37f700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9f8157990 con 0x7fb9f81559d0 2023-10-06T03:30:21.796 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:21.789+0000 7fb9feb7e700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb9f8159e80 0x7fb9f8156c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:21.796 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:21.789+0000 7fb9fdb7c700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb9f8155da0 0x7fb9f8158a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:21.796 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:21.789+0000 7fb9fe37d700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb9f81559d0 0x7fb9f8153ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:21.796 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:21.789+0000 7fb9feb7e700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb9f8159e80 0x7fb9f8156c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:40046/0 (socket says 172.21.15.110:40046) 2023-10-06T03:30:21.796 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:21.789+0000 7fb9feb7e700 1 -- 172.21.15.110:0/1628572244 learned_addr learned my addr 172.21.15.110:0/1628572244 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:30:21.796 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:21.789+0000 7fb9feb7e700 1 -- 172.21.15.110:0/1628572244 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb9f81559d0 msgr2=0x7fb9f8153ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:21.796 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:21.789+0000 7fb9feb7e700 1 --2- 172.21.15.110:0/1628572244 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb9f81559d0 0x7fb9f8153ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:21.796 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:21.789+0000 7fb9feb7e700 1 -- 172.21.15.110:0/1628572244 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb9f8155da0 msgr2=0x7fb9f8158a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:21.796 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:21.789+0000 7fb9feb7e700 1 --2- 172.21.15.110:0/1628572244 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb9f8155da0 0x7fb9f8158a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:21.797 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:21.789+0000 7fb9feb7e700 1 -- 172.21.15.110:0/1628572244 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9ec009420 con 0x7fb9f8159e80 2023-10-06T03:30:21.797 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:21.789+0000 7fb9feb7e700 1 --2- 172.21.15.110:0/1628572244 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb9f8159e80 0x7fb9f8156c30 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fb9e800e8d0 tx=0x7fb9e80091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:30:21.797 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:21.789+0000 7fb9f67fc700 1 -- 172.21.15.110:0/1628572244 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9e8013680 con 0x7fb9f8159e80 2023-10-06T03:30:21.797 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:21.789+0000 7fb9ff37f700 1 -- 172.21.15.110:0/1628572244 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9f8157c70 con 0x7fb9f8159e80 2023-10-06T03:30:21.797 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:21.789+0000 7fb9f67fc700 1 -- 172.21.15.110:0/1628572244 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9e80137e0 con 0x7fb9f8159e80 2023-10-06T03:30:21.797 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:21.789+0000 7fb9f67fc700 1 -- 172.21.15.110:0/1628572244 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9e8013ba0 con 0x7fb9f8159e80 2023-10-06T03:30:21.797 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:21.789+0000 7fb9ff37f700 1 -- 172.21.15.110:0/1628572244 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9f8158230 con 0x7fb9f8159e80 2023-10-06T03:30:21.799 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:21.793+0000 7fb9ff37f700 1 -- 172.21.15.110:0/1628572244 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb9f8070ff0 con 0x7fb9f8159e80 2023-10-06T03:30:21.799 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:21.793+0000 7fb9f67fc700 1 -- 172.21.15.110:0/1628572244 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7fb9e80123d0 con 0x7fb9f8159e80 2023-10-06T03:30:21.799 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:21.793+0000 7fb9f67fc700 1 --2- 172.21.15.110:0/1628572244 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fb9e4060580 0x7fb9e4062a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:21.799 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:21.793+0000 7fb9f67fc700 1 -- 172.21.15.110:0/1628572244 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 6867+0+0 (secure 0 0 0) 0x7fb9e807ce10 con 0x7fb9f8159e80 2023-10-06T03:30:21.799 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:21.793+0000 7fb9fe37d700 1 --2- 172.21.15.110:0/1628572244 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fb9e4060580 0x7fb9e4062a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:21.800 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:21.793+0000 7fb9fe37d700 1 --2- 172.21.15.110:0/1628572244 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fb9e4060580 0x7fb9e4062a30 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fb9ec0091e0 tx=0x7fb9ec017000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:30:21.810 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:21.801+0000 7fb9f67fc700 1 -- 172.21.15.110:0/1628572244 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fb9e8080030 con 0x7fb9f8159e80 2023-10-06T03:30:21.961 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:21.957+0000 7fb9ff37f700 1 -- 172.21.15.110:0/1628572244 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb9f80ca480 con 0x7fb9f8159e80 2023-10-06T03:30:21.961 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:21.957+0000 7fb9f67fc700 1 -- 172.21.15.110:0/1628572244 <== mon.0 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v30) v1 ==== 74+0+18850 (secure 0 0 0) 0x7fb9e804ced0 con 0x7fb9f8159e80 2023-10-06T03:30:21.962 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-10-06T03:30:21.963 INFO:teuthology.orchestra.run.smithi110.stdout:{"epoch":30,"fsid":"93b8f6ac-4c92-4b45-b8e4-b59617024aa2","created":"2023-10-06T03:29:10.519583+0000","modified":"2023-10-06T03:30:16.473472+0000","last_up_change":"2023-10-06T03:29:21.960511+0000","last_in_change":"2023-10-06T03:29:12.560436+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-06T03:29:20.703743+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2023-10-06T03:29:28.307281+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2023-10-06T03:29:32.247251+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data","create_time":"2023-10-06T03:29:32.880086+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}}},{"pool":5,"pool_name":"cephfs_data_ec","create_time":"2023-10-06T03:29:36.613779+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"28","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"5866eb6c-2adc-46e4-be7a-e5bcda0f4065","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6824","nonce":33394},{"type":"v1","addr":"172.21.15.110:6825","nonce":33394}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6826","nonce":33394},{"type":"v1","addr":"172.21.15.110:6827","nonce":33394}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6830","nonce":33394},{"type":"v1","addr":"172.21.15.110:6831","nonce":33394}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6828","nonce":33394},{"type":"v1","addr":"172.21.15.110:6829","nonce":33394}]},"public_addr":"172.21.15.110:6825/33394","cluster_addr":"172.21.15.110:6827/33394","heartbeat_back_addr":"172.21.15.110:6831/33394","heartbeat_front_addr":"172.21.15.110:6829/33394","state":["exists","up"]},{"osd":1,"uuid":"a38de118-062b-4a05-9782-2ceddf6ff4df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6800","nonce":33396},{"type":"v1","addr":"172.21.15.110:6801","nonce":33396}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6802","nonce":33396},{"type":"v1","addr":"172.21.15.110:6803","nonce":33396}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6806","nonce":33396},{"type":"v1","addr":"172.21.15.110:6807","nonce":33396}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6804","nonce":33396},{"type":"v1","addr":"172.21.15.110:6805","nonce":33396}]},"public_addr":"172.21.15.110:6801/33396","cluster_addr":"172.21.15.110:6803/33396","heartbeat_back_addr":"172.21.15.110:6807/33396","heartbeat_front_addr":"172.21.15.110:6805/33396","state":["exists","up"]},{"osd":2,"uuid":"56e39be2-3407-4b69-877e-cf16617b06d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6816","nonce":33397},{"type":"v1","addr":"172.21.15.110:6817","nonce":33397}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6818","nonce":33397},{"type":"v1","addr":"172.21.15.110:6819","nonce":33397}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6822","nonce":33397},{"type":"v1","addr":"172.21.15.110:6823","nonce":33397}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6820","nonce":33397},{"type":"v1","addr":"172.21.15.110:6821","nonce":33397}]},"public_addr":"172.21.15.110:6817/33397","cluster_addr":"172.21.15.110:6819/33397","heartbeat_back_addr":"172.21.15.110:6823/33397","heartbeat_front_addr":"172.21.15.110:6821/33397","state":["exists","up"]},{"osd":3,"uuid":"01c82d13-c1d4-4bde-b55e-3e0089b40115","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6808","nonce":33395},{"type":"v1","addr":"172.21.15.110:6809","nonce":33395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6810","nonce":33395},{"type":"v1","addr":"172.21.15.110:6811","nonce":33395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6814","nonce":33395},{"type":"v1","addr":"172.21.15.110:6815","nonce":33395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6812","nonce":33395},{"type":"v1","addr":"172.21.15.110:6813","nonce":33395}]},"public_addr":"172.21.15.110:6809/33395","cluster_addr":"172.21.15.110:6811/33395","heartbeat_back_addr":"172.21.15.110:6815/33395","heartbeat_front_addr":"172.21.15.110:6813/33395","state":["exists","up"]},{"osd":4,"uuid":"86983430-5fa3-4d17-998f-03e2cd843237","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":33683},{"type":"v1","addr":"172.21.15.192:6817","nonce":33683}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":33683},{"type":"v1","addr":"172.21.15.192:6819","nonce":33683}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":33683},{"type":"v1","addr":"172.21.15.192:6823","nonce":33683}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6820","nonce":33683},{"type":"v1","addr":"172.21.15.192:6821","nonce":33683}]},"public_addr":"172.21.15.192:6817/33683","cluster_addr":"172.21.15.192:6819/33683","heartbeat_back_addr":"172.21.15.192:6823/33683","heartbeat_front_addr":"172.21.15.192:6821/33683","state":["exists","up"]},{"osd":5,"uuid":"6945257f-821d-4619-a63f-ebe857eba6f4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":33684},{"type":"v1","addr":"172.21.15.192:6801","nonce":33684}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":33684},{"type":"v1","addr":"172.21.15.192:6803","nonce":33684}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":33684},{"type":"v1","addr":"172.21.15.192:6807","nonce":33684}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6804","nonce":33684},{"type":"v1","addr":"172.21.15.192:6805","nonce":33684}]},"public_addr":"172.21.15.192:6801/33684","cluster_addr":"172.21.15.192:6803/33684","heartbeat_back_addr":"172.21.15.192:6807/33684","heartbeat_front_addr":"172.21.15.192:6805/33684","state":["exists","up"]},{"osd":6,"uuid":"7854d1ae-7060-4780-8f97-73cea5e7a12f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6824","nonce":33686},{"type":"v1","addr":"172.21.15.192:6825","nonce":33686}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":33686},{"type":"v1","addr":"172.21.15.192:6827","nonce":33686}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":33686},{"type":"v1","addr":"172.21.15.192:6831","nonce":33686}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6828","nonce":33686},{"type":"v1","addr":"172.21.15.192:6829","nonce":33686}]},"public_addr":"172.21.15.192:6825/33686","cluster_addr":"172.21.15.192:6827/33686","heartbeat_back_addr":"172.21.15.192:6831/33686","heartbeat_front_addr":"172.21.15.192:6829/33686","state":["exists","up"]},{"osd":7,"uuid":"f1ebdfd5-b859-482b-9f97-c18c4a26f445","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6808","nonce":33685},{"type":"v1","addr":"172.21.15.192:6809","nonce":33685}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":33685},{"type":"v1","addr":"172.21.15.192:6811","nonce":33685}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":33685},{"type":"v1","addr":"172.21.15.192:6815","nonce":33685}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6812","nonce":33685},{"type":"v1","addr":"172.21.15.192:6813","nonce":33685}]},"public_addr":"172.21.15.192:6809/33685","cluster_addr":"172.21.15.192:6811/33685","heartbeat_back_addr":"172.21.15.192:6815/33685","heartbeat_front_addr":"172.21.15.192:6813/33685","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T03:29:18.002048+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T03:29:17.478576+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T03:29:18.034944+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T03:29:17.628513+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T03:29:18.047338+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T03:29:17.772349+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T03:29:17.883958+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T03:29:18.151838+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"4.7","mappings":[{"from":1,"to":2}]},{"pgid":"4.3b","mappings":[{"from":3,"to":2}]}],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-06T03:30:21.965 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:21.961+0000 7fb9ff37f700 1 -- 172.21.15.110:0/1628572244 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fb9e4060580 msgr2=0x7fb9e4062a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:21.965 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:21.961+0000 7fb9ff37f700 1 --2- 172.21.15.110:0/1628572244 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fb9e4060580 0x7fb9e4062a30 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fb9ec0091e0 tx=0x7fb9ec017000 comp rx=0 tx=0).stop 2023-10-06T03:30:21.965 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:21.961+0000 7fb9ff37f700 1 -- 172.21.15.110:0/1628572244 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb9f8159e80 msgr2=0x7fb9f8156c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:21.965 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:21.961+0000 7fb9ff37f700 1 --2- 172.21.15.110:0/1628572244 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb9f8159e80 0x7fb9f8156c30 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fb9e800e8d0 tx=0x7fb9e80091a0 comp rx=0 tx=0).stop 2023-10-06T03:30:21.965 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:21.961+0000 7fb9ff37f700 1 -- 172.21.15.110:0/1628572244 shutdown_connections 2023-10-06T03:30:21.965 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:21.961+0000 7fb9ff37f700 1 --2- 172.21.15.110:0/1628572244 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fb9e4060580 0x7fb9e4062a30 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:21.965 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:21.961+0000 7fb9ff37f700 1 --2- 172.21.15.110:0/1628572244 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb9f8159e80 0x7fb9f8156c30 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:21.966 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:21.961+0000 7fb9ff37f700 1 --2- 172.21.15.110:0/1628572244 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb9f8155da0 0x7fb9f8158a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:21.966 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:21.961+0000 7fb9ff37f700 1 --2- 172.21.15.110:0/1628572244 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb9f81559d0 0x7fb9f8153ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:21.966 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:21.961+0000 7fb9ff37f700 1 -- 172.21.15.110:0/1628572244 >> 172.21.15.110:0/1628572244 conn(0x7fb9f80cc470 msgr2=0x7fb9f80c70b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:30:21.966 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:21.961+0000 7fb9ff37f700 1 -- 172.21.15.110:0/1628572244 shutdown_connections 2023-10-06T03:30:21.966 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:21.961+0000 7fb9ff37f700 1 -- 172.21.15.110:0/1628572244 wait complete. 2023-10-06T03:30:21.977 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2023-10-06T03:30:22.091 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:22.085+0000 7f2591850700 1 Processor -- start 2023-10-06T03:30:22.091 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:22.085+0000 7f2591850700 1 -- start start 2023-10-06T03:30:22.092 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:22.089+0000 7f2591850700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f258c156d60 0x7f258c157130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:22.092 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:22.089+0000 7f2591850700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f258c069cf0 0x7f258c0c6ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:22.092 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:22.089+0000 7f2591850700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f258c158090 con 0x7f258c0c6a90 2023-10-06T03:30:22.092 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:22.089+0000 7f2591850700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f258c15bc80 con 0x7f258c156d60 2023-10-06T03:30:22.092 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:22.089+0000 7f2591850700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f258c1577a0 con 0x7f258c069cf0 2023-10-06T03:30:22.093 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:22.089+0000 7f258b7fe700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f258c069cf0 0x7f258c0c6ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:22.093 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:22.089+0000 7f258b7fe700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f258c069cf0 0x7f258c0c6ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3301/0 says I am v2:172.21.15.110:39768/0 (socket says 172.21.15.110:39768) 2023-10-06T03:30:22.093 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:22.089+0000 7f258affd700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f258c156d60 0x7f258c157130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:22.093 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:22.089+0000 7f258b7fe700 1 -- 172.21.15.110:0/2993373067 learned_addr learned my addr 172.21.15.110:0/2993373067 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:30:22.094 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:22.089+0000 7f258a7fc700 1 -- 172.21.15.110:0/2993373067 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 180229497 0 0) 0x7f258c158090 con 0x7f258c0c6a90 2023-10-06T03:30:22.094 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:22.089+0000 7f258a7fc700 1 -- 172.21.15.110:0/2993373067 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2578003570 con 0x7f258c0c6a90 2023-10-06T03:30:22.094 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:22.089+0000 7f258b7fe700 1 -- 172.21.15.110:0/2993373067 >> v2:172.21.15.192:3300/0 conn(0x7f258c156d60 msgr2=0x7f258c157130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:30:22.094 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:22.089+0000 7f258b7fe700 1 --2- 172.21.15.110:0/2993373067 >> v2:172.21.15.192:3300/0 conn(0x7f258c156d60 0x7f258c157130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:22.094 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:22.089+0000 7f258b7fe700 1 -- 172.21.15.110:0/2993373067 >> v1:172.21.15.192:6789/0 conn(0x7f258c0c6a90 legacy=0x7f258c0695e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:22.094 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:22.089+0000 7f258b7fe700 1 -- 172.21.15.110:0/2993373067 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f258c06cc80 con 0x7f258c069cf0 2023-10-06T03:30:22.094 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:22.089+0000 7f258b7fe700 1 --2- 172.21.15.110:0/2993373067 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f258c069cf0 0x7f258c0c6ee0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f2584009f20 tx=0x7f2584009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dd2cf2e5ef4d5bd8 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:30:22.094 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:22.089+0000 7f258a7fc700 1 -- 172.21.15.110:0/2993373067 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f258400e070 con 0x7f258c069cf0 2023-10-06T03:30:22.095 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:22.089+0000 7f258a7fc700 1 -- 172.21.15.110:0/2993373067 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f258400b970 con 0x7f258c069cf0 2023-10-06T03:30:22.095 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:22.089+0000 7f258a7fc700 1 -- 172.21.15.110:0/2993373067 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25840092e0 con 0x7f258c069cf0 2023-10-06T03:30:22.095 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:22.089+0000 7f2591850700 1 -- 172.21.15.110:0/2993373067 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f258c069cf0 msgr2=0x7f258c0c6ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:22.096 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:22.089+0000 7f2591850700 1 --2- 172.21.15.110:0/2993373067 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f258c069cf0 0x7f258c0c6ee0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f2584009f20 tx=0x7f2584009840 comp rx=0 tx=0).stop 2023-10-06T03:30:22.096 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:22.089+0000 7f2591850700 1 -- 172.21.15.110:0/2993373067 shutdown_connections 2023-10-06T03:30:22.096 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:22.089+0000 7f2591850700 1 --2- 172.21.15.110:0/2993373067 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f258c069cf0 0x7f258c0c6ee0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:22.096 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:22.089+0000 7f2591850700 1 --2- 172.21.15.110:0/2993373067 >> v2:172.21.15.192:3300/0 conn(0x7f258c156d60 0x7f258c157130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:22.096 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:22.089+0000 7f2591850700 1 -- 172.21.15.110:0/2993373067 >> 172.21.15.110:0/2993373067 conn(0x7f258c0cc710 msgr2=0x7f258c0c8670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:30:22.096 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:22.089+0000 7f2591850700 1 -- 172.21.15.110:0/2993373067 shutdown_connections 2023-10-06T03:30:22.096 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:22.089+0000 7f2591850700 1 -- 172.21.15.110:0/2993373067 wait complete. 2023-10-06T03:30:22.097 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:22.093+0000 7f2591850700 1 Processor -- start 2023-10-06T03:30:22.097 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:22.093+0000 7f2591850700 1 -- start start 2023-10-06T03:30:22.100 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:22.093+0000 7f2591850700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f258c069cf0 0x7f258c151540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:22.100 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:22.097+0000 7f2591850700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f258c0c6a90 0x7f258c0c2c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:22.100 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:22.097+0000 7f2591850700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f258c156d60 0x7f258c0acca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:22.100 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:22.097+0000 7f2591850700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f258c0afdf0 con 0x7f258c156d60 2023-10-06T03:30:22.100 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:22.097+0000 7f2591850700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f258c0aff80 con 0x7f258c0c6a90 2023-10-06T03:30:22.101 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:22.097+0000 7f2591850700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f258c0c3580 con 0x7f258c069cf0 2023-10-06T03:30:22.101 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:22.097+0000 7f258b7fe700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f258c156d60 0x7f258c0acca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:22.101 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:22.097+0000 7f258b7fe700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f258c156d60 0x7f258c0acca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:40052/0 (socket says 172.21.15.110:40052) 2023-10-06T03:30:22.101 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:22.097+0000 7f258b7fe700 1 -- 172.21.15.110:0/2690751268 learned_addr learned my addr 172.21.15.110:0/2690751268 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:30:22.101 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:22.097+0000 7f258affd700 1 --2- 172.21.15.110:0/2690751268 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f258c069cf0 0x7f258c151540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:22.101 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:22.097+0000 7f25827fc700 1 --2- 172.21.15.110:0/2690751268 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f258c0c6a90 0x7f258c0c2c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:22.101 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:22.097+0000 7f258b7fe700 1 -- 172.21.15.110:0/2690751268 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f258c069cf0 msgr2=0x7f258c151540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:22.101 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:22.097+0000 7f258b7fe700 1 --2- 172.21.15.110:0/2690751268 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f258c069cf0 0x7f258c151540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:22.101 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:22.097+0000 7f258b7fe700 1 -- 172.21.15.110:0/2690751268 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f258c0c6a90 msgr2=0x7f258c0c2c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:22.101 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:22.097+0000 7f258b7fe700 1 --2- 172.21.15.110:0/2690751268 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f258c0c6a90 0x7f258c0c2c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:22.102 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:22.097+0000 7f258b7fe700 1 -- 172.21.15.110:0/2690751268 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2584009ae0 con 0x7f258c156d60 2023-10-06T03:30:22.102 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:22.097+0000 7f258b7fe700 1 --2- 172.21.15.110:0/2690751268 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f258c156d60 0x7f258c0acca0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f25840072f0 tx=0x7f2584018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:30:22.103 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:22.097+0000 7f25837fe700 1 -- 172.21.15.110:0/2690751268 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2584003300 con 0x7f258c156d60 2023-10-06T03:30:22.103 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:22.097+0000 7f25837fe700 1 -- 172.21.15.110:0/2690751268 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2584007b00 con 0x7f258c156d60 2023-10-06T03:30:22.103 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:22.097+0000 7f2591850700 1 -- 172.21.15.110:0/2690751268 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f258c0c3860 con 0x7f258c156d60 2023-10-06T03:30:22.103 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:22.097+0000 7f25837fe700 1 -- 172.21.15.110:0/2690751268 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f258401d420 con 0x7f258c156d60 2023-10-06T03:30:22.103 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:22.097+0000 7f2591850700 1 -- 172.21.15.110:0/2690751268 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f258c151a80 con 0x7f258c156d60 2023-10-06T03:30:22.103 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:22.097+0000 7f2591850700 1 -- 172.21.15.110:0/2690751268 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f258c0b0110 con 0x7f258c156d60 2023-10-06T03:30:22.104 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:22.101+0000 7f25837fe700 1 -- 172.21.15.110:0/2690751268 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7f2584003050 con 0x7f258c156d60 2023-10-06T03:30:22.104 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:22.101+0000 7f25837fe700 1 --2- 172.21.15.110:0/2690751268 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f256c060550 0x7f256c062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:22.104 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:22.101+0000 7f25837fe700 1 -- 172.21.15.110:0/2690751268 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 6867+0+0 (secure 0 0 0) 0x7f258407e840 con 0x7f258c156d60 2023-10-06T03:30:22.104 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:22.101+0000 7f258affd700 1 --2- 172.21.15.110:0/2690751268 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f256c060550 0x7f256c062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:22.105 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:22.101+0000 7f258affd700 1 --2- 172.21.15.110:0/2690751268 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f256c060550 0x7f256c062a00 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f257c009d10 tx=0x7f257c0091a0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:30:22.113 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:22.109+0000 7f25837fe700 1 -- 172.21.15.110:0/2690751268 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f258404e9a0 con 0x7f258c156d60 2023-10-06T03:30:22.310 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:22.305+0000 7f2591850700 1 -- 172.21.15.110:0/2690751268 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f258c06ff60 con 0x7f258c156d60 2023-10-06T03:30:22.311 INFO:tasks.ceph.mon.a.smithi110.stderr:2023-10-06T03:30:22.309+0000 7f5c90378700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-10-06T03:30:22.318 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:22.313+0000 7f25837fe700 1 -- 172.21.15.110:0/2690751268 <== mon.0 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v9) v1 ==== 161+0+0 (secure 0 0 0) 0x7f258402a090 con 0x7f258c156d60 2023-10-06T03:30:22.322 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:22.317+0000 7f2591850700 1 -- 172.21.15.110:0/2690751268 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f256c060550 msgr2=0x7f256c062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:22.322 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:22.317+0000 7f2591850700 1 --2- 172.21.15.110:0/2690751268 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f256c060550 0x7f256c062a00 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f257c009d10 tx=0x7f257c0091a0 comp rx=0 tx=0).stop 2023-10-06T03:30:22.322 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:22.317+0000 7f2591850700 1 -- 172.21.15.110:0/2690751268 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f258c156d60 msgr2=0x7f258c0acca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:22.322 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:22.317+0000 7f2591850700 1 --2- 172.21.15.110:0/2690751268 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f258c156d60 0x7f258c0acca0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f25840072f0 tx=0x7f2584018000 comp rx=0 tx=0).stop 2023-10-06T03:30:22.323 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:22.317+0000 7f2591850700 1 -- 172.21.15.110:0/2690751268 shutdown_connections 2023-10-06T03:30:22.323 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:22.317+0000 7f2591850700 1 --2- 172.21.15.110:0/2690751268 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f256c060550 0x7f256c062a00 secure :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f257c009d10 tx=0x7f257c0091a0 comp rx=0 tx=0).stop 2023-10-06T03:30:22.323 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:22.317+0000 7f2591850700 1 --2- 172.21.15.110:0/2690751268 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f258c156d60 0x7f258c0acca0 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:22.323 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:22.317+0000 7f2591850700 1 --2- 172.21.15.110:0/2690751268 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f258c0c6a90 0x7f258c0c2c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:22.323 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:22.317+0000 7f2591850700 1 --2- 172.21.15.110:0/2690751268 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f258c069cf0 0x7f258c151540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:22.323 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:22.317+0000 7f2591850700 1 -- 172.21.15.110:0/2690751268 >> 172.21.15.110:0/2690751268 conn(0x7f258c0cc710 msgr2=0x7f258c0c1310 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:30:22.323 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:22.317+0000 7f2591850700 1 -- 172.21.15.110:0/2690751268 shutdown_connections 2023-10-06T03:30:22.323 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:22.317+0000 7f2591850700 1 -- 172.21.15.110:0/2690751268 wait complete. 2023-10-06T03:30:22.324 INFO:teuthology.orchestra.run.smithi110.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2023-10-06T03:30:22.324 INFO:tasks.ceph.mds.a.smithi110.stderr: -3325> 2023-10-06T03:29:35.498+0000 7f8399c1b700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-06T03:30:22.334 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2023-10-06T03:30:22.350 INFO:tasks.ceph.mds.a.smithi110.stdout:starting mds.a at 2023-10-06T03:30:22.438 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:22.433+0000 7fb6f6fbe700 1 Processor -- start 2023-10-06T03:30:22.439 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:22.433+0000 7fb6f6fbe700 1 -- start start 2023-10-06T03:30:22.440 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:22.437+0000 7fb6f6fbe700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fb6f0150800 0x7fb6f00c6bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:22.440 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:22.437+0000 7fb6f6fbe700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6f0157700 con 0x7fb6f0150430 2023-10-06T03:30:22.440 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:22.437+0000 7fb6f6fbe700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6f0156df0 con 0x7fb6f00cb5b0 2023-10-06T03:30:22.440 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:22.437+0000 7fb6f6fbe700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb6f01570d0 con 0x7fb6f0150800 2023-10-06T03:30:22.441 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:22.437+0000 7fb6f4d5a700 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7fb6f0150430 0x7fb6f01565a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:56546/0 (socket says 172.21.15.110:56546) 2023-10-06T03:30:22.441 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:22.437+0000 7fb6f4d5a700 1 -- 172.21.15.110:0/2282083311 learned_addr learned my addr 172.21.15.110:0/2282083311 (peer_addr_for_me v1:172.21.15.110:0/0) 2023-10-06T03:30:22.441 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:22.437+0000 7fb6effff700 1 --2- 172.21.15.110:0/2282083311 >> v2:172.21.15.110:3300/0 conn(0x7fb6f0150800 0x7fb6f00c6bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:22.442 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:22.437+0000 7fb6ef7fe700 1 -- 172.21.15.110:0/2282083311 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2973357052 0 0) 0x7fb6f0157700 con 0x7fb6f0150430 2023-10-06T03:30:22.443 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:22.437+0000 7fb6ef7fe700 1 -- 172.21.15.110:0/2282083311 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb6dc003570 con 0x7fb6f0150430 2023-10-06T03:30:22.443 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:22.437+0000 7fb6ef7fe700 1 -- 172.21.15.110:0/2282083311 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2133571681 0 0) 0x7fb6f0156df0 con 0x7fb6f00cb5b0 2023-10-06T03:30:22.443 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:22.437+0000 7fb6ef7fe700 1 -- 172.21.15.110:0/2282083311 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb6f0157700 con 0x7fb6f00cb5b0 2023-10-06T03:30:22.443 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:22.437+0000 7fb6effff700 1 -- 172.21.15.110:0/2282083311 >> v1:172.21.15.192:6789/0 conn(0x7fb6f00cb5b0 legacy=0x7fb6f00cb9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:22.443 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:22.437+0000 7fb6effff700 1 -- 172.21.15.110:0/2282083311 >> v1:172.21.15.110:6789/0 conn(0x7fb6f0150430 legacy=0x7fb6f01565a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:22.443 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:22.437+0000 7fb6effff700 1 -- 172.21.15.110:0/2282083311 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6f01573b0 con 0x7fb6f0150800 2023-10-06T03:30:22.443 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:22.437+0000 7fb6effff700 1 --2- 172.21.15.110:0/2282083311 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb6f0150800 0x7fb6f00c6bb0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fb6d80098e0 tx=0x7fb6d80091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ad79f989900d102 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:30:22.443 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:22.437+0000 7fb6ef7fe700 1 -- 172.21.15.110:0/2282083311 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6d8011630 con 0x7fb6f0150800 2023-10-06T03:30:22.443 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:22.437+0000 7fb6ef7fe700 1 -- 172.21.15.110:0/2282083311 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb6d8011790 con 0x7fb6f0150800 2023-10-06T03:30:22.444 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:22.437+0000 7fb6ef7fe700 1 -- 172.21.15.110:0/2282083311 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6d8011b10 con 0x7fb6f0150800 2023-10-06T03:30:22.444 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:22.437+0000 7fb6f6fbe700 1 -- 172.21.15.110:0/2282083311 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb6f0150800 msgr2=0x7fb6f00c6bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:22.444 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:22.437+0000 7fb6f6fbe700 1 --2- 172.21.15.110:0/2282083311 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb6f0150800 0x7fb6f00c6bb0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fb6d80098e0 tx=0x7fb6d80091a0 comp rx=0 tx=0).stop 2023-10-06T03:30:22.444 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:22.437+0000 7fb6f6fbe700 1 -- 172.21.15.110:0/2282083311 shutdown_connections 2023-10-06T03:30:22.444 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:22.437+0000 7fb6f6fbe700 1 --2- 172.21.15.110:0/2282083311 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb6f0150800 0x7fb6f00c6bb0 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:22.444 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:22.437+0000 7fb6f6fbe700 1 -- 172.21.15.110:0/2282083311 >> 172.21.15.110:0/2282083311 conn(0x7fb6f00c8f00 msgr2=0x7fb6f00c9300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:30:22.444 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:22.437+0000 7fb6f6fbe700 1 -- 172.21.15.110:0/2282083311 shutdown_connections 2023-10-06T03:30:22.444 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:22.437+0000 7fb6f6fbe700 1 -- 172.21.15.110:0/2282083311 wait complete. 2023-10-06T03:30:22.446 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:22.441+0000 7fb6f6fbe700 1 Processor -- start 2023-10-06T03:30:22.446 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:22.441+0000 7fb6f6fbe700 1 -- start start 2023-10-06T03:30:22.447 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:22.441+0000 7fb6f6fbe700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb6f00cb5b0 0x7fb6f013be30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:22.447 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:22.441+0000 7fb6f6fbe700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb6f0150430 0x7fb6f00b14d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:22.447 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:22.441+0000 7fb6f6fbe700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb6f0169360 0x7fb6f01697c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:22.447 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:22.441+0000 7fb6f6fbe700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6f0169d00 con 0x7fb6f0150430 2023-10-06T03:30:22.447 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:22.441+0000 7fb6f6fbe700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6f00b0a80 con 0x7fb6f00cb5b0 2023-10-06T03:30:22.447 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:22.441+0000 7fb6f6fbe700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6f00b1040 con 0x7fb6f0169360 2023-10-06T03:30:22.447 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:22.441+0000 7fb6effff700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb6f0150430 0x7fb6f00b14d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:22.447 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:22.441+0000 7fb6effff700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb6f0150430 0x7fb6f00b14d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:40066/0 (socket says 172.21.15.110:40066) 2023-10-06T03:30:22.447 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:22.441+0000 7fb6effff700 1 -- 172.21.15.110:0/894460323 learned_addr learned my addr 172.21.15.110:0/894460323 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:30:22.447 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:22.441+0000 7fb6f4d5a700 1 --2- 172.21.15.110:0/894460323 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb6f00cb5b0 0x7fb6f013be30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:22.448 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:22.441+0000 7fb6f555b700 1 --2- 172.21.15.110:0/894460323 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb6f0169360 0x7fb6f01697c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:22.448 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:22.441+0000 7fb6effff700 1 -- 172.21.15.110:0/894460323 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb6f0169360 msgr2=0x7fb6f01697c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:22.448 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:22.441+0000 7fb6effff700 1 --2- 172.21.15.110:0/894460323 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb6f0169360 0x7fb6f01697c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:22.448 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:22.441+0000 7fb6effff700 1 -- 172.21.15.110:0/894460323 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb6f00cb5b0 msgr2=0x7fb6f013be30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:22.448 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:22.441+0000 7fb6effff700 1 --2- 172.21.15.110:0/894460323 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb6f00cb5b0 0x7fb6f013be30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:22.448 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:22.441+0000 7fb6effff700 1 -- 172.21.15.110:0/894460323 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6d8009420 con 0x7fb6f0150430 2023-10-06T03:30:22.448 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:22.441+0000 7fb6effff700 1 --2- 172.21.15.110:0/894460323 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb6f0150430 0x7fb6f00b14d0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fb6d8011630 tx=0x7fb6d8017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:30:22.448 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:22.441+0000 7fb6ecff9700 1 -- 172.21.15.110:0/894460323 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6d8011900 con 0x7fb6f0150430 2023-10-06T03:30:22.448 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:22.441+0000 7fb6ecff9700 1 -- 172.21.15.110:0/894460323 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb6d8011a60 con 0x7fb6f0150430 2023-10-06T03:30:22.448 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:22.441+0000 7fb6f6fbe700 1 -- 172.21.15.110:0/894460323 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6f0169ef0 con 0x7fb6f0150430 2023-10-06T03:30:22.448 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:22.441+0000 7fb6ecff9700 1 -- 172.21.15.110:0/894460323 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6d801a660 con 0x7fb6f0150430 2023-10-06T03:30:22.449 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:22.441+0000 7fb6f6fbe700 1 -- 172.21.15.110:0/894460323 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6f00b0cd0 con 0x7fb6f0150430 2023-10-06T03:30:22.450 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:22.445+0000 7fb6f6fbe700 1 -- 172.21.15.110:0/894460323 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb6f00b11d0 con 0x7fb6f0150430 2023-10-06T03:30:22.450 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:22.445+0000 7fb6ecff9700 1 -- 172.21.15.110:0/894460323 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7fb6d801aa10 con 0x7fb6f0150430 2023-10-06T03:30:22.450 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:22.445+0000 7fb6ecff9700 1 --2- 172.21.15.110:0/894460323 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fb6d0060550 0x7fb6d0062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:22.450 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:22.445+0000 7fb6ecff9700 1 -- 172.21.15.110:0/894460323 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6953+0+0 (secure 0 0 0) 0x7fb6d807e7e0 con 0x7fb6f0150430 2023-10-06T03:30:22.450 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:22.445+0000 7fb6f4d5a700 1 --2- 172.21.15.110:0/894460323 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fb6d0060550 0x7fb6d0062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:22.450 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:22.445+0000 7fb6f4d5a700 1 --2- 172.21.15.110:0/894460323 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fb6d0060550 0x7fb6d0062a00 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fb6f0067d30 tx=0x7fb6e8009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:30:22.457 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:22.453+0000 7fb6ecff9700 1 -- 172.21.15.110:0/894460323 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fb6d804e8f0 con 0x7fb6f0150430 2023-10-06T03:30:22.655 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:22.653+0000 7fb6f6fbe700 1 -- 172.21.15.110:0/894460323 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fb6f006ff60 con 0x7fb6f0150430 2023-10-06T03:30:23.323 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:23.317+0000 7fb6ecff9700 1 -- 172.21.15.110:0/894460323 <== mon.0 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v10) v1 ==== 104+0+0 (secure 0 0 0) 0x7fb6d8051f10 con 0x7fb6f0150430 2023-10-06T03:30:23.328 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:23.325+0000 7fb6f6fbe700 1 -- 172.21.15.110:0/894460323 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fb6d0060550 msgr2=0x7fb6d0062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:23.328 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:23.325+0000 7fb6f6fbe700 1 --2- 172.21.15.110:0/894460323 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fb6d0060550 0x7fb6d0062a00 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fb6f0067d30 tx=0x7fb6e8009040 comp rx=0 tx=0).stop 2023-10-06T03:30:23.328 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:23.325+0000 7fb6f6fbe700 1 -- 172.21.15.110:0/894460323 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb6f0150430 msgr2=0x7fb6f00b14d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:23.328 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:23.325+0000 7fb6f6fbe700 1 --2- 172.21.15.110:0/894460323 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb6f0150430 0x7fb6f00b14d0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fb6d8011630 tx=0x7fb6d8017000 comp rx=0 tx=0).stop 2023-10-06T03:30:23.328 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:23.325+0000 7fb6f6fbe700 1 -- 172.21.15.110:0/894460323 shutdown_connections 2023-10-06T03:30:23.329 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:23.325+0000 7fb6f6fbe700 1 --2- 172.21.15.110:0/894460323 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fb6d0060550 0x7fb6d0062a00 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:23.329 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:23.325+0000 7fb6f6fbe700 1 --2- 172.21.15.110:0/894460323 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fb6f0169360 0x7fb6f01697c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:23.329 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:23.325+0000 7fb6f6fbe700 1 --2- 172.21.15.110:0/894460323 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb6f0150430 0x7fb6f00b14d0 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:23.329 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:23.325+0000 7fb6f6fbe700 1 --2- 172.21.15.110:0/894460323 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb6f00cb5b0 0x7fb6f013be30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:23.329 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:23.325+0000 7fb6f6fbe700 1 -- 172.21.15.110:0/894460323 >> 172.21.15.110:0/894460323 conn(0x7fb6f00c8f00 msgr2=0x7fb6f014d130 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:30:23.329 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:23.325+0000 7fb6f6fbe700 1 -- 172.21.15.110:0/894460323 shutdown_connections 2023-10-06T03:30:23.329 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:23.325+0000 7fb6f6fbe700 1 -- 172.21.15.110:0/894460323 wait complete. 2023-10-06T03:30:23.343 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2023-10-06T03:30:23.462 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:23.457+0000 7f0257537700 1 Processor -- start 2023-10-06T03:30:23.462 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:23.457+0000 7f0257537700 1 -- start start 2023-10-06T03:30:23.464 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:23.461+0000 7f0257537700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f0250158d10 0x7f02501590e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:23.464 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:23.461+0000 7f0257537700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f025006bbd0 con 0x7f02500ce9b0 2023-10-06T03:30:23.464 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:23.461+0000 7f0257537700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02500ceec0 con 0x7f02500cb060 2023-10-06T03:30:23.464 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:23.461+0000 7f0257537700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f02500cf1a0 con 0x7f0250158d10 2023-10-06T03:30:23.464 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:23.461+0000 7f0254ad2700 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f02500ce9b0 0x7f02500ca950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:56558/0 (socket says 172.21.15.110:56558) 2023-10-06T03:30:23.464 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:23.461+0000 7f0254ad2700 1 -- 172.21.15.110:0/2628603858 learned_addr learned my addr 172.21.15.110:0/2628603858 (peer_addr_for_me v1:172.21.15.110:0/0) 2023-10-06T03:30:23.464 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:23.461+0000 7f02552d3700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f0250158d10 0x7f02501590e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:23.466 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:23.461+0000 7f0247fff700 1 -- 172.21.15.110:0/2628603858 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2557968581 0 0) 0x7f025006bbd0 con 0x7f02500ce9b0 2023-10-06T03:30:23.466 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:23.461+0000 7f0247fff700 1 -- 172.21.15.110:0/2628603858 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f023c003570 con 0x7f02500ce9b0 2023-10-06T03:30:23.466 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:23.461+0000 7f02552d3700 1 -- 172.21.15.110:0/2628603858 >> v1:172.21.15.192:6789/0 conn(0x7f02500cb060 legacy=0x7f025006c060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:23.466 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:23.461+0000 7f0247fff700 1 -- 172.21.15.110:0/2628603858 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 743650229 0 0) 0x7f02500ceec0 con 0x7f02500cb060 2023-10-06T03:30:23.466 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:23.461+0000 7f02552d3700 1 -- 172.21.15.110:0/2628603858 >> v1:172.21.15.110:6789/0 conn(0x7f02500ce9b0 legacy=0x7f02500ca950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:23.466 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:23.461+0000 7f02552d3700 1 -- 172.21.15.110:0/2628603858 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02500cf480 con 0x7f0250158d10 2023-10-06T03:30:23.467 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:23.461+0000 7f02552d3700 1 --2- 172.21.15.110:0/2628603858 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0250158d10 0x7f02501590e0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f02400098e0 tx=0x7f02400091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f387ff567cfcde71 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:30:23.467 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:23.461+0000 7f0247fff700 1 -- 172.21.15.110:0/2628603858 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f024c0013f0 con 0x7f0250158d10 2023-10-06T03:30:23.467 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:23.461+0000 7f0247fff700 1 -- 172.21.15.110:0/2628603858 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0240011420 con 0x7f0250158d10 2023-10-06T03:30:23.467 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:23.461+0000 7f0247fff700 1 -- 172.21.15.110:0/2628603858 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02400117c0 con 0x7f0250158d10 2023-10-06T03:30:23.467 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:23.461+0000 7f0257537700 1 -- 172.21.15.110:0/2628603858 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0250158d10 msgr2=0x7f02501590e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:23.467 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:23.461+0000 7f0257537700 1 --2- 172.21.15.110:0/2628603858 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0250158d10 0x7f02501590e0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f02400098e0 tx=0x7f02400091a0 comp rx=0 tx=0).stop 2023-10-06T03:30:23.467 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:23.461+0000 7f0257537700 1 -- 172.21.15.110:0/2628603858 shutdown_connections 2023-10-06T03:30:23.467 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:23.461+0000 7f0257537700 1 --2- 172.21.15.110:0/2628603858 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0250158d10 0x7f02501590e0 secure :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f02400098e0 tx=0x7f02400091a0 comp rx=0 tx=0).stop 2023-10-06T03:30:23.467 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:23.461+0000 7f0257537700 1 -- 172.21.15.110:0/2628603858 >> 172.21.15.110:0/2628603858 conn(0x7f025006e860 msgr2=0x7f02500d0070 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:30:23.467 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:23.461+0000 7f0257537700 1 -- 172.21.15.110:0/2628603858 shutdown_connections 2023-10-06T03:30:23.468 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:23.461+0000 7f0257537700 1 -- 172.21.15.110:0/2628603858 wait complete. 2023-10-06T03:30:23.468 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:23.465+0000 7f0257537700 1 Processor -- start 2023-10-06T03:30:23.468 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:23.465+0000 7f0257537700 1 -- start start 2023-10-06T03:30:23.470 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:23.465+0000 7f0257537700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f02500cb060 0x7f02500b5590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:23.471 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:23.465+0000 7f0257537700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f02500ce9b0 0x7f0250154930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:23.471 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:23.465+0000 7f0257537700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f02500aec70 0x7f0250156710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:23.471 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:23.465+0000 7f0257537700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02501554a0 con 0x7f02500cb060 2023-10-06T03:30:23.471 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:23.465+0000 7f0257537700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02500af170 con 0x7f02500aec70 2023-10-06T03:30:23.471 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:23.465+0000 7f0257537700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02500af450 con 0x7f02500ce9b0 2023-10-06T03:30:23.471 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:23.465+0000 7f02552d3700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f02500cb060 0x7f02500b5590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:23.471 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:23.465+0000 7f02552d3700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f02500cb060 0x7f02500b5590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:40086/0 (socket says 172.21.15.110:40086) 2023-10-06T03:30:23.471 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:23.465+0000 7f02552d3700 1 -- 172.21.15.110:0/3298506381 learned_addr learned my addr 172.21.15.110:0/3298506381 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:30:23.471 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:23.465+0000 7f0255ad4700 1 --2- 172.21.15.110:0/3298506381 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f02500aec70 0x7f0250156710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:23.471 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:23.465+0000 7f0254ad2700 1 --2- 172.21.15.110:0/3298506381 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f02500ce9b0 0x7f0250154930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:23.471 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:23.465+0000 7f02552d3700 1 -- 172.21.15.110:0/3298506381 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f02500ce9b0 msgr2=0x7f0250154930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:23.472 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:23.465+0000 7f02552d3700 1 --2- 172.21.15.110:0/3298506381 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f02500ce9b0 0x7f0250154930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:23.472 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:23.465+0000 7f02552d3700 1 -- 172.21.15.110:0/3298506381 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f02500aec70 msgr2=0x7f0250156710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:23.472 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:23.465+0000 7f02552d3700 1 --2- 172.21.15.110:0/3298506381 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f02500aec70 0x7f0250156710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:23.472 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:23.465+0000 7f02552d3700 1 -- 172.21.15.110:0/3298506381 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0240009420 con 0x7f02500cb060 2023-10-06T03:30:23.472 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:23.465+0000 7f0255ad4700 1 --2- 172.21.15.110:0/3298506381 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f02500aec70 0x7f0250156710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-06T03:30:23.473 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:23.465+0000 7f02552d3700 1 --2- 172.21.15.110:0/3298506381 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f02500cb060 0x7f02500b5590 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f02400030c0 tx=0x7f0240017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:30:23.473 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:23.469+0000 7f02457fa700 1 -- 172.21.15.110:0/3298506381 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0240011e90 con 0x7f02500cb060 2023-10-06T03:30:23.473 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:23.469+0000 7f02457fa700 1 -- 172.21.15.110:0/3298506381 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0240010ac0 con 0x7f02500cb060 2023-10-06T03:30:23.473 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:23.469+0000 7f02457fa700 1 -- 172.21.15.110:0/3298506381 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0240010e60 con 0x7f02500cb060 2023-10-06T03:30:23.473 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:23.469+0000 7f0257537700 1 -- 172.21.15.110:0/3298506381 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02500af730 con 0x7f02500cb060 2023-10-06T03:30:23.473 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:23.469+0000 7f0257537700 1 -- 172.21.15.110:0/3298506381 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02500c74a0 con 0x7f02500cb060 2023-10-06T03:30:23.473 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:23.469+0000 7f0257537700 1 -- 172.21.15.110:0/3298506381 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f025006ff60 con 0x7f02500cb060 2023-10-06T03:30:23.474 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:23.469+0000 7f02457fa700 1 -- 172.21.15.110:0/3298506381 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7f024001d640 con 0x7f02500cb060 2023-10-06T03:30:23.475 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:23.473+0000 7f02457fa700 1 --2- 172.21.15.110:0/3298506381 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f0234060550 0x7f0234062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:23.475 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:23.473+0000 7f02457fa700 1 -- 172.21.15.110:0/3298506381 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6953+0+0 (secure 0 0 0) 0x7f024007e3a0 con 0x7f02500cb060 2023-10-06T03:30:23.475 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:23.473+0000 7f0254ad2700 1 --2- 172.21.15.110:0/3298506381 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f0234060550 0x7f0234062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:23.476 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:23.473+0000 7f0254ad2700 1 --2- 172.21.15.110:0/3298506381 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f0234060550 0x7f0234062a00 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f0250157340 tx=0x7f024c001af0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:30:23.482 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:23.477+0000 7f02457fa700 1 -- 172.21.15.110:0/3298506381 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f024004e410 con 0x7f02500cb060 2023-10-06T03:30:23.647 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:23.641+0000 7f0257537700 1 -- 172.21.15.110:0/3298506381 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f02500ca730 con 0x7f02500cb060 2023-10-06T03:30:24.540 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:24.537+0000 7f02457fa700 1 -- 172.21.15.110:0/3298506381 <== mon.0 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v32) v1 ==== 181+0+0 (secure 0 0 0) 0x7f0240051a30 con 0x7f02500cb060 2023-10-06T03:30:24.553 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:24.549+0000 7f0257537700 1 -- 172.21.15.110:0/3298506381 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f0234060550 msgr2=0x7f0234062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:24.553 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:24.549+0000 7f0257537700 1 --2- 172.21.15.110:0/3298506381 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f0234060550 0x7f0234062a00 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f0250157340 tx=0x7f024c001af0 comp rx=0 tx=0).stop 2023-10-06T03:30:24.553 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:24.549+0000 7f0257537700 1 -- 172.21.15.110:0/3298506381 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f02500cb060 msgr2=0x7f02500b5590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:24.553 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:24.549+0000 7f0257537700 1 --2- 172.21.15.110:0/3298506381 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f02500cb060 0x7f02500b5590 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f02400030c0 tx=0x7f0240017000 comp rx=0 tx=0).stop 2023-10-06T03:30:24.553 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:24.549+0000 7f0257537700 1 -- 172.21.15.110:0/3298506381 shutdown_connections 2023-10-06T03:30:24.553 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:24.549+0000 7f0257537700 1 --2- 172.21.15.110:0/3298506381 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f0234060550 0x7f0234062a00 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:24.553 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:24.549+0000 7f0257537700 1 --2- 172.21.15.110:0/3298506381 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f02500aec70 0x7f0250156710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:24.553 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:24.549+0000 7f0257537700 1 --2- 172.21.15.110:0/3298506381 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f02500ce9b0 0x7f0250154930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:24.554 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:24.549+0000 7f0257537700 1 --2- 172.21.15.110:0/3298506381 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f02500cb060 0x7f02500b5590 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:24.554 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:24.549+0000 7f0257537700 1 -- 172.21.15.110:0/3298506381 >> 172.21.15.110:0/3298506381 conn(0x7f025006e860 msgr2=0x7f02500b00e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:30:24.554 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:24.549+0000 7f0257537700 1 -- 172.21.15.110:0/3298506381 shutdown_connections 2023-10-06T03:30:24.554 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:24.549+0000 7f0257537700 1 -- 172.21.15.110:0/3298506381 wait complete. 2023-10-06T03:30:24.555 INFO:teuthology.orchestra.run.smithi110.stderr:pool 'cephfs_metadata' removed 2023-10-06T03:30:24.565 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2023-10-06T03:30:24.672 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:24.669+0000 7fe651541700 1 Processor -- start 2023-10-06T03:30:24.672 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:24.669+0000 7fe651541700 1 -- start start 2023-10-06T03:30:24.673 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:24.669+0000 7fe651541700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe64c156de0 0x7fe64c0aa910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:24.673 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:24.669+0000 7fe651541700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe64c0ce880 con 0x7fe64c0ce430 2023-10-06T03:30:24.673 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:24.669+0000 7fe651541700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe64c0c3700 con 0x7fe64c0cc210 2023-10-06T03:30:24.673 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:24.669+0000 7fe651541700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe64c0ceac0 con 0x7fe64c156de0 2023-10-06T03:30:24.673 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:24.669+0000 7fe64b7fe700 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7fe64c0ce430 0x7fe64c0cbb00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:56570/0 (socket says 172.21.15.110:56570) 2023-10-06T03:30:24.674 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:24.669+0000 7fe64b7fe700 1 -- 172.21.15.110:0/1848759169 learned_addr learned my addr 172.21.15.110:0/1848759169 (peer_addr_for_me v1:172.21.15.110:0/0) 2023-10-06T03:30:24.674 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:24.669+0000 7fe64bfff700 1 --2- 172.21.15.110:0/1848759169 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe64c156de0 0x7fe64c0aa910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:24.674 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:24.669+0000 7fe64affd700 1 -- 172.21.15.110:0/1848759169 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1586931916 0 0) 0x7fe640001290 con 0x7fe64c0ce430 2023-10-06T03:30:24.674 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:24.669+0000 7fe64affd700 1 -- 172.21.15.110:0/1848759169 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe634003570 con 0x7fe64c0ce430 2023-10-06T03:30:24.676 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:24.669+0000 7fe64affd700 1 -- 172.21.15.110:0/1848759169 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3382363220 0 0) 0x7fe64c0c3700 con 0x7fe64c0cc210 2023-10-06T03:30:24.676 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:24.669+0000 7fe64affd700 1 -- 172.21.15.110:0/1848759169 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe640001290 con 0x7fe64c0cc210 2023-10-06T03:30:24.676 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:24.669+0000 7fe64affd700 1 -- 172.21.15.110:0/1848759169 <== mon.0 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1659630550 0 0) 0x7fe634003570 con 0x7fe64c0ce430 2023-10-06T03:30:24.677 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:24.669+0000 7fe64affd700 1 -- 172.21.15.110:0/1848759169 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe64c156de0 msgr2=0x7fe64c0aa910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:30:24.677 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:24.669+0000 7fe64affd700 1 --2- 172.21.15.110:0/1848759169 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe64c156de0 0x7fe64c0aa910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:24.677 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:24.669+0000 7fe64affd700 1 -- 172.21.15.110:0/1848759169 >> v1:172.21.15.192:6789/0 conn(0x7fe64c0cc210 legacy=0x7fe64c0c3b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:24.677 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:24.669+0000 7fe64affd700 1 -- 172.21.15.110:0/1848759169 --> v1:172.21.15.110:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe64c06c580 con 0x7fe64c0ce430 2023-10-06T03:30:24.677 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:24.669+0000 7fe64affd700 1 -- 172.21.15.110:0/1848759169 <== mon.0 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 118603703 0 0) 0x7fe640001780 con 0x7fe64c0ce430 2023-10-06T03:30:24.678 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:24.669+0000 7fe64affd700 1 -- 172.21.15.110:0/1848759169 >> v1:172.21.15.110:6789/0 conn(0x7fe64c0ce430 legacy=0x7fe64c0cbb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:24.678 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:24.669+0000 7fe64affd700 1 --2- 172.21.15.110:0/1848759169 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe634003c20 0x7fe634004010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:24.678 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:24.669+0000 7fe64affd700 1 --2- 172.21.15.110:0/1848759169 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe634004810 0x7fe634004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:24.678 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:24.669+0000 7fe64affd700 1 --2- 172.21.15.110:0/1848759169 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe64c156de0 0x7fe634005220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:24.678 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:24.669+0000 7fe64affd700 1 -- 172.21.15.110:0/1848759169 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe64c0ceac0 con 0x7fe634004810 2023-10-06T03:30:24.678 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:24.669+0000 7fe64affd700 1 -- 172.21.15.110:0/1848759169 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe63400bbf0 con 0x7fe634003c20 2023-10-06T03:30:24.678 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:24.669+0000 7fe64affd700 1 -- 172.21.15.110:0/1848759169 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe63400bdb0 con 0x7fe64c156de0 2023-10-06T03:30:24.678 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:24.669+0000 7fe64b7fe700 1 --2- 172.21.15.110:0/1848759169 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe634004810 0x7fe634004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:24.678 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:24.669+0000 7fe64bfff700 1 --2- 172.21.15.110:0/1848759169 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe634003c20 0x7fe634004010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:24.678 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:24.669+0000 7fe650d40700 1 --2- 172.21.15.110:0/1848759169 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe64c156de0 0x7fe634005220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:24.681 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:24.669+0000 7fe64b7fe700 1 -- 172.21.15.110:0/1848759169 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe64c156de0 msgr2=0x7fe634005220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:30:24.682 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:24.669+0000 7fe64b7fe700 1 --2- 172.21.15.110:0/1848759169 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe64c156de0 0x7fe634005220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:24.682 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:24.669+0000 7fe64b7fe700 1 -- 172.21.15.110:0/1848759169 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe634003c20 msgr2=0x7fe634004010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:30:24.682 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:24.669+0000 7fe64b7fe700 1 --2- 172.21.15.110:0/1848759169 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe634003c20 0x7fe634004010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:24.682 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:24.669+0000 7fe64b7fe700 1 -- 172.21.15.110:0/1848759169 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe634003570 con 0x7fe634004810 2023-10-06T03:30:24.682 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:24.669+0000 7fe64b7fe700 1 --2- 172.21.15.110:0/1848759169 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe634004810 0x7fe634004c80 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fe64000c3c0 tx=0x7fe640007970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6340f5286e3c039d server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:30:24.682 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:24.669+0000 7fe64affd700 1 -- 172.21.15.110:0/1848759169 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe64000cc90 con 0x7fe634004810 2023-10-06T03:30:24.682 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:24.669+0000 7fe64affd700 1 -- 172.21.15.110:0/1848759169 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe64000cdf0 con 0x7fe634004810 2023-10-06T03:30:24.682 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:24.669+0000 7fe64affd700 1 -- 172.21.15.110:0/1848759169 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe640007320 con 0x7fe634004810 2023-10-06T03:30:24.682 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:24.673+0000 7fe651541700 1 -- 172.21.15.110:0/1848759169 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe634004810 msgr2=0x7fe634004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:24.682 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:24.673+0000 7fe651541700 1 --2- 172.21.15.110:0/1848759169 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe634004810 0x7fe634004c80 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fe64000c3c0 tx=0x7fe640007970 comp rx=0 tx=0).stop 2023-10-06T03:30:24.682 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:24.673+0000 7fe650d40700 1 -- 172.21.15.110:0/1848759169 reap_dead start 2023-10-06T03:30:24.683 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:24.673+0000 7fe651541700 1 -- 172.21.15.110:0/1848759169 shutdown_connections 2023-10-06T03:30:24.683 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:24.673+0000 7fe651541700 1 -- 172.21.15.110:0/1848759169 >> 172.21.15.110:0/1848759169 conn(0x7fe64c06bfa0 msgr2=0x7fe64c0cdd20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:30:24.683 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:24.673+0000 7fe651541700 1 -- 172.21.15.110:0/1848759169 shutdown_connections 2023-10-06T03:30:24.683 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:24.673+0000 7fe651541700 1 -- 172.21.15.110:0/1848759169 wait complete. 2023-10-06T03:30:24.685 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:24.673+0000 7fe651541700 1 Processor -- start 2023-10-06T03:30:24.686 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:24.673+0000 7fe651541700 1 -- start start 2023-10-06T03:30:24.686 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:24.677+0000 7fe651541700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe634004810 0x7fe64c1677d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:24.686 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:24.677+0000 7fe651541700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe64c167f70 0x7fe64c1524a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:24.686 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:24.677+0000 7fe651541700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe64c1649e0 0x7fe64c0ca000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:24.686 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:24.677+0000 7fe651541700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe64c167d10 con 0x7fe634004810 2023-10-06T03:30:24.686 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:24.677+0000 7fe651541700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe64c1529e0 con 0x7fe64c167f70 2023-10-06T03:30:24.686 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:24.677+0000 7fe651541700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe64c1684e0 con 0x7fe64c1649e0 2023-10-06T03:30:24.686 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:24.677+0000 7fe64bfff700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe634004810 0x7fe64c1677d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:24.686 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:24.677+0000 7fe64bfff700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe634004810 0x7fe64c1677d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:40106/0 (socket says 172.21.15.110:40106) 2023-10-06T03:30:24.686 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:24.677+0000 7fe64bfff700 1 -- 172.21.15.110:0/470700616 learned_addr learned my addr 172.21.15.110:0/470700616 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:30:24.687 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:24.677+0000 7fe64b7fe700 1 --2- 172.21.15.110:0/470700616 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe64c167f70 0x7fe64c1524a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:24.687 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:24.677+0000 7fe650d40700 1 --2- 172.21.15.110:0/470700616 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe64c1649e0 0x7fe64c0ca000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:24.687 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:24.677+0000 7fe64bfff700 1 -- 172.21.15.110:0/470700616 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe64c1649e0 msgr2=0x7fe64c0ca000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:24.687 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:24.677+0000 7fe64bfff700 1 --2- 172.21.15.110:0/470700616 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe64c1649e0 0x7fe64c0ca000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:24.687 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:24.677+0000 7fe64bfff700 1 -- 172.21.15.110:0/470700616 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe64c167f70 msgr2=0x7fe64c1524a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:24.687 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:24.677+0000 7fe64bfff700 1 --2- 172.21.15.110:0/470700616 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe64c167f70 0x7fe64c1524a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:24.687 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:24.677+0000 7fe64bfff700 1 -- 172.21.15.110:0/470700616 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe64c1687c0 con 0x7fe634004810 2023-10-06T03:30:24.687 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:24.677+0000 7fe64bfff700 1 --2- 172.21.15.110:0/470700616 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe634004810 0x7fe64c1677d0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fe63c011990 tx=0x7fe63c008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:30:24.688 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:24.677+0000 7fe633fff700 1 -- 172.21.15.110:0/470700616 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe63c015670 con 0x7fe634004810 2023-10-06T03:30:24.688 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:24.677+0000 7fe633fff700 1 -- 172.21.15.110:0/470700616 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe63c0157d0 con 0x7fe634004810 2023-10-06T03:30:24.688 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:24.677+0000 7fe651541700 1 -- 172.21.15.110:0/470700616 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe64c168aa0 con 0x7fe634004810 2023-10-06T03:30:24.688 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:24.677+0000 7fe633fff700 1 -- 172.21.15.110:0/470700616 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe63c015b90 con 0x7fe634004810 2023-10-06T03:30:24.688 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:24.677+0000 7fe651541700 1 -- 172.21.15.110:0/470700616 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe64c169060 con 0x7fe634004810 2023-10-06T03:30:24.688 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:24.677+0000 7fe651541700 1 -- 172.21.15.110:0/470700616 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe64c070ff0 con 0x7fe634004810 2023-10-06T03:30:24.688 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:24.681+0000 7fe633fff700 1 -- 172.21.15.110:0/470700616 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7fe63c013410 con 0x7fe634004810 2023-10-06T03:30:24.688 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:24.681+0000 7fe633fff700 1 --2- 172.21.15.110:0/470700616 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fe638060580 0x7fe638062a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:24.689 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:24.681+0000 7fe64b7fe700 1 --2- 172.21.15.110:0/470700616 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fe638060580 0x7fe638062a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:24.689 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:24.681+0000 7fe633fff700 1 -- 172.21.15.110:0/470700616 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6515+0+0 (secure 0 0 0) 0x7fe63c07d1a0 con 0x7fe634004810 2023-10-06T03:30:24.689 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:24.681+0000 7fe64b7fe700 1 --2- 172.21.15.110:0/470700616 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fe638060580 0x7fe638062a30 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fe64000c3c0 tx=0x7fe640015000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:30:24.693 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:24.689+0000 7fe633fff700 1 -- 172.21.15.110:0/470700616 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fe63c04d3c0 con 0x7fe634004810 2023-10-06T03:30:24.853 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:24.849+0000 7fe651541700 1 -- 172.21.15.110:0/470700616 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fe64c0c9600 con 0x7fe634004810 2023-10-06T03:30:25.570 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:25.565+0000 7fe633fff700 1 -- 172.21.15.110:0/470700616 <== mon.0 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v33) v1 ==== 169+0+0 (secure 0 0 0) 0x7fe63c0509e0 con 0x7fe634004810 2023-10-06T03:30:25.576 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:25.573+0000 7fe651541700 1 -- 172.21.15.110:0/470700616 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fe638060580 msgr2=0x7fe638062a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:25.576 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:25.573+0000 7fe651541700 1 --2- 172.21.15.110:0/470700616 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fe638060580 0x7fe638062a30 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fe64000c3c0 tx=0x7fe640015000 comp rx=0 tx=0).stop 2023-10-06T03:30:25.577 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:25.573+0000 7fe651541700 1 -- 172.21.15.110:0/470700616 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe634004810 msgr2=0x7fe64c1677d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:25.577 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:25.573+0000 7fe651541700 1 --2- 172.21.15.110:0/470700616 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe634004810 0x7fe64c1677d0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fe63c011990 tx=0x7fe63c008040 comp rx=0 tx=0).stop 2023-10-06T03:30:25.577 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:25.573+0000 7fe651541700 1 -- 172.21.15.110:0/470700616 shutdown_connections 2023-10-06T03:30:25.578 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:25.573+0000 7fe651541700 1 --2- 172.21.15.110:0/470700616 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fe638060580 0x7fe638062a30 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:25.578 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:25.573+0000 7fe651541700 1 --2- 172.21.15.110:0/470700616 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe64c1649e0 0x7fe64c0ca000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:25.578 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:25.573+0000 7fe651541700 1 --2- 172.21.15.110:0/470700616 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe64c167f70 0x7fe64c1524a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:25.578 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:25.573+0000 7fe651541700 1 --2- 172.21.15.110:0/470700616 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe634004810 0x7fe64c1677d0 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:25.578 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:25.573+0000 7fe651541700 1 -- 172.21.15.110:0/470700616 >> 172.21.15.110:0/470700616 conn(0x7fe64c06bfa0 msgr2=0x7fe64c0c8870 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:30:25.579 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:25.577+0000 7fe651541700 1 -- 172.21.15.110:0/470700616 shutdown_connections 2023-10-06T03:30:25.580 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:25.577+0000 7fe651541700 1 -- 172.21.15.110:0/470700616 wait complete. 2023-10-06T03:30:25.584 INFO:teuthology.orchestra.run.smithi110.stderr:pool 'cephfs_data' removed 2023-10-06T03:30:25.597 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data_ec cephfs_data_ec --yes-i-really-really-mean-it 2023-10-06T03:30:25.706 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:25.701+0000 7fe50bc32700 1 Processor -- start 2023-10-06T03:30:25.706 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:25.701+0000 7fe50bc32700 1 -- start start 2023-10-06T03:30:25.707 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:25.701+0000 7fe50bc32700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fe504159150 0x7fe5040b2d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:25.707 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:25.701+0000 7fe50bc32700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5040cb730 con 0x7fe504158d80 2023-10-06T03:30:25.707 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:25.701+0000 7fe50bc32700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5041595a0 con 0x7fe5040b1de0 2023-10-06T03:30:25.707 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:25.701+0000 7fe50bc32700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe50406d1a0 con 0x7fe504159150 2023-10-06T03:30:25.707 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:25.701+0000 7fe5091cd700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fe504159150 0x7fe5040b2d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:25.707 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:25.701+0000 7fe5099ce700 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7fe504158d80 0x7fe50406c6e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:56584/0 (socket says 172.21.15.110:56584) 2023-10-06T03:30:25.707 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:25.701+0000 7fe5091cd700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fe504159150 0x7fe5040b2d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:40108/0 (socket says 172.21.15.110:40108) 2023-10-06T03:30:25.707 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:25.701+0000 7fe5099ce700 1 -- 172.21.15.110:0/3142357665 learned_addr learned my addr 172.21.15.110:0/3142357665 (peer_addr_for_me v1:172.21.15.110:0/0) 2023-10-06T03:30:25.709 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:25.705+0000 7fe5091cd700 1 -- 172.21.15.110:0/3142357665 >> v1:172.21.15.192:6789/0 conn(0x7fe5040b1de0 legacy=0x7fe5040b2260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:25.709 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:25.705+0000 7fe5091cd700 1 -- 172.21.15.110:0/3142357665 >> v1:172.21.15.110:6789/0 conn(0x7fe504158d80 legacy=0x7fe50406c6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:25.709 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:25.705+0000 7fe5091cd700 1 -- 172.21.15.110:0/3142357665 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5040cae00 con 0x7fe504159150 2023-10-06T03:30:25.709 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:25.705+0000 7fe5091cd700 1 --2- 172.21.15.110:0/3142357665 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe504159150 0x7fe5040b2d50 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fe500009910 tx=0x7fe5000091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9acc81a96579a1d3 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:30:25.709 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:25.705+0000 7fe5089cc700 1 -- 172.21.15.110:0/3142357665 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe50000b820 con 0x7fe504159150 2023-10-06T03:30:25.709 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:25.705+0000 7fe5089cc700 1 -- 172.21.15.110:0/3142357665 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe50000b980 con 0x7fe504159150 2023-10-06T03:30:25.710 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:25.705+0000 7fe5089cc700 1 -- 172.21.15.110:0/3142357665 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe50000bd20 con 0x7fe504159150 2023-10-06T03:30:25.710 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:25.705+0000 7fe50bc32700 1 -- 172.21.15.110:0/3142357665 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe504159150 msgr2=0x7fe5040b2d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:25.710 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:25.705+0000 7fe50bc32700 1 --2- 172.21.15.110:0/3142357665 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe504159150 0x7fe5040b2d50 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fe500009910 tx=0x7fe5000091a0 comp rx=0 tx=0).stop 2023-10-06T03:30:25.710 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:25.705+0000 7fe50bc32700 1 -- 172.21.15.110:0/3142357665 shutdown_connections 2023-10-06T03:30:25.710 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:25.705+0000 7fe50bc32700 1 --2- 172.21.15.110:0/3142357665 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe504159150 0x7fe5040b2d50 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:25.710 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:25.705+0000 7fe50bc32700 1 -- 172.21.15.110:0/3142357665 >> 172.21.15.110:0/3142357665 conn(0x7fe5040d0a50 msgr2=0x7fe5040cc9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:30:25.710 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:25.705+0000 7fe50bc32700 1 -- 172.21.15.110:0/3142357665 shutdown_connections 2023-10-06T03:30:25.710 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:25.705+0000 7fe50bc32700 1 -- 172.21.15.110:0/3142357665 wait complete. 2023-10-06T03:30:25.711 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:25.705+0000 7fe50bc32700 1 Processor -- start 2023-10-06T03:30:25.711 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:25.705+0000 7fe50bc32700 1 -- start start 2023-10-06T03:30:25.712 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:25.709+0000 7fe50bc32700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe5040b1de0 0x7fe5040c4460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:25.713 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:25.709+0000 7fe50bc32700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe504158d80 0x7fe50415a110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:25.713 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:25.709+0000 7fe50bc32700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe504159150 0x7fe504149300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:25.713 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:25.709+0000 7fe50bc32700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5040c49a0 con 0x7fe504159150 2023-10-06T03:30:25.713 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:25.709+0000 7fe50bc32700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5040c7960 con 0x7fe5040b1de0 2023-10-06T03:30:25.713 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:25.709+0000 7fe50bc32700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe50415a650 con 0x7fe504158d80 2023-10-06T03:30:25.713 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:25.709+0000 7fe5099ce700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe5040b1de0 0x7fe5040c4460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:25.713 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:25.709+0000 7fe50a1cf700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe504159150 0x7fe504149300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:25.713 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:25.709+0000 7fe5091cd700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe504158d80 0x7fe50415a110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:25.713 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:25.709+0000 7fe5099ce700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe5040b1de0 0x7fe5040c4460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.110:59980/0 (socket says 172.21.15.110:59980) 2023-10-06T03:30:25.713 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:25.709+0000 7fe5091cd700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe504158d80 0x7fe50415a110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3301/0 says I am v2:172.21.15.110:39834/0 (socket says 172.21.15.110:39834) 2023-10-06T03:30:25.713 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:25.709+0000 7fe50a1cf700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe504159150 0x7fe504149300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:40118/0 (socket says 172.21.15.110:40118) 2023-10-06T03:30:25.714 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:25.709+0000 7fe5099ce700 1 -- 172.21.15.110:0/4014752269 learned_addr learned my addr 172.21.15.110:0/4014752269 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:30:25.715 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:25.709+0000 7fe50a1cf700 1 -- 172.21.15.110:0/4014752269 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe504158d80 msgr2=0x7fe50415a110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:25.715 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:25.709+0000 7fe50a1cf700 1 --2- 172.21.15.110:0/4014752269 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe504158d80 0x7fe50415a110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:25.715 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:25.709+0000 7fe50a1cf700 1 -- 172.21.15.110:0/4014752269 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe5040b1de0 msgr2=0x7fe5040c4460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:25.715 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:25.709+0000 7fe50a1cf700 1 --2- 172.21.15.110:0/4014752269 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe5040b1de0 0x7fe5040c4460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:25.715 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:25.709+0000 7fe50a1cf700 1 -- 172.21.15.110:0/4014752269 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe500009420 con 0x7fe504159150 2023-10-06T03:30:25.715 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:25.709+0000 7fe50a1cf700 1 --2- 172.21.15.110:0/4014752269 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe504159150 0x7fe504149300 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fe4fc00c830 tx=0x7fe4fc001690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:30:25.715 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:25.709+0000 7fe4f5ffb700 1 -- 172.21.15.110:0/4014752269 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4fc012630 con 0x7fe504159150 2023-10-06T03:30:25.715 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:25.709+0000 7fe4f5ffb700 1 -- 172.21.15.110:0/4014752269 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4fc012790 con 0x7fe504159150 2023-10-06T03:30:25.715 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:25.709+0000 7fe50bc32700 1 -- 172.21.15.110:0/4014752269 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe50415a8d0 con 0x7fe504159150 2023-10-06T03:30:25.716 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:25.709+0000 7fe4f5ffb700 1 -- 172.21.15.110:0/4014752269 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4fc012b50 con 0x7fe504159150 2023-10-06T03:30:25.716 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:25.709+0000 7fe50bc32700 1 -- 172.21.15.110:0/4014752269 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5040c7310 con 0x7fe504159150 2023-10-06T03:30:25.716 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:25.713+0000 7fe50bc32700 1 -- 172.21.15.110:0/4014752269 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe5040c7af0 con 0x7fe504159150 2023-10-06T03:30:25.717 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:25.713+0000 7fe4f5ffb700 1 -- 172.21.15.110:0/4014752269 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7fe4fc011380 con 0x7fe504159150 2023-10-06T03:30:25.717 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:25.713+0000 7fe4f5ffb700 1 --2- 172.21.15.110:0/4014752269 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fe4e8060550 0x7fe4e8062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:25.718 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:25.713+0000 7fe4f5ffb700 1 -- 172.21.15.110:0/4014752269 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 6043+0+0 (secure 0 0 0) 0x7fe4fc07cb10 con 0x7fe504159150 2023-10-06T03:30:25.718 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:25.713+0000 7fe5099ce700 1 --2- 172.21.15.110:0/4014752269 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fe4e8060550 0x7fe4e8062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:25.718 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:25.713+0000 7fe5099ce700 1 --2- 172.21.15.110:0/4014752269 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fe4e8060550 0x7fe4e8062a00 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fe4f8007130 tx=0x7fe4f8007f70 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:30:25.725 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:25.721+0000 7fe4f5ffb700 1 -- 172.21.15.110:0/4014752269 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fe4fc04cf10 con 0x7fe504159150 2023-10-06T03:30:25.887 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:25.881+0000 7fe50bc32700 1 -- 172.21.15.110:0/4014752269 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fe50406ff60 con 0x7fe504159150 2023-10-06T03:30:26.579 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:26.573+0000 7fe4f5ffb700 1 -- 172.21.15.110:0/4014752269 <== mon.0 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data_ec' removed v34) v1 ==== 178+0+0 (secure 0 0 0) 0x7fe4fc050530 con 0x7fe504159150 2023-10-06T03:30:26.593 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:26.589+0000 7fe50bc32700 1 -- 172.21.15.110:0/4014752269 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fe4e8060550 msgr2=0x7fe4e8062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:26.593 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:26.589+0000 7fe50bc32700 1 --2- 172.21.15.110:0/4014752269 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fe4e8060550 0x7fe4e8062a00 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fe4f8007130 tx=0x7fe4f8007f70 comp rx=0 tx=0).stop 2023-10-06T03:30:26.593 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:26.589+0000 7fe50bc32700 1 -- 172.21.15.110:0/4014752269 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe504159150 msgr2=0x7fe504149300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:26.593 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:26.589+0000 7fe50bc32700 1 --2- 172.21.15.110:0/4014752269 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe504159150 0x7fe504149300 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fe4fc00c830 tx=0x7fe4fc001690 comp rx=0 tx=0).stop 2023-10-06T03:30:26.595 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:26.589+0000 7fe50bc32700 1 -- 172.21.15.110:0/4014752269 shutdown_connections 2023-10-06T03:30:26.595 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:26.589+0000 7fe50bc32700 1 --2- 172.21.15.110:0/4014752269 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fe4e8060550 0x7fe4e8062a00 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:26.595 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:26.589+0000 7fe50bc32700 1 --2- 172.21.15.110:0/4014752269 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe504159150 0x7fe504149300 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:26.595 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:26.589+0000 7fe50bc32700 1 --2- 172.21.15.110:0/4014752269 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe504158d80 0x7fe50415a110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:26.595 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:26.589+0000 7fe50bc32700 1 --2- 172.21.15.110:0/4014752269 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe5040b1de0 0x7fe5040c4460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:26.595 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:26.589+0000 7fe50bc32700 1 -- 172.21.15.110:0/4014752269 >> 172.21.15.110:0/4014752269 conn(0x7fe5040d0a50 msgr2=0x7fe5041559b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:30:26.596 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:26.589+0000 7fe50bc32700 1 -- 172.21.15.110:0/4014752269 shutdown_connections 2023-10-06T03:30:26.596 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:26.589+0000 7fe50bc32700 1 -- 172.21.15.110:0/4014752269 wait complete. 2023-10-06T03:30:26.597 INFO:teuthology.orchestra.run.smithi110.stderr:pool 'cephfs_data_ec' removed 2023-10-06T03:30:26.611 INFO:tasks.ceph.mds.a:Restarting daemon 2023-10-06T03:30:26.611 INFO:tasks.ceph.mds.a:Stopping old one... 2023-10-06T03:30:26.611 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-10-06T03:30:26.611 INFO:teuthology.orchestra.run:waiting for 300 2023-10-06T03:30:26.612 INFO:tasks.ceph.mds.c:Restarting daemon 2023-10-06T03:30:26.612 INFO:tasks.ceph.mds.c:Stopping old one... 2023-10-06T03:30:26.612 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-10-06T03:30:26.612 INFO:teuthology.orchestra.run:waiting for 300 2023-10-06T03:30:26.612 INFO:tasks.ceph.mds.b:Restarting daemon 2023-10-06T03:30:26.612 INFO:tasks.ceph.mds.b:Stopping old one... 2023-10-06T03:30:26.613 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2023-10-06T03:30:26.613 INFO:teuthology.orchestra.run:waiting for 300 2023-10-06T03:30:26.613 INFO:tasks.ceph.mds.d:Restarting daemon 2023-10-06T03:30:26.613 INFO:tasks.ceph.mds.d:Stopping old one... 2023-10-06T03:30:26.613 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2023-10-06T03:30:26.613 INFO:teuthology.orchestra.run:waiting for 300 2023-10-06T03:30:26.621 INFO:tasks.ceph.mds.a:Stopped 2023-10-06T03:30:26.621 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-10-06T03:30:26.623 INFO:tasks.ceph.mds.c:Stopped 2023-10-06T03:30:26.623 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-10-06T03:30:26.624 INFO:tasks.ceph.mds.a:Started 2023-10-06T03:30:26.624 DEBUG:teuthology.parallel:result is None 2023-10-06T03:30:26.625 INFO:tasks.ceph.mds.c:Started 2023-10-06T03:30:26.625 DEBUG:teuthology.parallel:result is None 2023-10-06T03:30:26.626 INFO:tasks.ceph.mds.d:Stopped 2023-10-06T03:30:26.626 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-10-06T03:30:26.627 INFO:tasks.ceph.mds.b:Stopped 2023-10-06T03:30:26.628 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2023-10-06T03:30:26.671 INFO:tasks.ceph.mds.d:Started 2023-10-06T03:30:26.671 DEBUG:teuthology.parallel:result is None 2023-10-06T03:30:26.673 INFO:tasks.ceph.mds.b:Started 2023-10-06T03:30:26.673 DEBUG:teuthology.parallel:result is None 2023-10-06T03:30:26.673 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2023-10-06T03:30:26.690 INFO:tasks.ceph.mds.c.smithi110.stdout:starting mds.c at 2023-10-06T03:30:26.691 INFO:tasks.ceph.mds.a.smithi110.stdout:starting mds.a at 2023-10-06T03:30:26.744 INFO:tasks.ceph.mds.b.smithi192.stdout:starting mds.b at 2023-10-06T03:30:26.745 INFO:tasks.ceph.mds.d.smithi192.stdout:starting mds.d at 2023-10-06T03:30:26.771 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:26.765+0000 7f4a4927b700 1 Processor -- start 2023-10-06T03:30:26.771 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:26.765+0000 7f4a4927b700 1 -- start start 2023-10-06T03:30:26.773 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:26.769+0000 7f4a4927b700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f4a440cb6f0 0x7f4a4406b7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:26.774 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:26.769+0000 7f4a4927b700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a4406c480 con 0x7f4a441565e0 2023-10-06T03:30:26.774 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:26.769+0000 7f4a4927b700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a44155b00 con 0x7f4a4414f710 2023-10-06T03:30:26.774 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:26.769+0000 7f4a4927b700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4a4406c760 con 0x7f4a440cb6f0 2023-10-06T03:30:26.774 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:26.769+0000 7f4a437fe700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f4a440cb6f0 0x7f4a4406b7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:26.774 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:26.769+0000 7f4a48a7a700 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f4a441565e0 0x7f4a44158ef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:56600/0 (socket says 172.21.15.110:56600) 2023-10-06T03:30:26.774 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:26.769+0000 7f4a48a7a700 1 -- 172.21.15.110:0/276482805 learned_addr learned my addr 172.21.15.110:0/276482805 (peer_addr_for_me v1:172.21.15.110:0/0) 2023-10-06T03:30:26.774 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:26.769+0000 7f4a437fe700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f4a440cb6f0 0x7f4a4406b7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:40156/0 (socket says 172.21.15.110:40156) 2023-10-06T03:30:26.774 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:26.769+0000 7f4a43fff700 1 --1- 172.21.15.110:0/276482805 >> v1:172.21.15.192:6789/0 conn(0x7f4a4414f710 0x7f4a440cafe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.110:36220/0 (socket says 172.21.15.110:36220) 2023-10-06T03:30:26.775 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:26.769+0000 7f4a437fe700 1 -- 172.21.15.110:0/276482805 >> v1:172.21.15.192:6789/0 conn(0x7f4a4414f710 legacy=0x7f4a440cafe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:26.775 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:26.769+0000 7f4a437fe700 1 -- 172.21.15.110:0/276482805 >> v1:172.21.15.110:6789/0 conn(0x7f4a441565e0 legacy=0x7f4a44158ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:26.775 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:26.769+0000 7f4a42ffd700 1 -- 172.21.15.110:0/276482805 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2995766091 0 0) 0x7f4a44155b00 con 0x7f4a4414f710 2023-10-06T03:30:26.775 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:26.769+0000 7f4a437fe700 1 -- 172.21.15.110:0/276482805 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a44155de0 con 0x7f4a440cb6f0 2023-10-06T03:30:26.776 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:26.769+0000 7f4a437fe700 1 --2- 172.21.15.110:0/276482805 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4a440cb6f0 0x7f4a4406b7c0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f4a380098e0 tx=0x7f4a380091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d9a9c0e148c89571 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:30:26.776 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:26.769+0000 7f4a42ffd700 1 -- 172.21.15.110:0/276482805 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a38009e80 con 0x7f4a440cb6f0 2023-10-06T03:30:26.776 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:26.769+0000 7f4a42ffd700 1 -- 172.21.15.110:0/276482805 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a3800b760 con 0x7f4a440cb6f0 2023-10-06T03:30:26.776 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:26.769+0000 7f4a42ffd700 1 -- 172.21.15.110:0/276482805 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a3800bae0 con 0x7f4a440cb6f0 2023-10-06T03:30:26.777 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:26.773+0000 7f4a4927b700 1 -- 172.21.15.110:0/276482805 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4a440cb6f0 msgr2=0x7f4a4406b7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:26.777 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:26.773+0000 7f4a4927b700 1 --2- 172.21.15.110:0/276482805 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4a440cb6f0 0x7f4a4406b7c0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f4a380098e0 tx=0x7f4a380091a0 comp rx=0 tx=0).stop 2023-10-06T03:30:26.777 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:26.773+0000 7f4a4927b700 1 -- 172.21.15.110:0/276482805 shutdown_connections 2023-10-06T03:30:26.777 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:26.773+0000 7f4a4927b700 1 --2- 172.21.15.110:0/276482805 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4a440cb6f0 0x7f4a4406b7c0 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:26.777 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:26.773+0000 7f4a4927b700 1 -- 172.21.15.110:0/276482805 >> 172.21.15.110:0/276482805 conn(0x7f4a4406e080 msgr2=0x7f4a4406e450 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:30:26.778 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:26.773+0000 7f4a4927b700 1 -- 172.21.15.110:0/276482805 shutdown_connections 2023-10-06T03:30:26.778 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:26.773+0000 7f4a4927b700 1 -- 172.21.15.110:0/276482805 wait complete. 2023-10-06T03:30:26.779 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:26.773+0000 7f4a4927b700 1 Processor -- start 2023-10-06T03:30:26.779 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:26.773+0000 7f4a4927b700 1 -- start start 2023-10-06T03:30:26.781 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:26.777+0000 7f4a4927b700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4a440cb6f0 0x7f4a4414e120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:26.782 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:26.777+0000 7f4a4927b700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4a4414f710 0x7f4a4415af00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:26.782 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:26.777+0000 7f4a4927b700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4a441565e0 0x7f4a4415b440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:26.782 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:26.777+0000 7f4a4927b700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a440b0b30 con 0x7f4a441565e0 2023-10-06T03:30:26.782 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:26.777+0000 7f4a4927b700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a441525b0 con 0x7f4a4414f710 2023-10-06T03:30:26.782 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:26.777+0000 7f4a4927b700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a440b0200 con 0x7f4a440cb6f0 2023-10-06T03:30:26.782 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:26.777+0000 7f4a48a7a700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4a441565e0 0x7f4a4415b440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:26.782 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:26.777+0000 7f4a48a7a700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4a441565e0 0x7f4a4415b440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:40158/0 (socket says 172.21.15.110:40158) 2023-10-06T03:30:26.782 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:26.777+0000 7f4a48a7a700 1 -- 172.21.15.110:0/424785635 learned_addr learned my addr 172.21.15.110:0/424785635 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:30:26.782 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:26.777+0000 7f4a43fff700 1 --2- 172.21.15.110:0/424785635 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4a440cb6f0 0x7f4a4414e120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:26.782 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:26.777+0000 7f4a437fe700 1 --2- 172.21.15.110:0/424785635 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4a4414f710 0x7f4a4415af00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:26.783 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:26.777+0000 7f4a48a7a700 1 -- 172.21.15.110:0/424785635 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4a440cb6f0 msgr2=0x7f4a4414e120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:26.783 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:26.777+0000 7f4a48a7a700 1 --2- 172.21.15.110:0/424785635 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4a440cb6f0 0x7f4a4414e120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:26.783 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:26.777+0000 7f4a48a7a700 1 -- 172.21.15.110:0/424785635 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4a4414f710 msgr2=0x7f4a4415af00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:26.783 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:26.777+0000 7f4a48a7a700 1 --2- 172.21.15.110:0/424785635 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4a4414f710 0x7f4a4415af00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:26.783 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:26.777+0000 7f4a48a7a700 1 -- 172.21.15.110:0/424785635 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a38009420 con 0x7f4a441565e0 2023-10-06T03:30:26.783 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:26.777+0000 7f4a437fe700 1 --2- 172.21.15.110:0/424785635 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4a4414f710 0x7f4a4415af00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-06T03:30:26.783 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:26.777+0000 7f4a48a7a700 1 --2- 172.21.15.110:0/424785635 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4a441565e0 0x7f4a4415b440 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f4a3c00ce00 tx=0x7f4a3c001b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:30:26.783 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:26.777+0000 7f4a2bfff700 1 -- 172.21.15.110:0/424785635 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a3c00ab60 con 0x7f4a441565e0 2023-10-06T03:30:26.783 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:26.777+0000 7f4a2bfff700 1 -- 172.21.15.110:0/424785635 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a3c00acc0 con 0x7f4a441565e0 2023-10-06T03:30:26.783 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:26.777+0000 7f4a4927b700 1 -- 172.21.15.110:0/424785635 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a440b04e0 con 0x7f4a441565e0 2023-10-06T03:30:26.783 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:26.777+0000 7f4a2bfff700 1 -- 172.21.15.110:0/424785635 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a3c007340 con 0x7f4a441565e0 2023-10-06T03:30:26.784 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:26.777+0000 7f4a4927b700 1 -- 172.21.15.110:0/424785635 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a440b0fc0 con 0x7f4a441565e0 2023-10-06T03:30:26.784 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:26.777+0000 7f4a4927b700 1 -- 172.21.15.110:0/424785635 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a440b0ce0 con 0x7f4a441565e0 2023-10-06T03:30:26.784 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:26.777+0000 7f4a2bfff700 1 -- 172.21.15.110:0/424785635 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7f4a3c0076f0 con 0x7f4a441565e0 2023-10-06T03:30:26.784 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:26.781+0000 7f4a2bfff700 1 --2- 172.21.15.110:0/424785635 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f4a30060550 0x7f4a30062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:26.785 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:26.781+0000 7f4a43fff700 1 --2- 172.21.15.110:0/424785635 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f4a30060550 0x7f4a30062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:26.785 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:26.781+0000 7f4a2bfff700 1 -- 172.21.15.110:0/424785635 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5612+0+0 (secure 0 0 0) 0x7f4a3c07cc90 con 0x7f4a441565e0 2023-10-06T03:30:26.785 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:26.781+0000 7f4a43fff700 1 --2- 172.21.15.110:0/424785635 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f4a30060550 0x7f4a30062a00 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f4a340011e0 tx=0x7f4a34007bf0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:30:26.793 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:26.789+0000 7f4a2bfff700 1 -- 172.21.15.110:0/424785635 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f4a440b0ce0 con 0x7f4a441565e0 2023-10-06T03:30:26.962 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:26.957+0000 7f4a4927b700 1 -- 172.21.15.110:0/424785635 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f4a440b0ce0 con 0x7f4a441565e0 2023-10-06T03:30:27.594 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:27.589+0000 7f4a2bfff700 1 -- 172.21.15.110:0/424785635 <== mon.0 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v36) v1 ==== 97+0+0 (secure 0 0 0) 0x7f4a440b0ce0 con 0x7f4a441565e0 2023-10-06T03:30:27.599 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:27.593+0000 7f4a4927b700 1 -- 172.21.15.110:0/424785635 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f4a30060550 msgr2=0x7f4a30062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:27.599 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:27.593+0000 7f4a4927b700 1 --2- 172.21.15.110:0/424785635 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f4a30060550 0x7f4a30062a00 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f4a340011e0 tx=0x7f4a34007bf0 comp rx=0 tx=0).stop 2023-10-06T03:30:27.599 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:27.593+0000 7f4a4927b700 1 -- 172.21.15.110:0/424785635 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4a441565e0 msgr2=0x7f4a4415b440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:27.599 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:27.593+0000 7f4a4927b700 1 --2- 172.21.15.110:0/424785635 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4a441565e0 0x7f4a4415b440 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f4a3c00ce00 tx=0x7f4a3c001b60 comp rx=0 tx=0).stop 2023-10-06T03:30:27.599 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:27.593+0000 7f4a4927b700 1 -- 172.21.15.110:0/424785635 shutdown_connections 2023-10-06T03:30:27.600 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:27.593+0000 7f4a4927b700 1 --2- 172.21.15.110:0/424785635 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f4a30060550 0x7f4a30062a00 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:27.600 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:27.593+0000 7f4a4927b700 1 --2- 172.21.15.110:0/424785635 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4a441565e0 0x7f4a4415b440 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:27.600 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:27.593+0000 7f4a4927b700 1 --2- 172.21.15.110:0/424785635 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4a4414f710 0x7f4a4415af00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:27.600 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:27.593+0000 7f4a4927b700 1 --2- 172.21.15.110:0/424785635 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4a440cb6f0 0x7f4a4414e120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:27.600 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:27.593+0000 7f4a4927b700 1 -- 172.21.15.110:0/424785635 >> 172.21.15.110:0/424785635 conn(0x7f4a4406e080 msgr2=0x7f4a44155080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:30:27.600 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:27.593+0000 7f4a4927b700 1 -- 172.21.15.110:0/424785635 shutdown_connections 2023-10-06T03:30:27.600 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:27.593+0000 7f4a4927b700 1 -- 172.21.15.110:0/424785635 wait complete. 2023-10-06T03:30:27.601 INFO:teuthology.orchestra.run.smithi110.stderr: removed all blocklist entries 2023-10-06T03:30:27.616 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2023-10-06T03:30:27.773 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:27.769+0000 7fe1a0909700 1 Processor -- start 2023-10-06T03:30:27.773 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:27.769+0000 7fe1a0909700 1 -- start start 2023-10-06T03:30:27.775 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:27.769+0000 7fe1a0909700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe19c156b80 0x7fe19c0c68d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:27.775 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:27.769+0000 7fe1a0909700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fe19c0c6e10 0x7fe19c0ad560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:27.775 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:27.769+0000 7fe1a0909700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe19c0c7f30 con 0x7fe19c156f50 2023-10-06T03:30:27.775 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:27.769+0000 7fe1a0909700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe19c0c7320 con 0x7fe19c0c6e10 2023-10-06T03:30:27.776 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:27.769+0000 7fe1a0909700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe19c0c7600 con 0x7fe19c156b80 2023-10-06T03:30:27.776 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:27.769+0000 7fe19bfff700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fe19c0c6e10 0x7fe19c0ad560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:27.776 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:27.769+0000 7fe19bfff700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fe19c0c6e10 0x7fe19c0ad560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:40170/0 (socket says 172.21.15.110:40170) 2023-10-06T03:30:27.776 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:27.769+0000 7fe19bfff700 1 -- 172.21.15.110:0/4271062458 learned_addr learned my addr 172.21.15.110:0/4271062458 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:30:27.776 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:27.769+0000 7fe19b7fe700 1 --2- 172.21.15.110:0/4271062458 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe19c156b80 0x7fe19c0c68d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:27.776 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:27.769+0000 7fe19bfff700 1 -- 172.21.15.110:0/4271062458 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe19c156b80 msgr2=0x7fe19c0c68d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:30:27.776 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:27.769+0000 7fe19bfff700 1 --2- 172.21.15.110:0/4271062458 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe19c156b80 0x7fe19c0c68d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:27.776 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:27.769+0000 7fe19bfff700 1 -- 172.21.15.110:0/4271062458 >> v1:172.21.15.192:6789/0 conn(0x7fe19c156f50 legacy=0x7fe19c0b0e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:27.776 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:27.769+0000 7fe19bfff700 1 -- 172.21.15.110:0/4271062458 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe19c0c78e0 con 0x7fe19c0c6e10 2023-10-06T03:30:27.776 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:27.769+0000 7fe19b7fe700 1 --2- 172.21.15.110:0/4271062458 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe19c156b80 0x7fe19c0c68d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-06T03:30:27.777 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:27.773+0000 7fe19bfff700 1 --2- 172.21.15.110:0/4271062458 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe19c0c6e10 0x7fe19c0ad560 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fe1940099b0 tx=0x7fe194009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=971e97a6d674b1d4 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:30:27.778 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:27.773+0000 7fe19a7fc700 1 -- 172.21.15.110:0/4271062458 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe19400b860 con 0x7fe19c0c6e10 2023-10-06T03:30:27.778 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:27.773+0000 7fe19a7fc700 1 -- 172.21.15.110:0/4271062458 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe19400b9c0 con 0x7fe19c0c6e10 2023-10-06T03:30:27.778 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:27.773+0000 7fe19a7fc700 1 -- 172.21.15.110:0/4271062458 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe19400bd60 con 0x7fe19c0c6e10 2023-10-06T03:30:27.778 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:27.773+0000 7fe1a0909700 1 -- 172.21.15.110:0/4271062458 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe19c0c6e10 msgr2=0x7fe19c0ad560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:27.778 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:27.773+0000 7fe1a0909700 1 --2- 172.21.15.110:0/4271062458 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe19c0c6e10 0x7fe19c0ad560 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fe1940099b0 tx=0x7fe194009250 comp rx=0 tx=0).stop 2023-10-06T03:30:27.778 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:27.773+0000 7fe1a0909700 1 -- 172.21.15.110:0/4271062458 shutdown_connections 2023-10-06T03:30:27.779 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:27.773+0000 7fe1a0909700 1 --2- 172.21.15.110:0/4271062458 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe19c0c6e10 0x7fe19c0ad560 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:27.779 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:27.773+0000 7fe1a0909700 1 --2- 172.21.15.110:0/4271062458 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe19c156b80 0x7fe19c0c68d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:27.779 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:27.773+0000 7fe1a0909700 1 -- 172.21.15.110:0/4271062458 >> 172.21.15.110:0/4271062458 conn(0x7fe19c0cc470 msgr2=0x7fe19c0c8410 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:30:27.779 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:27.773+0000 7fe1a0909700 1 -- 172.21.15.110:0/4271062458 shutdown_connections 2023-10-06T03:30:27.779 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:27.773+0000 7fe1a0909700 1 -- 172.21.15.110:0/4271062458 wait complete. 2023-10-06T03:30:27.779 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:27.773+0000 7fe1a0909700 1 Processor -- start 2023-10-06T03:30:27.779 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:27.773+0000 7fe1a0909700 1 -- start start 2023-10-06T03:30:27.781 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:27.777+0000 7fe1a0909700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe19c0c6e10 0x7fe19c15a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:27.781 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:27.777+0000 7fe1a0909700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe19c156b80 0x7fe19c15ae30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:27.781 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:27.777+0000 7fe1a0909700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe19c156f50 0x7fe19c0c1300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:27.781 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:27.777+0000 7fe1a0909700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe19c158df0 con 0x7fe19c156f50 2023-10-06T03:30:27.781 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:27.777+0000 7fe1a0909700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe19c157f00 con 0x7fe19c156b80 2023-10-06T03:30:27.781 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:27.777+0000 7fe1a0909700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe19c1581e0 con 0x7fe19c0c6e10 2023-10-06T03:30:27.781 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:27.777+0000 7fe19bfff700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe19c156f50 0x7fe19c0c1300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:27.781 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:27.777+0000 7fe19affd700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe19c156b80 0x7fe19c15ae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:27.782 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:27.777+0000 7fe19b7fe700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe19c0c6e10 0x7fe19c15a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:27.782 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:27.777+0000 7fe19bfff700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe19c156f50 0x7fe19c0c1300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:40184/0 (socket says 172.21.15.110:40184) 2023-10-06T03:30:27.782 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:27.777+0000 7fe19bfff700 1 -- 172.21.15.110:0/1342041360 learned_addr learned my addr 172.21.15.110:0/1342041360 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:30:27.783 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:27.777+0000 7fe19bfff700 1 -- 172.21.15.110:0/1342041360 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe19c0c6e10 msgr2=0x7fe19c15a8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:27.783 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:27.777+0000 7fe19bfff700 1 --2- 172.21.15.110:0/1342041360 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe19c0c6e10 0x7fe19c15a8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:27.783 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:27.777+0000 7fe19bfff700 1 -- 172.21.15.110:0/1342041360 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe19c156b80 msgr2=0x7fe19c15ae30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:27.783 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:27.777+0000 7fe19bfff700 1 --2- 172.21.15.110:0/1342041360 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe19c156b80 0x7fe19c15ae30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:27.783 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:27.777+0000 7fe19bfff700 1 -- 172.21.15.110:0/1342041360 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1940094f0 con 0x7fe19c156f50 2023-10-06T03:30:27.784 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:27.777+0000 7fe19bfff700 1 --2- 172.21.15.110:0/1342041360 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe19c156f50 0x7fe19c0c1300 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7fe1940090b0 tx=0x7fe194017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:30:27.784 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:27.777+0000 7fe1837fe700 1 -- 172.21.15.110:0/1342041360 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe19400b860 con 0x7fe19c156f50 2023-10-06T03:30:27.784 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:27.777+0000 7fe1837fe700 1 -- 172.21.15.110:0/1342041360 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe19401f070 con 0x7fe19c156f50 2023-10-06T03:30:27.784 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:27.777+0000 7fe1a0909700 1 -- 172.21.15.110:0/1342041360 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe19c1584c0 con 0x7fe19c156f50 2023-10-06T03:30:27.784 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:27.777+0000 7fe1837fe700 1 -- 172.21.15.110:0/1342041360 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe194011ea0 con 0x7fe19c156f50 2023-10-06T03:30:27.784 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:27.777+0000 7fe1a0909700 1 -- 172.21.15.110:0/1342041360 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe19c158a80 con 0x7fe19c156f50 2023-10-06T03:30:27.784 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:27.781+0000 7fe1a0909700 1 -- 172.21.15.110:0/1342041360 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe19c158fa0 con 0x7fe19c156f50 2023-10-06T03:30:27.785 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:27.781+0000 7fe1837fe700 1 -- 172.21.15.110:0/1342041360 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7fe19401aaa0 con 0x7fe19c156f50 2023-10-06T03:30:27.785 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:27.781+0000 7fe1837fe700 1 --2- 172.21.15.110:0/1342041360 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fe188060580 0x7fe188062a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:27.785 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:27.781+0000 7fe19b7fe700 1 --2- 172.21.15.110:0/1342041360 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fe188060580 0x7fe188062a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:27.785 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:27.781+0000 7fe1837fe700 1 -- 172.21.15.110:0/1342041360 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 5526+0+0 (secure 0 0 0) 0x7fe194021070 con 0x7fe19c156f50 2023-10-06T03:30:27.786 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:27.781+0000 7fe19b7fe700 1 --2- 172.21.15.110:0/1342041360 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fe188060580 0x7fe188062a30 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fe190006f00 tx=0x7fe190008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:30:27.793 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:27.789+0000 7fe1837fe700 1 -- 172.21.15.110:0/1342041360 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fe19c158fa0 con 0x7fe19c156f50 2023-10-06T03:30:28.010 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:28.005+0000 7fe1a0909700 1 -- 172.21.15.110:0/1342041360 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7fe19c070ff0 con 0x7fe19c156f50 2023-10-06T03:30:28.011 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:28.005+0000 7fe1837fe700 1 -- 172.21.15.110:0/1342041360 <== mon.0 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v2) v1 ==== 80+0+6645 (secure 0 0 0) 0x7fe194015090 con 0x7fe19c156f50 2023-10-06T03:30:28.011 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-10-06T03:30:28.012 INFO:teuthology.orchestra.run.smithi110.stdout:{ 2023-10-06T03:30:28.012 INFO:teuthology.orchestra.run.smithi110.stdout: "auth_dump": [ 2023-10-06T03:30:28.012 INFO:teuthology.orchestra.run.smithi110.stdout: { 2023-10-06T03:30:28.012 INFO:teuthology.orchestra.run.smithi110.stdout: "entity": "mds.a", 2023-10-06T03:30:28.012 INFO:teuthology.orchestra.run.smithi110.stdout: "key": "AQDnfh9l2k3cIRAAP843sFwH4LEZoCwTml4blQ==", 2023-10-06T03:30:28.013 INFO:teuthology.orchestra.run.smithi110.stdout: "caps": { 2023-10-06T03:30:28.013 INFO:teuthology.orchestra.run.smithi110.stdout: "mds": "allow", 2023-10-06T03:30:28.013 INFO:teuthology.orchestra.run.smithi110.stdout: "mgr": "allow *", 2023-10-06T03:30:28.013 INFO:teuthology.orchestra.run.smithi110.stdout: "mon": "allow *", 2023-10-06T03:30:28.013 INFO:teuthology.orchestra.run.smithi110.stdout: "osd": "allow *" 2023-10-06T03:30:28.013 INFO:teuthology.orchestra.run.smithi110.stdout: } 2023-10-06T03:30:28.013 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2023-10-06T03:30:28.014 INFO:teuthology.orchestra.run.smithi110.stdout: { 2023-10-06T03:30:28.014 INFO:teuthology.orchestra.run.smithi110.stdout: "entity": "mds.b", 2023-10-06T03:30:28.014 INFO:teuthology.orchestra.run.smithi110.stdout: "key": "AQDnfh9lq/rVKxAAKyRhV1HafDy0SM+kEboXOQ==", 2023-10-06T03:30:28.014 INFO:teuthology.orchestra.run.smithi110.stdout: "caps": { 2023-10-06T03:30:28.014 INFO:teuthology.orchestra.run.smithi110.stdout: "mds": "allow", 2023-10-06T03:30:28.014 INFO:teuthology.orchestra.run.smithi110.stdout: "mgr": "allow *", 2023-10-06T03:30:28.014 INFO:teuthology.orchestra.run.smithi110.stdout: "mon": "allow *", 2023-10-06T03:30:28.015 INFO:teuthology.orchestra.run.smithi110.stdout: "osd": "allow *" 2023-10-06T03:30:28.015 INFO:teuthology.orchestra.run.smithi110.stdout: } 2023-10-06T03:30:28.015 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2023-10-06T03:30:28.015 INFO:teuthology.orchestra.run.smithi110.stdout: { 2023-10-06T03:30:28.015 INFO:teuthology.orchestra.run.smithi110.stdout: "entity": "mds.c", 2023-10-06T03:30:28.016 INFO:teuthology.orchestra.run.smithi110.stdout: "key": "AQDnfh9lzTNRJRAAeLHxIFxyWzK0ECu9UEhjSA==", 2023-10-06T03:30:28.016 INFO:teuthology.orchestra.run.smithi110.stdout: "caps": { 2023-10-06T03:30:28.016 INFO:teuthology.orchestra.run.smithi110.stdout: "mds": "allow", 2023-10-06T03:30:28.016 INFO:teuthology.orchestra.run.smithi110.stdout: "mgr": "allow *", 2023-10-06T03:30:28.016 INFO:teuthology.orchestra.run.smithi110.stdout: "mon": "allow *", 2023-10-06T03:30:28.016 INFO:teuthology.orchestra.run.smithi110.stdout: "osd": "allow *" 2023-10-06T03:30:28.017 INFO:teuthology.orchestra.run.smithi110.stdout: } 2023-10-06T03:30:28.017 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2023-10-06T03:30:28.017 INFO:teuthology.orchestra.run.smithi110.stdout: { 2023-10-06T03:30:28.017 INFO:teuthology.orchestra.run.smithi110.stdout: "entity": "mds.d", 2023-10-06T03:30:28.017 INFO:teuthology.orchestra.run.smithi110.stdout: "key": "AQDnfh9lAHsbLxAA3HjlMJWG9MusdTNd0t/OBw==", 2023-10-06T03:30:28.017 INFO:teuthology.orchestra.run.smithi110.stdout: "caps": { 2023-10-06T03:30:28.017 INFO:teuthology.orchestra.run.smithi110.stdout: "mds": "allow", 2023-10-06T03:30:28.017 INFO:teuthology.orchestra.run.smithi110.stdout: "mgr": "allow *", 2023-10-06T03:30:28.018 INFO:teuthology.orchestra.run.smithi110.stdout: "mon": "allow *", 2023-10-06T03:30:28.018 INFO:teuthology.orchestra.run.smithi110.stdout: "osd": "allow *" 2023-10-06T03:30:28.018 INFO:teuthology.orchestra.run.smithi110.stdout: } 2023-10-06T03:30:28.018 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2023-10-06T03:30:28.018 INFO:teuthology.orchestra.run.smithi110.stdout: { 2023-10-06T03:30:28.018 INFO:teuthology.orchestra.run.smithi110.stdout: "entity": "osd.0", 2023-10-06T03:30:28.018 INFO:teuthology.orchestra.run.smithi110.stdout: "key": "AQDofh9ldKF3HhAAYvogispXU05WmG+yC/SnaA==", 2023-10-06T03:30:28.018 INFO:teuthology.orchestra.run.smithi110.stdout: "caps": { 2023-10-06T03:30:28.018 INFO:teuthology.orchestra.run.smithi110.stdout: "mgr": "allow profile osd", 2023-10-06T03:30:28.018 INFO:teuthology.orchestra.run.smithi110.stdout: "mon": "allow profile osd", 2023-10-06T03:30:28.018 INFO:teuthology.orchestra.run.smithi110.stdout: "osd": "allow *" 2023-10-06T03:30:28.018 INFO:teuthology.orchestra.run.smithi110.stdout: } 2023-10-06T03:30:28.019 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2023-10-06T03:30:28.019 INFO:teuthology.orchestra.run.smithi110.stdout: { 2023-10-06T03:30:28.019 INFO:teuthology.orchestra.run.smithi110.stdout: "entity": "osd.1", 2023-10-06T03:30:28.019 INFO:teuthology.orchestra.run.smithi110.stdout: "key": "AQDrfh9lrZcsIBAA6tz8bfWAcz7Oti3fQgKVbQ==", 2023-10-06T03:30:28.019 INFO:teuthology.orchestra.run.smithi110.stdout: "caps": { 2023-10-06T03:30:28.019 INFO:teuthology.orchestra.run.smithi110.stdout: "mgr": "allow profile osd", 2023-10-06T03:30:28.019 INFO:teuthology.orchestra.run.smithi110.stdout: "mon": "allow profile osd", 2023-10-06T03:30:28.019 INFO:teuthology.orchestra.run.smithi110.stdout: "osd": "allow *" 2023-10-06T03:30:28.019 INFO:teuthology.orchestra.run.smithi110.stdout: } 2023-10-06T03:30:28.019 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2023-10-06T03:30:28.019 INFO:teuthology.orchestra.run.smithi110.stdout: { 2023-10-06T03:30:28.019 INFO:teuthology.orchestra.run.smithi110.stdout: "entity": "osd.2", 2023-10-06T03:30:28.020 INFO:teuthology.orchestra.run.smithi110.stdout: "key": "AQDufh9lC27aJBAAjATAn2aGrDjlOuVJx/iEpw==", 2023-10-06T03:30:28.020 INFO:teuthology.orchestra.run.smithi110.stdout: "caps": { 2023-10-06T03:30:28.020 INFO:teuthology.orchestra.run.smithi110.stdout: "mgr": "allow profile osd", 2023-10-06T03:30:28.020 INFO:teuthology.orchestra.run.smithi110.stdout: "mon": "allow profile osd", 2023-10-06T03:30:28.020 INFO:teuthology.orchestra.run.smithi110.stdout: "osd": "allow *" 2023-10-06T03:30:28.020 INFO:teuthology.orchestra.run.smithi110.stdout: } 2023-10-06T03:30:28.020 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2023-10-06T03:30:28.020 INFO:teuthology.orchestra.run.smithi110.stdout: { 2023-10-06T03:30:28.020 INFO:teuthology.orchestra.run.smithi110.stdout: "entity": "osd.3", 2023-10-06T03:30:28.020 INFO:teuthology.orchestra.run.smithi110.stdout: "key": "AQDxfh9lZb2JJhAAfBL3LEcvAP+ZzWh3sZzw+w==", 2023-10-06T03:30:28.020 INFO:teuthology.orchestra.run.smithi110.stdout: "caps": { 2023-10-06T03:30:28.020 INFO:teuthology.orchestra.run.smithi110.stdout: "mgr": "allow profile osd", 2023-10-06T03:30:28.021 INFO:teuthology.orchestra.run.smithi110.stdout: "mon": "allow profile osd", 2023-10-06T03:30:28.021 INFO:teuthology.orchestra.run.smithi110.stdout: "osd": "allow *" 2023-10-06T03:30:28.021 INFO:teuthology.orchestra.run.smithi110.stdout: } 2023-10-06T03:30:28.021 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2023-10-06T03:30:28.021 INFO:teuthology.orchestra.run.smithi110.stdout: { 2023-10-06T03:30:28.021 INFO:teuthology.orchestra.run.smithi110.stdout: "entity": "osd.4", 2023-10-06T03:30:28.021 INFO:teuthology.orchestra.run.smithi110.stdout: "key": "AQD3fh9lkOzRFBAAy8fDHNpPO1M6ebXU8QDvyw==", 2023-10-06T03:30:28.021 INFO:teuthology.orchestra.run.smithi110.stdout: "caps": { 2023-10-06T03:30:28.021 INFO:teuthology.orchestra.run.smithi110.stdout: "mgr": "allow profile osd", 2023-10-06T03:30:28.021 INFO:teuthology.orchestra.run.smithi110.stdout: "mon": "allow profile osd", 2023-10-06T03:30:28.021 INFO:teuthology.orchestra.run.smithi110.stdout: "osd": "allow *" 2023-10-06T03:30:28.022 INFO:teuthology.orchestra.run.smithi110.stdout: } 2023-10-06T03:30:28.022 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2023-10-06T03:30:28.022 INFO:teuthology.orchestra.run.smithi110.stdout: { 2023-10-06T03:30:28.022 INFO:teuthology.orchestra.run.smithi110.stdout: "entity": "osd.5", 2023-10-06T03:30:28.022 INFO:teuthology.orchestra.run.smithi110.stdout: "key": "AQD6fh9lj46oJhAAxiJqS3dqdeqaBS+m8omEKQ==", 2023-10-06T03:30:28.022 INFO:teuthology.orchestra.run.smithi110.stdout: "caps": { 2023-10-06T03:30:28.022 INFO:teuthology.orchestra.run.smithi110.stdout: "mgr": "allow profile osd", 2023-10-06T03:30:28.022 INFO:teuthology.orchestra.run.smithi110.stdout: "mon": "allow profile osd", 2023-10-06T03:30:28.022 INFO:teuthology.orchestra.run.smithi110.stdout: "osd": "allow *" 2023-10-06T03:30:28.022 INFO:teuthology.orchestra.run.smithi110.stdout: } 2023-10-06T03:30:28.022 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2023-10-06T03:30:28.023 INFO:teuthology.orchestra.run.smithi110.stdout: { 2023-10-06T03:30:28.023 INFO:teuthology.orchestra.run.smithi110.stdout: "entity": "osd.6", 2023-10-06T03:30:28.023 INFO:teuthology.orchestra.run.smithi110.stdout: "key": "AQD9fh9lxYvJKxAAwGvw/+afqFHVCYdJ+Dr0kw==", 2023-10-06T03:30:28.023 INFO:teuthology.orchestra.run.smithi110.stdout: "caps": { 2023-10-06T03:30:28.023 INFO:teuthology.orchestra.run.smithi110.stdout: "mgr": "allow profile osd", 2023-10-06T03:30:28.023 INFO:teuthology.orchestra.run.smithi110.stdout: "mon": "allow profile osd", 2023-10-06T03:30:28.023 INFO:teuthology.orchestra.run.smithi110.stdout: "osd": "allow *" 2023-10-06T03:30:28.023 INFO:teuthology.orchestra.run.smithi110.stdout: } 2023-10-06T03:30:28.023 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2023-10-06T03:30:28.023 INFO:teuthology.orchestra.run.smithi110.stdout: { 2023-10-06T03:30:28.023 INFO:teuthology.orchestra.run.smithi110.stdout: "entity": "osd.7", 2023-10-06T03:30:28.023 INFO:teuthology.orchestra.run.smithi110.stdout: "key": "AQAAfx9lY3tFLhAAvVSBIrNSWYBJZflMyT8eLw==", 2023-10-06T03:30:28.024 INFO:teuthology.orchestra.run.smithi110.stdout: "caps": { 2023-10-06T03:30:28.024 INFO:teuthology.orchestra.run.smithi110.stdout: "mgr": "allow profile osd", 2023-10-06T03:30:28.024 INFO:teuthology.orchestra.run.smithi110.stdout: "mon": "allow profile osd", 2023-10-06T03:30:28.024 INFO:teuthology.orchestra.run.smithi110.stdout: "osd": "allow *" 2023-10-06T03:30:28.024 INFO:teuthology.orchestra.run.smithi110.stdout: } 2023-10-06T03:30:28.024 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2023-10-06T03:30:28.024 INFO:teuthology.orchestra.run.smithi110.stdout: { 2023-10-06T03:30:28.024 INFO:teuthology.orchestra.run.smithi110.stdout: "entity": "client.0", 2023-10-06T03:30:28.024 INFO:teuthology.orchestra.run.smithi110.stdout: "key": "AQDnfh9lfTlDMhAAJ5dLa302KWyGhkB1sSMs8A==", 2023-10-06T03:30:28.024 INFO:teuthology.orchestra.run.smithi110.stdout: "caps": { 2023-10-06T03:30:28.024 INFO:teuthology.orchestra.run.smithi110.stdout: "mds": "allow", 2023-10-06T03:30:28.024 INFO:teuthology.orchestra.run.smithi110.stdout: "mgr": "allow r", 2023-10-06T03:30:28.024 INFO:teuthology.orchestra.run.smithi110.stdout: "mon": "allow rw", 2023-10-06T03:30:28.025 INFO:teuthology.orchestra.run.smithi110.stdout: "osd": "allow rwx" 2023-10-06T03:30:28.025 INFO:teuthology.orchestra.run.smithi110.stdout: } 2023-10-06T03:30:28.025 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2023-10-06T03:30:28.025 INFO:teuthology.orchestra.run.smithi110.stdout: { 2023-10-06T03:30:28.025 INFO:teuthology.orchestra.run.smithi110.stdout: "entity": "client.1", 2023-10-06T03:30:28.025 INFO:teuthology.orchestra.run.smithi110.stdout: "key": "AQDnfh9lurKGNxAALcTu8iRFmiO7SiFKaMHAOw==", 2023-10-06T03:30:28.025 INFO:teuthology.orchestra.run.smithi110.stdout: "caps": { 2023-10-06T03:30:28.025 INFO:teuthology.orchestra.run.smithi110.stdout: "mds": "allow", 2023-10-06T03:30:28.025 INFO:teuthology.orchestra.run.smithi110.stdout: "mgr": "allow r", 2023-10-06T03:30:28.025 INFO:teuthology.orchestra.run.smithi110.stdout: "mon": "allow rw", 2023-10-06T03:30:28.025 INFO:teuthology.orchestra.run.smithi110.stdout: "osd": "allow rwx" 2023-10-06T03:30:28.025 INFO:teuthology.orchestra.run.smithi110.stdout: } 2023-10-06T03:30:28.026 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2023-10-06T03:30:28.026 INFO:teuthology.orchestra.run.smithi110.stdout: { 2023-10-06T03:30:28.026 INFO:teuthology.orchestra.run.smithi110.stdout: "entity": "client.2", 2023-10-06T03:30:28.026 INFO:teuthology.orchestra.run.smithi110.stdout: "key": "AQDnfh9l42YJNRAAERQlQnUirB+3D3cl8QkkJg==", 2023-10-06T03:30:28.026 INFO:teuthology.orchestra.run.smithi110.stdout: "caps": { 2023-10-06T03:30:28.026 INFO:teuthology.orchestra.run.smithi110.stdout: "mds": "allow", 2023-10-06T03:30:28.026 INFO:teuthology.orchestra.run.smithi110.stdout: "mgr": "allow r", 2023-10-06T03:30:28.026 INFO:teuthology.orchestra.run.smithi110.stdout: "mon": "allow rw", 2023-10-06T03:30:28.026 INFO:teuthology.orchestra.run.smithi110.stdout: "osd": "allow rwx" 2023-10-06T03:30:28.026 INFO:teuthology.orchestra.run.smithi110.stdout: } 2023-10-06T03:30:28.026 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2023-10-06T03:30:28.026 INFO:teuthology.orchestra.run.smithi110.stdout: { 2023-10-06T03:30:28.027 INFO:teuthology.orchestra.run.smithi110.stdout: "entity": "client.3", 2023-10-06T03:30:28.027 INFO:teuthology.orchestra.run.smithi110.stdout: "key": "AQDnfh9lNbT3ORAAr/6KGmfvwFrZzYzRjnuLuw==", 2023-10-06T03:30:28.027 INFO:teuthology.orchestra.run.smithi110.stdout: "caps": { 2023-10-06T03:30:28.027 INFO:teuthology.orchestra.run.smithi110.stdout: "mds": "allow", 2023-10-06T03:30:28.027 INFO:teuthology.orchestra.run.smithi110.stdout: "mgr": "allow r", 2023-10-06T03:30:28.027 INFO:teuthology.orchestra.run.smithi110.stdout: "mon": "allow rw", 2023-10-06T03:30:28.027 INFO:teuthology.orchestra.run.smithi110.stdout: "osd": "allow rwx" 2023-10-06T03:30:28.027 INFO:teuthology.orchestra.run.smithi110.stdout: } 2023-10-06T03:30:28.027 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2023-10-06T03:30:28.027 INFO:teuthology.orchestra.run.smithi110.stdout: { 2023-10-06T03:30:28.027 INFO:teuthology.orchestra.run.smithi110.stdout: "entity": "client.admin", 2023-10-06T03:30:28.028 INFO:teuthology.orchestra.run.smithi110.stdout: "key": "AQDnfh9lXo0yEhAA5HC6GpInujr0bCCkD93DvQ==", 2023-10-06T03:30:28.028 INFO:teuthology.orchestra.run.smithi110.stdout: "caps": { 2023-10-06T03:30:28.028 INFO:teuthology.orchestra.run.smithi110.stdout: "mds": "allow *", 2023-10-06T03:30:28.028 INFO:teuthology.orchestra.run.smithi110.stdout: "mgr": "allow *", 2023-10-06T03:30:28.028 INFO:teuthology.orchestra.run.smithi110.stdout: "mon": "allow *", 2023-10-06T03:30:28.028 INFO:teuthology.orchestra.run.smithi110.stdout: "osd": "allow *" 2023-10-06T03:30:28.028 INFO:teuthology.orchestra.run.smithi110.stdout: } 2023-10-06T03:30:28.028 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2023-10-06T03:30:28.028 INFO:teuthology.orchestra.run.smithi110.stdout: { 2023-10-06T03:30:28.028 INFO:teuthology.orchestra.run.smithi110.stdout: "entity": "client.bootstrap-mds", 2023-10-06T03:30:28.028 INFO:teuthology.orchestra.run.smithi110.stdout: "key": "AQAGfx9lCmYJHxAAy1LwA3HXuKL3sUW2r2enGw==", 2023-10-06T03:30:28.028 INFO:teuthology.orchestra.run.smithi110.stdout: "caps": { 2023-10-06T03:30:28.029 INFO:teuthology.orchestra.run.smithi110.stdout: "mon": "allow profile bootstrap-mds" 2023-10-06T03:30:28.029 INFO:teuthology.orchestra.run.smithi110.stdout: } 2023-10-06T03:30:28.029 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2023-10-06T03:30:28.029 INFO:teuthology.orchestra.run.smithi110.stdout: { 2023-10-06T03:30:28.029 INFO:teuthology.orchestra.run.smithi110.stdout: "entity": "client.bootstrap-mgr", 2023-10-06T03:30:28.029 INFO:teuthology.orchestra.run.smithi110.stdout: "key": "AQAGfx9ldHwJHxAA6Oi2wUVC1lgKHsTV52ikJg==", 2023-10-06T03:30:28.029 INFO:teuthology.orchestra.run.smithi110.stdout: "caps": { 2023-10-06T03:30:28.029 INFO:teuthology.orchestra.run.smithi110.stdout: "mon": "allow profile bootstrap-mgr" 2023-10-06T03:30:28.029 INFO:teuthology.orchestra.run.smithi110.stdout: } 2023-10-06T03:30:28.029 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2023-10-06T03:30:28.030 INFO:teuthology.orchestra.run.smithi110.stdout: { 2023-10-06T03:30:28.030 INFO:teuthology.orchestra.run.smithi110.stdout: "entity": "client.bootstrap-osd", 2023-10-06T03:30:28.030 INFO:teuthology.orchestra.run.smithi110.stdout: "key": "AQAGfx9l5o8JHxAAle0qAKJL3abWW0MBAi/XXg==", 2023-10-06T03:30:28.030 INFO:teuthology.orchestra.run.smithi110.stdout: "caps": { 2023-10-06T03:30:28.030 INFO:teuthology.orchestra.run.smithi110.stdout: "mon": "allow profile bootstrap-osd" 2023-10-06T03:30:28.030 INFO:teuthology.orchestra.run.smithi110.stdout: } 2023-10-06T03:30:28.030 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2023-10-06T03:30:28.030 INFO:teuthology.orchestra.run.smithi110.stdout: { 2023-10-06T03:30:28.030 INFO:teuthology.orchestra.run.smithi110.stdout: "entity": "client.bootstrap-rbd", 2023-10-06T03:30:28.031 INFO:teuthology.orchestra.run.smithi110.stdout: "key": "AQAGfx9lxaAJHxAAzbjqqPOD/5m48i7ttf5B8w==", 2023-10-06T03:30:28.031 INFO:teuthology.orchestra.run.smithi110.stdout: "caps": { 2023-10-06T03:30:28.031 INFO:teuthology.orchestra.run.smithi110.stdout: "mon": "allow profile bootstrap-rbd" 2023-10-06T03:30:28.031 INFO:teuthology.orchestra.run.smithi110.stdout: } 2023-10-06T03:30:28.031 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2023-10-06T03:30:28.031 INFO:teuthology.orchestra.run.smithi110.stdout: { 2023-10-06T03:30:28.031 INFO:teuthology.orchestra.run.smithi110.stdout: "entity": "client.bootstrap-rbd-mirror", 2023-10-06T03:30:28.031 INFO:teuthology.orchestra.run.smithi110.stdout: "key": "AQAGfx9l9bEJHxAA0NZBfufkeCDfBRcvHDk4JQ==", 2023-10-06T03:30:28.032 INFO:teuthology.orchestra.run.smithi110.stdout: "caps": { 2023-10-06T03:30:28.032 INFO:teuthology.orchestra.run.smithi110.stdout: "mon": "allow profile bootstrap-rbd-mirror" 2023-10-06T03:30:28.032 INFO:teuthology.orchestra.run.smithi110.stdout: } 2023-10-06T03:30:28.032 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2023-10-06T03:30:28.032 INFO:teuthology.orchestra.run.smithi110.stdout: { 2023-10-06T03:30:28.032 INFO:teuthology.orchestra.run.smithi110.stdout: "entity": "client.bootstrap-rgw", 2023-10-06T03:30:28.032 INFO:teuthology.orchestra.run.smithi110.stdout: "key": "AQAGfx9l0sMJHxAA3uGAXQkwIzH2XxxzVOZn2Q==", 2023-10-06T03:30:28.033 INFO:teuthology.orchestra.run.smithi110.stdout: "caps": { 2023-10-06T03:30:28.033 INFO:teuthology.orchestra.run.smithi110.stdout: "mon": "allow profile bootstrap-rgw" 2023-10-06T03:30:28.033 INFO:teuthology.orchestra.run.smithi110.stdout: } 2023-10-06T03:30:28.033 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2023-10-06T03:30:28.033 INFO:teuthology.orchestra.run.smithi110.stdout: { 2023-10-06T03:30:28.033 INFO:teuthology.orchestra.run.smithi110.stdout: "entity": "mgr.x", 2023-10-06T03:30:28.033 INFO:teuthology.orchestra.run.smithi110.stdout: "key": "AQDnfh9lleAUHxAAPxyJMFMFjicyyb9MOxspEw==", 2023-10-06T03:30:28.034 INFO:teuthology.orchestra.run.smithi110.stdout: "caps": { 2023-10-06T03:30:28.034 INFO:teuthology.orchestra.run.smithi110.stdout: "mds": "allow *", 2023-10-06T03:30:28.034 INFO:teuthology.orchestra.run.smithi110.stdout: "mon": "allow profile mgr", 2023-10-06T03:30:28.034 INFO:teuthology.orchestra.run.smithi110.stdout: "osd": "allow *" 2023-10-06T03:30:28.034 INFO:teuthology.orchestra.run.smithi110.stdout: } 2023-10-06T03:30:28.034 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2023-10-06T03:30:28.034 INFO:teuthology.orchestra.run.smithi110.stdout: { 2023-10-06T03:30:28.034 INFO:teuthology.orchestra.run.smithi110.stdout: "entity": "mgr.y", 2023-10-06T03:30:28.035 INFO:teuthology.orchestra.run.smithi110.stdout: "key": "AQDnfh9lH5IzHBAAM72W1T6GBAluxB2R2Gcqtg==", 2023-10-06T03:30:28.035 INFO:teuthology.orchestra.run.smithi110.stdout: "caps": { 2023-10-06T03:30:28.035 INFO:teuthology.orchestra.run.smithi110.stdout: "mds": "allow *", 2023-10-06T03:30:28.035 INFO:teuthology.orchestra.run.smithi110.stdout: "mon": "allow profile mgr", 2023-10-06T03:30:28.035 INFO:teuthology.orchestra.run.smithi110.stdout: "osd": "allow *" 2023-10-06T03:30:28.035 INFO:teuthology.orchestra.run.smithi110.stdout: } 2023-10-06T03:30:28.035 INFO:teuthology.orchestra.run.smithi110.stdout: } 2023-10-06T03:30:28.035 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2023-10-06T03:30:28.036 INFO:teuthology.orchestra.run.smithi110.stdout:} 2023-10-06T03:30:28.038 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:28.009+0000 7fe1a0909700 1 -- 172.21.15.110:0/1342041360 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fe188060580 msgr2=0x7fe188062a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:28.038 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:28.009+0000 7fe1a0909700 1 --2- 172.21.15.110:0/1342041360 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fe188060580 0x7fe188062a30 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fe190006f00 tx=0x7fe190008040 comp rx=0 tx=0).stop 2023-10-06T03:30:28.038 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:28.009+0000 7fe1a0909700 1 -- 172.21.15.110:0/1342041360 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe19c156f50 msgr2=0x7fe19c0c1300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:28.038 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:28.009+0000 7fe1a0909700 1 --2- 172.21.15.110:0/1342041360 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe19c156f50 0x7fe19c0c1300 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7fe1940090b0 tx=0x7fe194017000 comp rx=0 tx=0).stop 2023-10-06T03:30:28.039 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:28.009+0000 7fe1a0909700 1 -- 172.21.15.110:0/1342041360 shutdown_connections 2023-10-06T03:30:28.039 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:28.009+0000 7fe1a0909700 1 --2- 172.21.15.110:0/1342041360 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fe188060580 0x7fe188062a30 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:28.039 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:28.009+0000 7fe1a0909700 1 --2- 172.21.15.110:0/1342041360 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe19c156f50 0x7fe19c0c1300 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:28.039 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:28.009+0000 7fe1a0909700 1 --2- 172.21.15.110:0/1342041360 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe19c156b80 0x7fe19c15ae30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:28.039 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:28.009+0000 7fe1a0909700 1 --2- 172.21.15.110:0/1342041360 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe19c0c6e10 0x7fe19c15a8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:28.039 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:28.009+0000 7fe1a0909700 1 -- 172.21.15.110:0/1342041360 >> 172.21.15.110:0/1342041360 conn(0x7fe19c0cc470 msgr2=0x7fe19c0b0e00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:30:28.039 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:28.009+0000 7fe1a0909700 1 -- 172.21.15.110:0/1342041360 shutdown_connections 2023-10-06T03:30:28.040 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:28.009+0000 7fe1a0909700 1 -- 172.21.15.110:0/1342041360 wait complete. 2023-10-06T03:30:28.040 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-mds 2023-10-06T03:30:28.147 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:28.145+0000 7fe18483f700 1 Processor -- start 2023-10-06T03:30:28.148 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:28.145+0000 7fe18483f700 1 -- start start 2023-10-06T03:30:28.149 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:28.145+0000 7fe18483f700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe18015ad60 0x7fe18015b130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:28.149 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:28.145+0000 7fe18483f700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fe1800adc30 0x7fe1800ae080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:28.149 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:28.145+0000 7fe18483f700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fe1800ab750 0x7fe1800abb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:28.149 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:28.145+0000 7fe18483f700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe18015aa10 con 0x7fe1800adc30 2023-10-06T03:30:28.149 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:28.145+0000 7fe18483f700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe1800695d0 con 0x7fe1800ab750 2023-10-06T03:30:28.149 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:28.145+0000 7fe18483f700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe18006c270 con 0x7fe18015ad60 2023-10-06T03:30:28.150 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:28.145+0000 7fe177fff700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fe1800adc30 0x7fe1800ae080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:28.150 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:28.145+0000 7fe177fff700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fe1800adc30 0x7fe1800ae080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:40186/0 (socket says 172.21.15.110:40186) 2023-10-06T03:30:28.150 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:28.145+0000 7fe177fff700 1 -- 172.21.15.110:0/1265628632 learned_addr learned my addr 172.21.15.110:0/1265628632 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:30:28.150 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:28.145+0000 7fe17f59e700 1 --2- 172.21.15.110:0/1265628632 >> v2:172.21.15.192:3300/0 conn(0x7fe1800ab750 0x7fe1800abb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:28.150 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:28.145+0000 7fe17ed9d700 1 --2- 172.21.15.110:0/1265628632 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe18015ad60 0x7fe18015b130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:28.150 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:28.145+0000 7fe177fff700 1 -- 172.21.15.110:0/1265628632 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe18015ad60 msgr2=0x7fe18015b130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:30:28.150 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:28.145+0000 7fe177fff700 1 --2- 172.21.15.110:0/1265628632 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe18015ad60 0x7fe18015b130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:28.150 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:28.145+0000 7fe177fff700 1 -- 172.21.15.110:0/1265628632 >> v2:172.21.15.192:3300/0 conn(0x7fe1800ab750 msgr2=0x7fe1800abb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:30:28.150 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:28.145+0000 7fe177fff700 1 --2- 172.21.15.110:0/1265628632 >> v2:172.21.15.192:3300/0 conn(0x7fe1800ab750 0x7fe1800abb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:28.150 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:28.145+0000 7fe177fff700 1 -- 172.21.15.110:0/1265628632 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe18006c550 con 0x7fe1800adc30 2023-10-06T03:30:28.151 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:28.145+0000 7fe17f59e700 1 --2- 172.21.15.110:0/1265628632 >> v2:172.21.15.192:3300/0 conn(0x7fe1800ab750 0x7fe1800abb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-06T03:30:28.151 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:28.145+0000 7fe177fff700 1 --2- 172.21.15.110:0/1265628632 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe1800adc30 0x7fe1800ae080 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fe1800c9580 tx=0x7fe168009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f2db2bf95adf2dde server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:30:28.152 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:28.145+0000 7fe17e59c700 1 -- 172.21.15.110:0/1265628632 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe16800b860 con 0x7fe1800adc30 2023-10-06T03:30:28.152 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:28.145+0000 7fe17e59c700 1 -- 172.21.15.110:0/1265628632 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe16800b9c0 con 0x7fe1800adc30 2023-10-06T03:30:28.152 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:28.145+0000 7fe17e59c700 1 -- 172.21.15.110:0/1265628632 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe16800bd80 con 0x7fe1800adc30 2023-10-06T03:30:28.152 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:28.145+0000 7fe18483f700 1 -- 172.21.15.110:0/1265628632 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe1800adc30 msgr2=0x7fe1800ae080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:28.152 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:28.145+0000 7fe18483f700 1 --2- 172.21.15.110:0/1265628632 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe1800adc30 0x7fe1800ae080 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fe1800c9580 tx=0x7fe168009250 comp rx=0 tx=0).stop 2023-10-06T03:30:28.152 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:28.145+0000 7fe18483f700 1 -- 172.21.15.110:0/1265628632 shutdown_connections 2023-10-06T03:30:28.152 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:28.145+0000 7fe18483f700 1 --2- 172.21.15.110:0/1265628632 >> v2:172.21.15.192:3300/0 conn(0x7fe1800ab750 0x7fe1800abb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:28.152 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:28.145+0000 7fe18483f700 1 --2- 172.21.15.110:0/1265628632 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe1800adc30 0x7fe1800ae080 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:28.153 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:28.145+0000 7fe18483f700 1 --2- 172.21.15.110:0/1265628632 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe18015ad60 0x7fe18015b130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:28.153 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:28.145+0000 7fe18483f700 1 -- 172.21.15.110:0/1265628632 >> 172.21.15.110:0/1265628632 conn(0x7fe18006e760 msgr2=0x7fe1800cba30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:30:28.153 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:28.145+0000 7fe18483f700 1 -- 172.21.15.110:0/1265628632 shutdown_connections 2023-10-06T03:30:28.153 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:28.145+0000 7fe18483f700 1 -- 172.21.15.110:0/1265628632 wait complete. 2023-10-06T03:30:28.153 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:28.149+0000 7fe18483f700 1 Processor -- start 2023-10-06T03:30:28.153 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:28.149+0000 7fe18483f700 1 -- start start 2023-10-06T03:30:28.155 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:28.149+0000 7fe18483f700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe1800ab750 0x7fe180159880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:28.155 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:28.149+0000 7fe18483f700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe1800adc30 0x7fe1800c6de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:28.155 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:28.149+0000 7fe18483f700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe18015ad60 0x7fe1800c7880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:28.155 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:28.149+0000 7fe18483f700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1801593f0 con 0x7fe1800ab750 2023-10-06T03:30:28.155 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:28.149+0000 7fe18483f700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1800c7320 con 0x7fe18015ad60 2023-10-06T03:30:28.155 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:28.149+0000 7fe18483f700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1801587e0 con 0x7fe1800adc30 2023-10-06T03:30:28.155 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:28.149+0000 7fe17ed9d700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe1800ab750 0x7fe180159880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:28.155 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:28.149+0000 7fe17ed9d700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe1800ab750 0x7fe180159880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:40194/0 (socket says 172.21.15.110:40194) 2023-10-06T03:30:28.155 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:28.149+0000 7fe17ed9d700 1 -- 172.21.15.110:0/1467041935 learned_addr learned my addr 172.21.15.110:0/1467041935 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:30:28.155 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:28.149+0000 7fe177fff700 1 --2- 172.21.15.110:0/1467041935 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe1800adc30 0x7fe1800c6de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:28.156 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:28.149+0000 7fe17f59e700 1 --2- 172.21.15.110:0/1467041935 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe18015ad60 0x7fe1800c7880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:28.157 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:28.149+0000 7fe17ed9d700 1 -- 172.21.15.110:0/1467041935 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe1800adc30 msgr2=0x7fe1800c6de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:28.157 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:28.149+0000 7fe17ed9d700 1 --2- 172.21.15.110:0/1467041935 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe1800adc30 0x7fe1800c6de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:28.157 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:28.149+0000 7fe17ed9d700 1 -- 172.21.15.110:0/1467041935 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe18015ad60 msgr2=0x7fe1800c7880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:28.157 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:28.149+0000 7fe17ed9d700 1 --2- 172.21.15.110:0/1467041935 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe18015ad60 0x7fe1800c7880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:28.157 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:28.149+0000 7fe17ed9d700 1 -- 172.21.15.110:0/1467041935 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1680094f0 con 0x7fe1800ab750 2023-10-06T03:30:28.157 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:28.149+0000 7fe17f59e700 1 --2- 172.21.15.110:0/1467041935 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe18015ad60 0x7fe1800c7880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-06T03:30:28.157 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:28.149+0000 7fe177fff700 1 --2- 172.21.15.110:0/1467041935 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe1800adc30 0x7fe1800c6de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-06T03:30:28.158 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:28.149+0000 7fe17ed9d700 1 --2- 172.21.15.110:0/1467041935 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe1800ab750 0x7fe180159880 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fe17000d860 tx=0x7fe170008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:30:28.158 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:28.149+0000 7fe176ffd700 1 -- 172.21.15.110:0/1467041935 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1700125f0 con 0x7fe1800ab750 2023-10-06T03:30:28.158 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:28.149+0000 7fe176ffd700 1 -- 172.21.15.110:0/1467041935 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe170012750 con 0x7fe1800ab750 2023-10-06T03:30:28.158 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:28.149+0000 7fe18483f700 1 -- 172.21.15.110:0/1467041935 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe180158ac0 con 0x7fe1800ab750 2023-10-06T03:30:28.158 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:28.149+0000 7fe176ffd700 1 -- 172.21.15.110:0/1467041935 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe170012b10 con 0x7fe1800ab750 2023-10-06T03:30:28.158 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:28.149+0000 7fe18483f700 1 -- 172.21.15.110:0/1467041935 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe180159080 con 0x7fe1800ab750 2023-10-06T03:30:28.159 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:28.153+0000 7fe18483f700 1 -- 172.21.15.110:0/1467041935 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe180070ff0 con 0x7fe1800ab750 2023-10-06T03:30:28.159 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:28.153+0000 7fe176ffd700 1 -- 172.21.15.110:0/1467041935 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7fe17001e460 con 0x7fe1800ab750 2023-10-06T03:30:28.159 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:28.153+0000 7fe176ffd700 1 --2- 172.21.15.110:0/1467041935 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fe160060580 0x7fe160062a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:28.159 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:28.153+0000 7fe176ffd700 1 -- 172.21.15.110:0/1467041935 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 5526+0+0 (secure 0 0 0) 0x7fe17007cdb0 con 0x7fe1800ab750 2023-10-06T03:30:28.159 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:28.153+0000 7fe177fff700 1 --2- 172.21.15.110:0/1467041935 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fe160060580 0x7fe160062a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:28.159 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:28.153+0000 7fe177fff700 1 --2- 172.21.15.110:0/1467041935 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fe160060580 0x7fe160062a30 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fe1800c9580 tx=0x7fe168017000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:30:28.165 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:28.161+0000 7fe176ffd700 1 -- 172.21.15.110:0/1467041935 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fe170050ee0 con 0x7fe1800ab750 2023-10-06T03:30:28.382 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:28.377+0000 7fe18483f700 1 -- 172.21.15.110:0/1467041935 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-mds"} v 0) v1 -- 0x7fe18006c830 con 0x7fe1800ab750 2023-10-06T03:30:28.390 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:28.385+0000 7fe176ffd700 1 -- 172.21.15.110:0/1467041935 <== mon.0 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-mds"}]=0 v3) v1 ==== 90+0+0 (secure 0 0 0) 0x7fe170050a70 con 0x7fe1800ab750 2023-10-06T03:30:28.393 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:28.389+0000 7fe18483f700 1 -- 172.21.15.110:0/1467041935 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fe160060580 msgr2=0x7fe160062a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:28.393 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:28.389+0000 7fe18483f700 1 --2- 172.21.15.110:0/1467041935 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fe160060580 0x7fe160062a30 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fe1800c9580 tx=0x7fe168017000 comp rx=0 tx=0).stop 2023-10-06T03:30:28.393 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:28.389+0000 7fe18483f700 1 -- 172.21.15.110:0/1467041935 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe1800ab750 msgr2=0x7fe180159880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:28.393 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:28.389+0000 7fe18483f700 1 --2- 172.21.15.110:0/1467041935 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe1800ab750 0x7fe180159880 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fe17000d860 tx=0x7fe170008040 comp rx=0 tx=0).stop 2023-10-06T03:30:28.393 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:28.389+0000 7fe18483f700 1 -- 172.21.15.110:0/1467041935 shutdown_connections 2023-10-06T03:30:28.393 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:28.389+0000 7fe18483f700 1 --2- 172.21.15.110:0/1467041935 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fe160060580 0x7fe160062a30 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:28.394 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:28.389+0000 7fe18483f700 1 --2- 172.21.15.110:0/1467041935 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe18015ad60 0x7fe1800c7880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:28.394 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:28.389+0000 7fe18483f700 1 --2- 172.21.15.110:0/1467041935 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe1800adc30 0x7fe1800c6de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:28.394 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:28.389+0000 7fe18483f700 1 --2- 172.21.15.110:0/1467041935 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe1800ab750 0x7fe180159880 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:28.394 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:28.389+0000 7fe18483f700 1 -- 172.21.15.110:0/1467041935 >> 172.21.15.110:0/1467041935 conn(0x7fe18006e760 msgr2=0x7fe1800ae8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:30:28.394 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:28.389+0000 7fe18483f700 1 -- 172.21.15.110:0/1467041935 shutdown_connections 2023-10-06T03:30:28.394 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:28.389+0000 7fe18483f700 1 -- 172.21.15.110:0/1467041935 wait complete. 2023-10-06T03:30:28.406 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-mgr 2023-10-06T03:30:28.514 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:28.509+0000 7fdd24d04700 1 Processor -- start 2023-10-06T03:30:28.514 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:28.509+0000 7fdd24d04700 1 -- start start 2023-10-06T03:30:28.516 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:28.513+0000 7fdd24d04700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fdd20155db0 0x7fdd2006c8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:28.516 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:28.513+0000 7fdd24d04700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fdd2006ce10 0x7fdd200c7f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:28.516 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:28.513+0000 7fdd24d04700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd200af020 con 0x7fdd201559e0 2023-10-06T03:30:28.516 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:28.513+0000 7fdd24d04700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdd2006d290 con 0x7fdd20155db0 2023-10-06T03:30:28.516 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:28.513+0000 7fdd24d04700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd200c7ad0 con 0x7fdd2006ce10 2023-10-06T03:30:28.516 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:28.513+0000 7fdd1f7fe700 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7fdd201559e0 0x7fdd200ae500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:56606/0 (socket says 172.21.15.110:56606) 2023-10-06T03:30:28.516 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:28.513+0000 7fdd1f7fe700 1 -- 172.21.15.110:0/2213110382 learned_addr learned my addr 172.21.15.110:0/2213110382 (peer_addr_for_me v1:172.21.15.110:0/0) 2023-10-06T03:30:28.516 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:28.513+0000 7fdd16ffd700 1 --2- 172.21.15.110:0/2213110382 >> v2:172.21.15.192:3300/0 conn(0x7fdd20155db0 0x7fdd2006c8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:28.516 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:28.513+0000 7fdd1ffff700 1 --2- 172.21.15.110:0/2213110382 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fdd2006ce10 0x7fdd200c7f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:28.518 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:28.513+0000 7fdd1effd700 1 -- 172.21.15.110:0/2213110382 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1551463122 0 0) 0x7fdd200af020 con 0x7fdd201559e0 2023-10-06T03:30:28.518 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:28.513+0000 7fdd1effd700 1 -- 172.21.15.110:0/2213110382 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdd0c003570 con 0x7fdd201559e0 2023-10-06T03:30:28.518 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:28.513+0000 7fdd16ffd700 1 -- 172.21.15.110:0/2213110382 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fdd2006ce10 msgr2=0x7fdd200c7f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:30:28.518 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:28.513+0000 7fdd16ffd700 1 --2- 172.21.15.110:0/2213110382 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fdd2006ce10 0x7fdd200c7f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:28.518 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:28.513+0000 7fdd16ffd700 1 -- 172.21.15.110:0/2213110382 >> v1:172.21.15.110:6789/0 conn(0x7fdd201559e0 legacy=0x7fdd200ae500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:28.518 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:28.513+0000 7fdd16ffd700 1 -- 172.21.15.110:0/2213110382 --> v2:172.21.15.192:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd20156200 con 0x7fdd20155db0 2023-10-06T03:30:28.518 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:28.513+0000 7fdd1ffff700 1 --2- 172.21.15.110:0/2213110382 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fdd2006ce10 0x7fdd200c7f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-06T03:30:28.518 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:28.513+0000 7fdd16ffd700 1 --2- 172.21.15.110:0/2213110382 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdd20155db0 0x7fdd2006c8d0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fdd080098e0 tx=0x7fdd080091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c74407e8e9b9e13d server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:30:28.518 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:28.513+0000 7fdd1effd700 1 -- 172.21.15.110:0/2213110382 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd08011630 con 0x7fdd20155db0 2023-10-06T03:30:28.518 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:28.513+0000 7fdd1effd700 1 -- 172.21.15.110:0/2213110382 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd08011790 con 0x7fdd20155db0 2023-10-06T03:30:28.519 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:28.513+0000 7fdd1effd700 1 -- 172.21.15.110:0/2213110382 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd08011b30 con 0x7fdd20155db0 2023-10-06T03:30:28.519 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:28.513+0000 7fdd24d04700 1 -- 172.21.15.110:0/2213110382 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdd20155db0 msgr2=0x7fdd2006c8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:28.519 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:28.513+0000 7fdd24d04700 1 --2- 172.21.15.110:0/2213110382 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdd20155db0 0x7fdd2006c8d0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fdd080098e0 tx=0x7fdd080091a0 comp rx=0 tx=0).stop 2023-10-06T03:30:28.520 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:28.513+0000 7fdd24d04700 1 -- 172.21.15.110:0/2213110382 shutdown_connections 2023-10-06T03:30:28.520 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:28.513+0000 7fdd24d04700 1 --2- 172.21.15.110:0/2213110382 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fdd2006ce10 0x7fdd200c7f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:28.520 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:28.513+0000 7fdd24d04700 1 --2- 172.21.15.110:0/2213110382 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdd20155db0 0x7fdd2006c8d0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:28.520 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:28.513+0000 7fdd24d04700 1 -- 172.21.15.110:0/2213110382 >> 172.21.15.110:0/2213110382 conn(0x7fdd2006b0d0 msgr2=0x7fdd200670a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:30:28.520 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:28.513+0000 7fdd24d04700 1 -- 172.21.15.110:0/2213110382 shutdown_connections 2023-10-06T03:30:28.520 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:28.513+0000 7fdd24d04700 1 -- 172.21.15.110:0/2213110382 wait complete. 2023-10-06T03:30:28.522 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:28.517+0000 7fdd24d04700 1 Processor -- start 2023-10-06T03:30:28.522 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:28.517+0000 7fdd24d04700 1 -- start start 2023-10-06T03:30:28.524 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:28.521+0000 7fdd24d04700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdd2006ce10 0x7fdd200c0f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:28.524 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:28.521+0000 7fdd24d04700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdd201559e0 0x7fdd20153350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:28.524 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:28.521+0000 7fdd24d04700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fdd20155db0 0x7fdd200b15d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:28.524 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:28.521+0000 7fdd24d04700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd200ca730 con 0x7fdd2006ce10 2023-10-06T03:30:28.524 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:28.521+0000 7fdd24d04700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd200ca9d0 con 0x7fdd201559e0 2023-10-06T03:30:28.524 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:28.521+0000 7fdd24d04700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd200cacb0 con 0x7fdd20155db0 2023-10-06T03:30:28.524 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:28.521+0000 7fdd1f7fe700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdd2006ce10 0x7fdd200c0f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:28.524 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:28.521+0000 7fdd1f7fe700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdd2006ce10 0x7fdd200c0f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:40206/0 (socket says 172.21.15.110:40206) 2023-10-06T03:30:28.524 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:28.521+0000 7fdd1f7fe700 1 -- 172.21.15.110:0/2002795920 learned_addr learned my addr 172.21.15.110:0/2002795920 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:30:28.525 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:28.521+0000 7fdd1ffff700 1 --2- 172.21.15.110:0/2002795920 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fdd20155db0 0x7fdd200b15d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:28.525 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:28.521+0000 7fdd16ffd700 1 --2- 172.21.15.110:0/2002795920 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdd201559e0 0x7fdd20153350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:28.526 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:28.521+0000 7fdd1f7fe700 1 -- 172.21.15.110:0/2002795920 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fdd20155db0 msgr2=0x7fdd200b15d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:28.526 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:28.521+0000 7fdd1f7fe700 1 --2- 172.21.15.110:0/2002795920 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fdd20155db0 0x7fdd200b15d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:28.526 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:28.521+0000 7fdd1f7fe700 1 -- 172.21.15.110:0/2002795920 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdd201559e0 msgr2=0x7fdd20153350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:28.526 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:28.521+0000 7fdd1f7fe700 1 --2- 172.21.15.110:0/2002795920 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdd201559e0 0x7fdd20153350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:28.526 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:28.521+0000 7fdd1f7fe700 1 -- 172.21.15.110:0/2002795920 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd18009a20 con 0x7fdd2006ce10 2023-10-06T03:30:28.526 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:28.521+0000 7fdd1f7fe700 1 --2- 172.21.15.110:0/2002795920 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdd2006ce10 0x7fdd200c0f60 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fdd1000c7c0 tx=0x7fdd10007e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:30:28.527 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:28.521+0000 7fdd17fff700 1 -- 172.21.15.110:0/2002795920 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd10005b30 con 0x7fdd2006ce10 2023-10-06T03:30:28.527 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:28.521+0000 7fdd17fff700 1 -- 172.21.15.110:0/2002795920 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd10005cb0 con 0x7fdd2006ce10 2023-10-06T03:30:28.527 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:28.521+0000 7fdd17fff700 1 -- 172.21.15.110:0/2002795920 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd10007320 con 0x7fdd2006ce10 2023-10-06T03:30:28.527 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:28.521+0000 7fdd24d04700 1 -- 172.21.15.110:0/2002795920 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd08009420 con 0x7fdd2006ce10 2023-10-06T03:30:28.527 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:28.521+0000 7fdd24d04700 1 -- 172.21.15.110:0/2002795920 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd200cb270 con 0x7fdd2006ce10 2023-10-06T03:30:28.527 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:28.521+0000 7fdd24d04700 1 -- 172.21.15.110:0/2002795920 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd20070ff0 con 0x7fdd2006ce10 2023-10-06T03:30:28.527 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:28.525+0000 7fdd17fff700 1 -- 172.21.15.110:0/2002795920 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7fdd1001b3b0 con 0x7fdd2006ce10 2023-10-06T03:30:28.528 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:28.525+0000 7fdd17fff700 1 --2- 172.21.15.110:0/2002795920 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fdd00060550 0x7fdd00062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:28.528 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:28.525+0000 7fdd17fff700 1 -- 172.21.15.110:0/2002795920 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 5526+0+0 (secure 0 0 0) 0x7fdd1007bbb0 con 0x7fdd2006ce10 2023-10-06T03:30:28.528 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:28.525+0000 7fdd16ffd700 1 --2- 172.21.15.110:0/2002795920 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fdd00060550 0x7fdd00062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:28.529 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:28.525+0000 7fdd16ffd700 1 --2- 172.21.15.110:0/2002795920 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fdd00060550 0x7fdd00062a00 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fdd080091e0 tx=0x7fdd08017000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:30:28.534 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:28.529+0000 7fdd17fff700 1 -- 172.21.15.110:0/2002795920 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fdd1004c310 con 0x7fdd2006ce10 2023-10-06T03:30:28.760 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:28.757+0000 7fdd24d04700 1 -- 172.21.15.110:0/2002795920 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-mgr"} v 0) v1 -- 0x7fdd200c71a0 con 0x7fdd2006ce10 2023-10-06T03:30:28.768 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:28.765+0000 7fdd17fff700 1 -- 172.21.15.110:0/2002795920 <== mon.0 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-mgr"}]=0 v4) v1 ==== 90+0+0 (secure 0 0 0) 0x7fdd100170a0 con 0x7fdd2006ce10 2023-10-06T03:30:28.771 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:28.765+0000 7fdd24d04700 1 -- 172.21.15.110:0/2002795920 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fdd00060550 msgr2=0x7fdd00062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:28.771 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:28.765+0000 7fdd24d04700 1 --2- 172.21.15.110:0/2002795920 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fdd00060550 0x7fdd00062a00 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fdd080091e0 tx=0x7fdd08017000 comp rx=0 tx=0).stop 2023-10-06T03:30:28.771 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:28.765+0000 7fdd24d04700 1 -- 172.21.15.110:0/2002795920 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdd2006ce10 msgr2=0x7fdd200c0f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:28.771 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:28.765+0000 7fdd24d04700 1 --2- 172.21.15.110:0/2002795920 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdd2006ce10 0x7fdd200c0f60 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fdd1000c7c0 tx=0x7fdd10007e00 comp rx=0 tx=0).stop 2023-10-06T03:30:28.771 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:28.765+0000 7fdd24d04700 1 -- 172.21.15.110:0/2002795920 shutdown_connections 2023-10-06T03:30:28.771 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:28.765+0000 7fdd24d04700 1 --2- 172.21.15.110:0/2002795920 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fdd00060550 0x7fdd00062a00 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:28.771 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:28.765+0000 7fdd24d04700 1 --2- 172.21.15.110:0/2002795920 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fdd20155db0 0x7fdd200b15d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:28.772 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:28.765+0000 7fdd24d04700 1 --2- 172.21.15.110:0/2002795920 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdd201559e0 0x7fdd20153350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:28.772 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:28.765+0000 7fdd24d04700 1 --2- 172.21.15.110:0/2002795920 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdd2006ce10 0x7fdd200c0f60 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:28.772 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:28.765+0000 7fdd24d04700 1 -- 172.21.15.110:0/2002795920 >> 172.21.15.110:0/2002795920 conn(0x7fdd2006b0d0 msgr2=0x7fdd200c62f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:30:28.772 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:28.765+0000 7fdd24d04700 1 -- 172.21.15.110:0/2002795920 shutdown_connections 2023-10-06T03:30:28.772 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:28.765+0000 7fdd24d04700 1 -- 172.21.15.110:0/2002795920 wait complete. 2023-10-06T03:30:28.784 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-osd 2023-10-06T03:30:28.896 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:28.893+0000 7ff9bcf07700 1 Processor -- start 2023-10-06T03:30:28.896 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:28.893+0000 7ff9bcf07700 1 -- start start 2023-10-06T03:30:28.897 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:28.893+0000 7ff9bcf07700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7ff9b806c9e0 0x7ff9b806a880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:28.897 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:28.893+0000 7ff9bcf07700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9b80c7440 con 0x7ff9b8069640 2023-10-06T03:30:28.897 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:28.893+0000 7ff9bcf07700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9b80ab990 con 0x7ff9b80ace60 2023-10-06T03:30:28.897 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:28.893+0000 7ff9bcf07700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff9b80bf3b0 con 0x7ff9b806c9e0 2023-10-06T03:30:28.897 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:28.893+0000 7ff9b7fff700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7ff9b806c9e0 0x7ff9b806a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:28.897 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:28.893+0000 7ff9b77fe700 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7ff9b8069640 0x7ff9b806b7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:56618/0 (socket says 172.21.15.110:56618) 2023-10-06T03:30:28.897 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:28.893+0000 7ff9b77fe700 1 -- 172.21.15.110:0/1856016903 learned_addr learned my addr 172.21.15.110:0/1856016903 (peer_addr_for_me v1:172.21.15.110:0/0) 2023-10-06T03:30:28.899 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:28.893+0000 7ff9b67fc700 1 -- 172.21.15.110:0/1856016903 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3310280870 0 0) 0x7ff9b80c7440 con 0x7ff9b8069640 2023-10-06T03:30:28.899 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:28.893+0000 7ff9b67fc700 1 -- 172.21.15.110:0/1856016903 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff9a0003570 con 0x7ff9b8069640 2023-10-06T03:30:28.899 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:28.893+0000 7ff9b67fc700 1 -- 172.21.15.110:0/1856016903 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2344210864 0 0) 0x7ff9b80ab990 con 0x7ff9b80ace60 2023-10-06T03:30:28.899 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:28.893+0000 7ff9b67fc700 1 -- 172.21.15.110:0/1856016903 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff9b80c7440 con 0x7ff9b80ace60 2023-10-06T03:30:28.900 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:28.893+0000 7ff9b7fff700 1 -- 172.21.15.110:0/1856016903 >> v1:172.21.15.192:6789/0 conn(0x7ff9b80ace60 legacy=0x7ff9b806bed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:28.900 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:28.893+0000 7ff9b7fff700 1 -- 172.21.15.110:0/1856016903 >> v1:172.21.15.110:6789/0 conn(0x7ff9b8069640 legacy=0x7ff9b806b7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:28.900 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:28.893+0000 7ff9b7fff700 1 -- 172.21.15.110:0/1856016903 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9b806c5e0 con 0x7ff9b806c9e0 2023-10-06T03:30:28.900 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:28.893+0000 7ff9b7fff700 1 --2- 172.21.15.110:0/1856016903 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff9b806c9e0 0x7ff9b806a880 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7ff9a80098e0 tx=0x7ff9a80091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e6128545fbb4384 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:30:28.900 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:28.893+0000 7ff9b67fc700 1 -- 172.21.15.110:0/1856016903 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9a8011630 con 0x7ff9b806c9e0 2023-10-06T03:30:28.900 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:28.893+0000 7ff9b67fc700 1 -- 172.21.15.110:0/1856016903 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9a8011790 con 0x7ff9b806c9e0 2023-10-06T03:30:28.900 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:28.893+0000 7ff9b67fc700 1 -- 172.21.15.110:0/1856016903 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9a8011b30 con 0x7ff9b806c9e0 2023-10-06T03:30:28.900 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:28.893+0000 7ff9bcf07700 1 -- 172.21.15.110:0/1856016903 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff9b806c9e0 msgr2=0x7ff9b806a880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:28.900 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:28.893+0000 7ff9bcf07700 1 --2- 172.21.15.110:0/1856016903 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff9b806c9e0 0x7ff9b806a880 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7ff9a80098e0 tx=0x7ff9a80091a0 comp rx=0 tx=0).stop 2023-10-06T03:30:28.900 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:28.893+0000 7ff9bcf07700 1 -- 172.21.15.110:0/1856016903 shutdown_connections 2023-10-06T03:30:28.900 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:28.893+0000 7ff9bcf07700 1 --2- 172.21.15.110:0/1856016903 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff9b806c9e0 0x7ff9b806a880 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:28.901 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:28.893+0000 7ff9bcf07700 1 -- 172.21.15.110:0/1856016903 >> 172.21.15.110:0/1856016903 conn(0x7ff9b806e760 msgr2=0x7ff9b80cba30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:30:28.901 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:28.897+0000 7ff9bcf07700 1 -- 172.21.15.110:0/1856016903 shutdown_connections 2023-10-06T03:30:28.901 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:28.897+0000 7ff9bcf07700 1 -- 172.21.15.110:0/1856016903 wait complete. 2023-10-06T03:30:28.903 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:28.897+0000 7ff9bcf07700 1 Processor -- start 2023-10-06T03:30:28.903 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:28.897+0000 7ff9bcf07700 1 -- start start 2023-10-06T03:30:28.903 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:28.897+0000 7ff9bcf07700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff9b8069640 0x7ff9b80c2260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:28.904 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:28.897+0000 7ff9bcf07700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff9b806c9e0 0x7ff9b8151e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:28.904 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:28.897+0000 7ff9bcf07700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff9b80ace60 0x7ff9b80c4440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:28.904 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:28.897+0000 7ff9bcf07700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9b80c1490 con 0x7ff9b806c9e0 2023-10-06T03:30:28.904 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:28.897+0000 7ff9bcf07700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9b815e990 con 0x7ff9b8069640 2023-10-06T03:30:28.904 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:28.897+0000 7ff9bcf07700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9b80c0880 con 0x7ff9b80ace60 2023-10-06T03:30:28.904 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:28.897+0000 7ff9b6ffd700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff9b806c9e0 0x7ff9b8151e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:28.904 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:28.897+0000 7ff9b6ffd700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff9b806c9e0 0x7ff9b8151e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:40234/0 (socket says 172.21.15.110:40234) 2023-10-06T03:30:28.904 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:28.897+0000 7ff9b6ffd700 1 -- 172.21.15.110:0/2732091811 learned_addr learned my addr 172.21.15.110:0/2732091811 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:30:28.904 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:28.897+0000 7ff9b77fe700 1 --2- 172.21.15.110:0/2732091811 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff9b8069640 0x7ff9b80c2260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:28.904 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:28.897+0000 7ff9b7fff700 1 --2- 172.21.15.110:0/2732091811 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff9b80ace60 0x7ff9b80c4440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:28.904 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:28.897+0000 7ff9b6ffd700 1 -- 172.21.15.110:0/2732091811 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff9b80ace60 msgr2=0x7ff9b80c4440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:28.905 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:28.897+0000 7ff9b6ffd700 1 --2- 172.21.15.110:0/2732091811 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff9b80ace60 0x7ff9b80c4440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:28.905 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:28.897+0000 7ff9b6ffd700 1 -- 172.21.15.110:0/2732091811 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff9b8069640 msgr2=0x7ff9b80c2260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:28.905 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:28.897+0000 7ff9b6ffd700 1 --2- 172.21.15.110:0/2732091811 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff9b8069640 0x7ff9b80c2260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:28.905 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:28.897+0000 7ff9b6ffd700 1 -- 172.21.15.110:0/2732091811 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9a8009420 con 0x7ff9b806c9e0 2023-10-06T03:30:28.905 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:28.897+0000 7ff9b6ffd700 1 --2- 172.21.15.110:0/2732091811 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff9b806c9e0 0x7ff9b8151e80 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7ff9b000c930 tx=0x7ff9b0002780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:30:28.905 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:28.897+0000 7ff99f7fe700 1 -- 172.21.15.110:0/2732091811 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9b000a7a0 con 0x7ff9b806c9e0 2023-10-06T03:30:28.905 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:28.897+0000 7ff99f7fe700 1 -- 172.21.15.110:0/2732091811 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9b000a900 con 0x7ff9b806c9e0 2023-10-06T03:30:28.905 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:28.897+0000 7ff9bcf07700 1 -- 172.21.15.110:0/2732091811 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9b80c0b60 con 0x7ff9b806c9e0 2023-10-06T03:30:28.905 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:28.897+0000 7ff99f7fe700 1 -- 172.21.15.110:0/2732091811 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9b000acc0 con 0x7ff9b806c9e0 2023-10-06T03:30:28.905 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:28.897+0000 7ff9bcf07700 1 -- 172.21.15.110:0/2732091811 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9b80c1120 con 0x7ff9b806c9e0 2023-10-06T03:30:28.906 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:28.901+0000 7ff9bcf07700 1 -- 172.21.15.110:0/2732091811 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff9b80c1640 con 0x7ff9b806c9e0 2023-10-06T03:30:28.906 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:28.901+0000 7ff99f7fe700 1 -- 172.21.15.110:0/2732091811 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7ff9b0012630 con 0x7ff9b806c9e0 2023-10-06T03:30:28.906 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:28.901+0000 7ff99f7fe700 1 --2- 172.21.15.110:0/2732091811 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7ff9a4060550 0x7ff9a4062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:28.907 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:28.901+0000 7ff9b77fe700 1 --2- 172.21.15.110:0/2732091811 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7ff9a4060550 0x7ff9a4062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:28.907 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:28.901+0000 7ff99f7fe700 1 -- 172.21.15.110:0/2732091811 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 5526+0+0 (secure 0 0 0) 0x7ff9b007cb20 con 0x7ff9b806c9e0 2023-10-06T03:30:28.907 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:28.901+0000 7ff9b77fe700 1 --2- 172.21.15.110:0/2732091811 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7ff9a4060550 0x7ff9a4062a00 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7ff9ac0058d0 tx=0x7ff9ac009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:30:28.912 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:28.909+0000 7ff99f7fe700 1 -- 172.21.15.110:0/2732091811 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7ff9b80c1640 con 0x7ff9b806c9e0 2023-10-06T03:30:29.131 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:29.129+0000 7ff9bcf07700 1 -- 172.21.15.110:0/2732091811 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-osd"} v 0) v1 -- 0x7ff9b80c1640 con 0x7ff9b806c9e0 2023-10-06T03:30:29.140 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:29.137+0000 7ff99f7fe700 1 -- 172.21.15.110:0/2732091811 <== mon.0 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-osd"}]=0 v5) v1 ==== 90+0+0 (secure 0 0 0) 0x7ff9b80c1640 con 0x7ff9b806c9e0 2023-10-06T03:30:29.144 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:29.141+0000 7ff9bcf07700 1 -- 172.21.15.110:0/2732091811 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7ff9a4060550 msgr2=0x7ff9a4062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:29.144 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:29.141+0000 7ff9bcf07700 1 --2- 172.21.15.110:0/2732091811 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7ff9a4060550 0x7ff9a4062a00 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7ff9ac0058d0 tx=0x7ff9ac009040 comp rx=0 tx=0).stop 2023-10-06T03:30:29.144 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:29.141+0000 7ff9bcf07700 1 -- 172.21.15.110:0/2732091811 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff9b806c9e0 msgr2=0x7ff9b8151e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:29.145 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:29.141+0000 7ff9bcf07700 1 --2- 172.21.15.110:0/2732091811 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff9b806c9e0 0x7ff9b8151e80 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7ff9b000c930 tx=0x7ff9b0002780 comp rx=0 tx=0).stop 2023-10-06T03:30:29.145 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:29.141+0000 7ff9bcf07700 1 -- 172.21.15.110:0/2732091811 shutdown_connections 2023-10-06T03:30:29.145 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:29.141+0000 7ff9bcf07700 1 --2- 172.21.15.110:0/2732091811 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7ff9a4060550 0x7ff9a4062a00 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:29.145 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:29.141+0000 7ff9bcf07700 1 --2- 172.21.15.110:0/2732091811 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff9b80ace60 0x7ff9b80c4440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:29.145 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:29.141+0000 7ff9bcf07700 1 --2- 172.21.15.110:0/2732091811 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff9b806c9e0 0x7ff9b8151e80 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:29.145 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:29.141+0000 7ff9bcf07700 1 --2- 172.21.15.110:0/2732091811 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff9b8069640 0x7ff9b80c2260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:29.145 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:29.141+0000 7ff9bcf07700 1 -- 172.21.15.110:0/2732091811 >> 172.21.15.110:0/2732091811 conn(0x7ff9b806e760 msgr2=0x7ff9b815edb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:30:29.145 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:29.141+0000 7ff9bcf07700 1 -- 172.21.15.110:0/2732091811 shutdown_connections 2023-10-06T03:30:29.145 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:29.141+0000 7ff9bcf07700 1 -- 172.21.15.110:0/2732091811 wait complete. 2023-10-06T03:30:29.159 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rbd 2023-10-06T03:30:29.266 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:29.261+0000 7fe94119b700 1 Processor -- start 2023-10-06T03:30:29.266 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:29.261+0000 7fe94119b700 1 -- start start 2023-10-06T03:30:29.268 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:29.265+0000 7fe94119b700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe93c0af5c0 0x7fe93c0cf570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:29.268 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:29.265+0000 7fe94119b700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fe93c0b1910 0x7fe93c06b7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:29.268 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:29.265+0000 7fe94119b700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe93c0afbf0 con 0x7fe93c0af1f0 2023-10-06T03:30:29.268 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:29.265+0000 7fe94119b700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe93c0cfe70 con 0x7fe93c0b1910 2023-10-06T03:30:29.268 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:29.265+0000 7fe94119b700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe93c0b0470 con 0x7fe93c0af5c0 2023-10-06T03:30:29.268 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:29.265+0000 7fe93b59e700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fe93c0b1910 0x7fe93c06b7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:29.268 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:29.265+0000 7fe93a59c700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe93c0af5c0 0x7fe93c0cf570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:29.268 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:29.265+0000 7fe93b59e700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fe93c0b1910 0x7fe93c06b7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.110:60112/0 (socket says 172.21.15.110:60112) 2023-10-06T03:30:29.268 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:29.265+0000 7fe93b59e700 1 -- 172.21.15.110:0/306390496 learned_addr learned my addr 172.21.15.110:0/306390496 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:30:29.269 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:29.265+0000 7fe93ad9d700 1 --1- 172.21.15.110:0/306390496 >> v1:172.21.15.192:6789/0 conn(0x7fe93c0af1f0 0x7fe93c0cee60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.110:36238/0 (socket says 172.21.15.110:36238) 2023-10-06T03:30:29.269 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:29.265+0000 7fe93b59e700 1 -- 172.21.15.110:0/306390496 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe93c0af5c0 msgr2=0x7fe93c0cf570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:30:29.269 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:29.265+0000 7fe93b59e700 1 --2- 172.21.15.110:0/306390496 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe93c0af5c0 0x7fe93c0cf570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:29.270 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:29.265+0000 7fe93b59e700 1 -- 172.21.15.110:0/306390496 >> v1:172.21.15.192:6789/0 conn(0x7fe93c0af1f0 legacy=0x7fe93c0cee60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:29.270 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:29.265+0000 7fe93b59e700 1 -- 172.21.15.110:0/306390496 --> v2:172.21.15.192:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe93c0cfb90 con 0x7fe93c0b1910 2023-10-06T03:30:29.270 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:29.265+0000 7fe93b59e700 1 --2- 172.21.15.110:0/306390496 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe93c0b1910 0x7fe93c06b7c0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fe934009f20 tx=0x7fe934009840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c969737c2e9734ee server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:30:29.270 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:29.265+0000 7fe939d9b700 1 -- 172.21.15.110:0/306390496 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe93400e070 con 0x7fe93c0b1910 2023-10-06T03:30:29.270 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:29.265+0000 7fe939d9b700 1 -- 172.21.15.110:0/306390496 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe93400b9d0 con 0x7fe93c0b1910 2023-10-06T03:30:29.270 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:29.265+0000 7fe939d9b700 1 -- 172.21.15.110:0/306390496 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9340092e0 con 0x7fe93c0b1910 2023-10-06T03:30:29.271 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:29.265+0000 7fe94119b700 1 -- 172.21.15.110:0/306390496 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe93c0b1910 msgr2=0x7fe93c06b7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:29.271 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:29.265+0000 7fe94119b700 1 --2- 172.21.15.110:0/306390496 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe93c0b1910 0x7fe93c06b7c0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fe934009f20 tx=0x7fe934009840 comp rx=0 tx=0).stop 2023-10-06T03:30:29.271 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:29.265+0000 7fe94119b700 1 -- 172.21.15.110:0/306390496 shutdown_connections 2023-10-06T03:30:29.271 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:29.265+0000 7fe94119b700 1 --2- 172.21.15.110:0/306390496 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe93c0b1910 0x7fe93c06b7c0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:29.271 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:29.265+0000 7fe94119b700 1 --2- 172.21.15.110:0/306390496 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe93c0af5c0 0x7fe93c0cf570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:29.272 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:29.265+0000 7fe94119b700 1 -- 172.21.15.110:0/306390496 >> 172.21.15.110:0/306390496 conn(0x7fe93c087ac0 msgr2=0x7fe93c0d0050 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:30:29.272 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:29.265+0000 7fe94119b700 1 -- 172.21.15.110:0/306390496 shutdown_connections 2023-10-06T03:30:29.272 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:29.265+0000 7fe94119b700 1 -- 172.21.15.110:0/306390496 wait complete. 2023-10-06T03:30:29.272 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:29.269+0000 7fe94119b700 1 Processor -- start 2023-10-06T03:30:29.272 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:29.269+0000 7fe94119b700 1 -- start start 2023-10-06T03:30:29.275 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:29.269+0000 7fe94119b700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe93c0af1f0 0x7fe93c152630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:29.275 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:29.269+0000 7fe94119b700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe93c0af5c0 0x7fe93c1542d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:29.275 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:29.269+0000 7fe94119b700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe93c0b1910 0x7fe93c154810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:29.275 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:29.269+0000 7fe94119b700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe93c154d50 con 0x7fe93c0b1910 2023-10-06T03:30:29.276 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:29.269+0000 7fe94119b700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe93c16a370 con 0x7fe93c0af1f0 2023-10-06T03:30:29.276 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:29.269+0000 7fe94119b700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe93c14eb10 con 0x7fe93c0af5c0 2023-10-06T03:30:29.276 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:29.269+0000 7fe93b59e700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe93c0b1910 0x7fe93c154810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:29.276 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:29.269+0000 7fe93a59c700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe93c0af5c0 0x7fe93c1542d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:29.276 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:29.269+0000 7fe93ad9d700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe93c0af1f0 0x7fe93c152630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:29.276 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:29.269+0000 7fe93b59e700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe93c0b1910 0x7fe93c154810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:40238/0 (socket says 172.21.15.110:40238) 2023-10-06T03:30:29.276 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:29.269+0000 7fe93b59e700 1 -- 172.21.15.110:0/4158313022 learned_addr learned my addr 172.21.15.110:0/4158313022 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:30:29.277 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:29.269+0000 7fe93b59e700 1 -- 172.21.15.110:0/4158313022 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe93c0af5c0 msgr2=0x7fe93c1542d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:29.277 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:29.269+0000 7fe93b59e700 1 --2- 172.21.15.110:0/4158313022 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe93c0af5c0 0x7fe93c1542d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:29.277 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:29.269+0000 7fe93b59e700 1 -- 172.21.15.110:0/4158313022 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe93c0af1f0 msgr2=0x7fe93c152630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:29.277 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:29.269+0000 7fe93b59e700 1 --2- 172.21.15.110:0/4158313022 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe93c0af1f0 0x7fe93c152630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:29.277 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:29.269+0000 7fe93b59e700 1 -- 172.21.15.110:0/4158313022 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe934009ae0 con 0x7fe93c0b1910 2023-10-06T03:30:29.277 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:29.269+0000 7fe93b59e700 1 --2- 172.21.15.110:0/4158313022 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe93c0b1910 0x7fe93c154810 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fe934009430 tx=0x7fe934018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:30:29.277 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:29.269+0000 7fe932ffd700 1 -- 172.21.15.110:0/4158313022 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe934007af0 con 0x7fe93c0b1910 2023-10-06T03:30:29.277 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:29.269+0000 7fe94119b700 1 -- 172.21.15.110:0/4158313022 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe93c155250 con 0x7fe93c0b1910 2023-10-06T03:30:29.277 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:29.269+0000 7fe932ffd700 1 -- 172.21.15.110:0/4158313022 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe934007c50 con 0x7fe93c0b1910 2023-10-06T03:30:29.277 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:29.269+0000 7fe932ffd700 1 -- 172.21.15.110:0/4158313022 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe93401d420 con 0x7fe93c0b1910 2023-10-06T03:30:29.277 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:29.269+0000 7fe94119b700 1 -- 172.21.15.110:0/4158313022 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe93c15ad10 con 0x7fe93c0b1910 2023-10-06T03:30:29.278 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:29.273+0000 7fe94119b700 1 -- 172.21.15.110:0/4158313022 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe93c06ff60 con 0x7fe93c0b1910 2023-10-06T03:30:29.278 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:29.273+0000 7fe932ffd700 1 -- 172.21.15.110:0/4158313022 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7fe934003040 con 0x7fe93c0b1910 2023-10-06T03:30:29.279 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:29.273+0000 7fe932ffd700 1 --2- 172.21.15.110:0/4158313022 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fe91c060550 0x7fe91c062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:29.279 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:29.273+0000 7fe932ffd700 1 -- 172.21.15.110:0/4158313022 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 5526+0+0 (secure 0 0 0) 0x7fe93407e500 con 0x7fe93c0b1910 2023-10-06T03:30:29.279 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:29.273+0000 7fe93ad9d700 1 --2- 172.21.15.110:0/4158313022 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fe91c060550 0x7fe91c062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:29.279 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:29.273+0000 7fe93ad9d700 1 --2- 172.21.15.110:0/4158313022 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fe91c060550 0x7fe91c062a00 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fe93c158670 tx=0x7fe924007eb0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:30:29.285 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:29.281+0000 7fe932ffd700 1 -- 172.21.15.110:0/4158313022 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fe93404eb00 con 0x7fe93c0b1910 2023-10-06T03:30:29.505 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:29.501+0000 7fe94119b700 1 -- 172.21.15.110:0/4158313022 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rbd"} v 0) v1 -- 0x7fe93c0afe20 con 0x7fe93c0b1910 2023-10-06T03:30:29.513 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:29.509+0000 7fe932ffd700 1 -- 172.21.15.110:0/4158313022 <== mon.0 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rbd"}]=0 v6) v1 ==== 90+0+0 (secure 0 0 0) 0x7fe934052120 con 0x7fe93c0b1910 2023-10-06T03:30:29.517 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:29.513+0000 7fe94119b700 1 -- 172.21.15.110:0/4158313022 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fe91c060550 msgr2=0x7fe91c062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:29.518 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:29.513+0000 7fe94119b700 1 --2- 172.21.15.110:0/4158313022 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fe91c060550 0x7fe91c062a00 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fe93c158670 tx=0x7fe924007eb0 comp rx=0 tx=0).stop 2023-10-06T03:30:29.518 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:29.513+0000 7fe94119b700 1 -- 172.21.15.110:0/4158313022 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe93c0b1910 msgr2=0x7fe93c154810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:29.518 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:29.513+0000 7fe94119b700 1 --2- 172.21.15.110:0/4158313022 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe93c0b1910 0x7fe93c154810 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fe934009430 tx=0x7fe934018000 comp rx=0 tx=0).stop 2023-10-06T03:30:29.518 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:29.513+0000 7fe94119b700 1 -- 172.21.15.110:0/4158313022 shutdown_connections 2023-10-06T03:30:29.518 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:29.513+0000 7fe94119b700 1 --2- 172.21.15.110:0/4158313022 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fe91c060550 0x7fe91c062a00 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:29.518 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:29.513+0000 7fe94119b700 1 --2- 172.21.15.110:0/4158313022 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe93c0b1910 0x7fe93c154810 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:29.518 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:29.513+0000 7fe94119b700 1 --2- 172.21.15.110:0/4158313022 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe93c0af5c0 0x7fe93c1542d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:29.518 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:29.513+0000 7fe94119b700 1 --2- 172.21.15.110:0/4158313022 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe93c0af1f0 0x7fe93c152630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:29.518 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:29.513+0000 7fe94119b700 1 -- 172.21.15.110:0/4158313022 >> 172.21.15.110:0/4158313022 conn(0x7fe93c087ac0 msgr2=0x7fe93c0c6550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:30:29.518 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:29.513+0000 7fe94119b700 1 -- 172.21.15.110:0/4158313022 shutdown_connections 2023-10-06T03:30:29.519 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:29.513+0000 7fe94119b700 1 -- 172.21.15.110:0/4158313022 wait complete. 2023-10-06T03:30:29.531 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rbd-mirror 2023-10-06T03:30:29.637 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:29.633+0000 7f3995c08700 1 Processor -- start 2023-10-06T03:30:29.637 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:29.633+0000 7f3995c08700 1 -- start start 2023-10-06T03:30:29.640 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:29.633+0000 7f3995c08700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f399006cfb0 0x7f39900cbd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:29.640 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:29.633+0000 7f3995c08700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39900caa40 con 0x7f3990158f30 2023-10-06T03:30:29.641 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:29.633+0000 7f3995c08700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39900cb8a0 con 0x7f3990158b60 2023-10-06T03:30:29.641 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:29.633+0000 7f3995c08700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f39900b3340 con 0x7f399006cfb0 2023-10-06T03:30:29.641 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:29.633+0000 7f398ffff700 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f3990158f30 0x7f399006c8a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:56626/0 (socket says 172.21.15.110:56626) 2023-10-06T03:30:29.641 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:29.633+0000 7f398ffff700 1 -- 172.21.15.110:0/3945888924 learned_addr learned my addr 172.21.15.110:0/3945888924 (peer_addr_for_me v1:172.21.15.110:0/0) 2023-10-06T03:30:29.641 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:29.633+0000 7f398f7fe700 1 -- 172.21.15.110:0/3945888924 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2940572692 0 0) 0x7f39900caa40 con 0x7f3990158f30 2023-10-06T03:30:29.641 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:29.633+0000 7f3995407700 1 --2- 172.21.15.110:0/3945888924 >> v2:172.21.15.192:3300/0 conn(0x7f399006cfb0 0x7f39900cbd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:29.641 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:29.633+0000 7f398f7fe700 1 -- 172.21.15.110:0/3945888924 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3978003570 con 0x7f3990158f30 2023-10-06T03:30:29.641 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:29.633+0000 7f398f7fe700 1 -- 172.21.15.110:0/3945888924 <== mon.0 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 181689981 0 0) 0x7f3978003570 con 0x7f3990158f30 2023-10-06T03:30:29.641 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:29.633+0000 7f398f7fe700 1 -- 172.21.15.110:0/3945888924 >> v2:172.21.15.192:3300/0 conn(0x7f399006cfb0 msgr2=0x7f39900cbd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:30:29.641 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:29.633+0000 7f398f7fe700 1 --2- 172.21.15.110:0/3945888924 >> v2:172.21.15.192:3300/0 conn(0x7f399006cfb0 0x7f39900cbd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:29.642 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:29.633+0000 7f398f7fe700 1 -- 172.21.15.110:0/3945888924 >> v1:172.21.15.192:6789/0 conn(0x7f3990158b60 legacy=0x7f39900b2c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:29.642 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:29.633+0000 7f398f7fe700 1 -- 172.21.15.110:0/3945888924 --> v1:172.21.15.110:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39900cac90 con 0x7f3990158f30 2023-10-06T03:30:29.642 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:29.633+0000 7f398f7fe700 1 -- 172.21.15.110:0/3945888924 <== mon.0 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 118603703 0 0) 0x7f3980001470 con 0x7f3990158f30 2023-10-06T03:30:29.642 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:29.633+0000 7f398f7fe700 1 -- 172.21.15.110:0/3945888924 >> v1:172.21.15.110:6789/0 conn(0x7f3990158f30 legacy=0x7f399006c8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:29.642 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:29.633+0000 7f398f7fe700 1 --2- 172.21.15.110:0/3945888924 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3978003a60 0x7f3978003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:29.642 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:29.633+0000 7f398f7fe700 1 --2- 172.21.15.110:0/3945888924 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3978004810 0x7f3978004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:29.642 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:29.633+0000 7f398f7fe700 1 --2- 172.21.15.110:0/3945888924 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f39780051c0 0x7f397800ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:29.642 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:29.633+0000 7f398f7fe700 1 -- 172.21.15.110:0/3945888924 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39900b3340 con 0x7f3978003a60 2023-10-06T03:30:29.642 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:29.633+0000 7f398f7fe700 1 -- 172.21.15.110:0/3945888924 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f397800c160 con 0x7f3978004810 2023-10-06T03:30:29.642 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:29.633+0000 7f398f7fe700 1 -- 172.21.15.110:0/3945888924 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f397800c320 con 0x7f39780051c0 2023-10-06T03:30:29.642 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:29.633+0000 7f3994c06700 1 --2- 172.21.15.110:0/3945888924 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3978003a60 0x7f3978003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:29.642 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:29.633+0000 7f3995407700 1 --2- 172.21.15.110:0/3945888924 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f39780051c0 0x7f397800ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:29.643 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:29.633+0000 7f398ffff700 1 --2- 172.21.15.110:0/3945888924 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3978004810 0x7f3978004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:29.643 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:29.637+0000 7f3994c06700 1 -- 172.21.15.110:0/3945888924 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f39780051c0 msgr2=0x7f397800ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:30:29.643 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:29.637+0000 7f3994c06700 1 --2- 172.21.15.110:0/3945888924 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f39780051c0 0x7f397800ba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:29.643 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:29.637+0000 7f3994c06700 1 -- 172.21.15.110:0/3945888924 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3978004810 msgr2=0x7f3978004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:30:29.643 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:29.637+0000 7f3994c06700 1 --2- 172.21.15.110:0/3945888924 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3978004810 0x7f3978004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:29.643 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:29.637+0000 7f3995407700 1 -- 172.21.15.110:0/3945888924 reap_dead start 2023-10-06T03:30:29.643 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:29.637+0000 7f3994c06700 1 -- 172.21.15.110:0/3945888924 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3978003570 con 0x7f3978003a60 2023-10-06T03:30:29.643 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:29.637+0000 7f398ffff700 1 --2- 172.21.15.110:0/3945888924 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3978004810 0x7f3978004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-06T03:30:29.643 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:29.637+0000 7f3994c06700 1 --2- 172.21.15.110:0/3945888924 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3978003a60 0x7f3978003e70 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f39840018b0 tx=0x7f3984007d30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e4e3d474bf96c418 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:30:29.643 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:29.637+0000 7f398f7fe700 1 -- 172.21.15.110:0/3945888924 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3984012420 con 0x7f3978003a60 2023-10-06T03:30:29.643 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:29.637+0000 7f398f7fe700 1 -- 172.21.15.110:0/3945888924 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39840125a0 con 0x7f3978003a60 2023-10-06T03:30:29.644 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:29.637+0000 7f398f7fe700 1 -- 172.21.15.110:0/3945888924 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3984012960 con 0x7f3978003a60 2023-10-06T03:30:29.644 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:29.637+0000 7f3995c08700 1 -- 172.21.15.110:0/3945888924 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3978003a60 msgr2=0x7f3978003e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:29.644 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:29.637+0000 7f3995c08700 1 --2- 172.21.15.110:0/3945888924 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3978003a60 0x7f3978003e70 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f39840018b0 tx=0x7f3984007d30 comp rx=0 tx=0).stop 2023-10-06T03:30:29.644 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:29.637+0000 7f3995c08700 1 -- 172.21.15.110:0/3945888924 shutdown_connections 2023-10-06T03:30:29.644 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:29.637+0000 7f3995c08700 1 --2- 172.21.15.110:0/3945888924 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3978003a60 0x7f3978003e70 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:29.644 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:29.637+0000 7f3995c08700 1 -- 172.21.15.110:0/3945888924 >> 172.21.15.110:0/3945888924 conn(0x7f39900d08f0 msgr2=0x7f39900cc8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:30:29.644 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:29.637+0000 7f3995c08700 1 -- 172.21.15.110:0/3945888924 shutdown_connections 2023-10-06T03:30:29.644 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:29.637+0000 7f3995c08700 1 -- 172.21.15.110:0/3945888924 wait complete. 2023-10-06T03:30:29.646 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:29.637+0000 7f3995c08700 1 Processor -- start 2023-10-06T03:30:29.647 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:29.637+0000 7f3995c08700 1 -- start start 2023-10-06T03:30:29.647 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:29.637+0000 7f3995c08700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3978003a60 0x7f39900c3970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:29.647 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:29.637+0000 7f3995c08700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f399015b050 0x7f39900c55c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:29.647 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:29.637+0000 7f3995c08700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3990151420 0x7f39900c5b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:29.647 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:29.637+0000 7f3995c08700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f399015b4c0 con 0x7f399015b050 2023-10-06T03:30:29.647 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:29.637+0000 7f3995c08700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f399015b7a0 con 0x7f3978003a60 2023-10-06T03:30:29.647 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:29.637+0000 7f3995c08700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f399015ba80 con 0x7f3990151420 2023-10-06T03:30:29.647 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:29.637+0000 7f398ffff700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f399015b050 0x7f39900c55c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:29.647 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:29.641+0000 7f3994c06700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3978003a60 0x7f39900c3970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:29.647 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:29.641+0000 7f398ffff700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f399015b050 0x7f39900c55c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:40258/0 (socket says 172.21.15.110:40258) 2023-10-06T03:30:29.648 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:29.641+0000 7f398ffff700 1 -- 172.21.15.110:0/3072518999 learned_addr learned my addr 172.21.15.110:0/3072518999 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:30:29.648 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:29.641+0000 7f3995407700 1 --2- 172.21.15.110:0/3072518999 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3990151420 0x7f39900c5b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:29.648 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:29.641+0000 7f398ffff700 1 -- 172.21.15.110:0/3072518999 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3990151420 msgr2=0x7f39900c5b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:29.648 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:29.641+0000 7f398ffff700 1 --2- 172.21.15.110:0/3072518999 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3990151420 0x7f39900c5b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:29.648 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:29.641+0000 7f398ffff700 1 -- 172.21.15.110:0/3072518999 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3978003a60 msgr2=0x7f39900c3970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:29.648 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:29.641+0000 7f398ffff700 1 --2- 172.21.15.110:0/3072518999 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3978003a60 0x7f39900c3970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:29.648 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:29.641+0000 7f398ffff700 1 -- 172.21.15.110:0/3072518999 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f399015bd60 con 0x7f399015b050 2023-10-06T03:30:29.648 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:29.641+0000 7f398ffff700 1 --2- 172.21.15.110:0/3072518999 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f399015b050 0x7f39900c55c0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f3980001c00 tx=0x7f3980001430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:30:29.648 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:29.641+0000 7f398cff9700 1 -- 172.21.15.110:0/3072518999 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3980005e30 con 0x7f399015b050 2023-10-06T03:30:29.648 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:29.641+0000 7f398cff9700 1 -- 172.21.15.110:0/3072518999 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f398000c5d0 con 0x7f399015b050 2023-10-06T03:30:29.648 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:29.641+0000 7f3995c08700 1 -- 172.21.15.110:0/3072518999 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39900c3eb0 con 0x7f399015b050 2023-10-06T03:30:29.649 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:29.641+0000 7f398cff9700 1 -- 172.21.15.110:0/3072518999 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f398000c990 con 0x7f399015b050 2023-10-06T03:30:29.649 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:29.641+0000 7f3995c08700 1 -- 172.21.15.110:0/3072518999 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39900c41b0 con 0x7f399015b050 2023-10-06T03:30:29.649 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:29.641+0000 7f3995c08700 1 -- 172.21.15.110:0/3072518999 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3990070ff0 con 0x7f399015b050 2023-10-06T03:30:29.649 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:29.641+0000 7f398cff9700 1 -- 172.21.15.110:0/3072518999 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7f398000cd40 con 0x7f399015b050 2023-10-06T03:30:29.649 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:29.645+0000 7f398cff9700 1 --2- 172.21.15.110:0/3072518999 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f397c060550 0x7f397c062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:29.650 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:29.645+0000 7f3994c06700 1 --2- 172.21.15.110:0/3072518999 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f397c060550 0x7f397c062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:29.650 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:29.645+0000 7f398cff9700 1 -- 172.21.15.110:0/3072518999 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 5526+0+0 (secure 0 0 0) 0x7f398007c4d0 con 0x7f399015b050 2023-10-06T03:30:29.650 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:29.645+0000 7f3994c06700 1 --2- 172.21.15.110:0/3072518999 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f397c060550 0x7f397c062a00 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f39840018b0 tx=0x7f3984015000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:30:29.654 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:29.649+0000 7f398cff9700 1 -- 172.21.15.110:0/3072518999 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f398004cb70 con 0x7f399015b050 2023-10-06T03:30:29.870 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:29.865+0000 7f3995c08700 1 -- 172.21.15.110:0/3072518999 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rbd-mirror"} v 0) v1 -- 0x7f39900c7380 con 0x7f399015b050 2023-10-06T03:30:29.879 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:29.873+0000 7f398cff9700 1 -- 172.21.15.110:0/3072518999 <== mon.0 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rbd-mirror"}]=0 v7) v1 ==== 97+0+0 (secure 0 0 0) 0x7f3980050190 con 0x7f399015b050 2023-10-06T03:30:29.882 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:29.877+0000 7f3995c08700 1 -- 172.21.15.110:0/3072518999 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f397c060550 msgr2=0x7f397c062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:29.882 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:29.877+0000 7f3995c08700 1 --2- 172.21.15.110:0/3072518999 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f397c060550 0x7f397c062a00 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f39840018b0 tx=0x7f3984015000 comp rx=0 tx=0).stop 2023-10-06T03:30:29.882 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:29.877+0000 7f3995c08700 1 -- 172.21.15.110:0/3072518999 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f399015b050 msgr2=0x7f39900c55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:29.882 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:29.877+0000 7f3995c08700 1 --2- 172.21.15.110:0/3072518999 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f399015b050 0x7f39900c55c0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f3980001c00 tx=0x7f3980001430 comp rx=0 tx=0).stop 2023-10-06T03:30:29.882 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:29.877+0000 7f3995c08700 1 -- 172.21.15.110:0/3072518999 shutdown_connections 2023-10-06T03:30:29.882 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:29.877+0000 7f3995c08700 1 --2- 172.21.15.110:0/3072518999 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f397c060550 0x7f397c062a00 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:29.883 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:29.877+0000 7f3995c08700 1 --2- 172.21.15.110:0/3072518999 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3990151420 0x7f39900c5b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:29.883 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:29.877+0000 7f3995c08700 1 --2- 172.21.15.110:0/3072518999 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f399015b050 0x7f39900c55c0 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:29.883 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:29.877+0000 7f3995c08700 1 --2- 172.21.15.110:0/3072518999 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3978003a60 0x7f39900c3970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:29.883 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:29.877+0000 7f3995c08700 1 -- 172.21.15.110:0/3072518999 >> 172.21.15.110:0/3072518999 conn(0x7f39900d08f0 msgr2=0x7f3990152400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:30:29.883 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:29.877+0000 7f3995c08700 1 -- 172.21.15.110:0/3072518999 shutdown_connections 2023-10-06T03:30:29.883 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:29.877+0000 7f3995c08700 1 -- 172.21.15.110:0/3072518999 wait complete. 2023-10-06T03:30:29.895 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rgw 2023-10-06T03:30:30.008 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.005+0000 7f968a507700 1 Processor -- start 2023-10-06T03:30:30.009 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.005+0000 7f968a507700 1 -- start start 2023-10-06T03:30:30.011 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.005+0000 7f968a507700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f968415cc80 0x7f96840c6420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:30.011 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.005+0000 7f968a507700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f968415d050 0x7f96840694a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:30.011 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.005+0000 7f968a507700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96840699e0 con 0x7f9684158990 2023-10-06T03:30:30.012 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.005+0000 7f968a507700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f968415d4c0 con 0x7f968415cc80 2023-10-06T03:30:30.012 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.005+0000 7f968a507700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f968406ace0 con 0x7f968415d050 2023-10-06T03:30:30.012 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.005+0000 7f9689505700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f968415cc80 0x7f96840c6420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:30.012 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.005+0000 7f9689505700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f968415cc80 0x7f96840c6420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:40270/0 (socket says 172.21.15.110:40270) 2023-10-06T03:30:30.012 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.005+0000 7f9689505700 1 -- 172.21.15.110:0/1785261276 learned_addr learned my addr 172.21.15.110:0/1785261276 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:30:30.012 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.005+0000 7f9688d04700 1 --2- 172.21.15.110:0/1785261276 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f968415d050 0x7f96840694a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:30.012 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.005+0000 7f9689505700 1 -- 172.21.15.110:0/1785261276 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f968415d050 msgr2=0x7f96840694a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:30:30.012 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.005+0000 7f9689505700 1 --2- 172.21.15.110:0/1785261276 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f968415d050 0x7f96840694a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:30.012 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.005+0000 7f9689505700 1 -- 172.21.15.110:0/1785261276 >> v1:172.21.15.192:6789/0 conn(0x7f9684158990 legacy=0x7f9684158dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:30.012 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.005+0000 7f9689505700 1 -- 172.21.15.110:0/1785261276 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9684069c40 con 0x7f968415cc80 2023-10-06T03:30:30.012 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.005+0000 7f9689505700 1 --2- 172.21.15.110:0/1785261276 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f968415cc80 0x7f96840c6420 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f96780099b0 tx=0x7f9678009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f19c1b1939016a31 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:30:30.013 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.005+0000 7f9683fff700 1 -- 172.21.15.110:0/1785261276 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9678009ea0 con 0x7f968415cc80 2023-10-06T03:30:30.013 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.005+0000 7f9683fff700 1 -- 172.21.15.110:0/1785261276 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f967800b860 con 0x7f968415cc80 2023-10-06T03:30:30.013 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.005+0000 7f9683fff700 1 -- 172.21.15.110:0/1785261276 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f967800bc20 con 0x7f968415cc80 2023-10-06T03:30:30.013 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.005+0000 7f968a507700 1 -- 172.21.15.110:0/1785261276 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f968415cc80 msgr2=0x7f96840c6420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:30.013 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.005+0000 7f968a507700 1 --2- 172.21.15.110:0/1785261276 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f968415cc80 0x7f96840c6420 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f96780099b0 tx=0x7f9678009250 comp rx=0 tx=0).stop 2023-10-06T03:30:30.013 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.009+0000 7f968a507700 1 -- 172.21.15.110:0/1785261276 shutdown_connections 2023-10-06T03:30:30.013 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.009+0000 7f968a507700 1 --2- 172.21.15.110:0/1785261276 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f968415d050 0x7f96840694a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:30.013 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.009+0000 7f968a507700 1 --2- 172.21.15.110:0/1785261276 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f968415cc80 0x7f96840c6420 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:30.013 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.009+0000 7f968a507700 1 -- 172.21.15.110:0/1785261276 >> 172.21.15.110:0/1785261276 conn(0x7f968406e760 msgr2=0x7f96840cba30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:30:30.013 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.009+0000 7f968a507700 1 -- 172.21.15.110:0/1785261276 shutdown_connections 2023-10-06T03:30:30.013 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.009+0000 7f968a507700 1 -- 172.21.15.110:0/1785261276 wait complete. 2023-10-06T03:30:30.015 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.009+0000 7f968a507700 1 Processor -- start 2023-10-06T03:30:30.016 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.009+0000 7f968a507700 1 -- start start 2023-10-06T03:30:30.016 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.009+0000 7f968a507700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9684158990 0x7f96841691d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:30.016 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.009+0000 7f968a507700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f968415cc80 0x7f96840ae8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:30.016 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.009+0000 7f968a507700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f968415d050 0x7f9684155690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:30.016 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.009+0000 7f968a507700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96840afd40 con 0x7f968415cc80 2023-10-06T03:30:30.016 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.009+0000 7f968a507700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96840af130 con 0x7f968415d050 2023-10-06T03:30:30.016 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.009+0000 7f968a507700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96840af410 con 0x7f9684158990 2023-10-06T03:30:30.016 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.009+0000 7f9688d04700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f968415cc80 0x7f96840ae8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:30.016 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.009+0000 7f9688d04700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f968415cc80 0x7f96840ae8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:40274/0 (socket says 172.21.15.110:40274) 2023-10-06T03:30:30.016 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.009+0000 7f9688d04700 1 -- 172.21.15.110:0/316689586 learned_addr learned my addr 172.21.15.110:0/316689586 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:30:30.016 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.009+0000 7f9689d06700 1 --2- 172.21.15.110:0/316689586 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f968415d050 0x7f9684155690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:30.017 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.009+0000 7f9689505700 1 --2- 172.21.15.110:0/316689586 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9684158990 0x7f96841691d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:30.017 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.009+0000 7f9688d04700 1 -- 172.21.15.110:0/316689586 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9684158990 msgr2=0x7f96841691d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:30.017 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.009+0000 7f9688d04700 1 --2- 172.21.15.110:0/316689586 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9684158990 0x7f96841691d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:30.017 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.009+0000 7f9688d04700 1 -- 172.21.15.110:0/316689586 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f968415d050 msgr2=0x7f9684155690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:30.017 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.009+0000 7f9688d04700 1 --2- 172.21.15.110:0/316689586 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f968415d050 0x7f9684155690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:30.017 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.009+0000 7f9688d04700 1 -- 172.21.15.110:0/316689586 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96780094f0 con 0x7f968415cc80 2023-10-06T03:30:30.017 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.009+0000 7f9689505700 1 --2- 172.21.15.110:0/316689586 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9684158990 0x7f96841691d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-06T03:30:30.017 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.009+0000 7f9688d04700 1 --2- 172.21.15.110:0/316689586 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f968415cc80 0x7f96840ae8e0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f967c009e50 tx=0x7f967c00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:30:30.017 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.009+0000 7f96817fa700 1 -- 172.21.15.110:0/316689586 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f967c00cb50 con 0x7f968415cc80 2023-10-06T03:30:30.017 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.009+0000 7f96817fa700 1 -- 172.21.15.110:0/316689586 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f967c00f040 con 0x7f968415cc80 2023-10-06T03:30:30.017 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.009+0000 7f968a507700 1 -- 172.21.15.110:0/316689586 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96840af6f0 con 0x7f968415cc80 2023-10-06T03:30:30.018 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.009+0000 7f96817fa700 1 -- 172.21.15.110:0/316689586 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f967c014480 con 0x7f968415cc80 2023-10-06T03:30:30.018 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.009+0000 7f968a507700 1 -- 172.21.15.110:0/316689586 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96840aee20 con 0x7f968415cc80 2023-10-06T03:30:30.019 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.013+0000 7f96817fa700 1 -- 172.21.15.110:0/316689586 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7f967c0147f0 con 0x7f968415cc80 2023-10-06T03:30:30.019 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.013+0000 7f968a507700 1 -- 172.21.15.110:0/316689586 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f96840afeb0 con 0x7f968415cc80 2023-10-06T03:30:30.019 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.013+0000 7f96817fa700 1 --2- 172.21.15.110:0/316689586 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f9670060550 0x7f9670062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:30.019 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.013+0000 7f96817fa700 1 -- 172.21.15.110:0/316689586 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 5526+0+0 (secure 0 0 0) 0x7f967c07d050 con 0x7f968415cc80 2023-10-06T03:30:30.019 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.013+0000 7f9689505700 1 --2- 172.21.15.110:0/316689586 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f9670060550 0x7f9670062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:30.019 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.013+0000 7f9689505700 1 --2- 172.21.15.110:0/316689586 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f9670060550 0x7f9670062a00 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f96780099b0 tx=0x7f9678017000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:30:30.026 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.021+0000 7f96817fa700 1 -- 172.21.15.110:0/316689586 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f967c04d6f0 con 0x7f968415cc80 2023-10-06T03:30:30.235 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.229+0000 7f968a507700 1 -- 172.21.15.110:0/316689586 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rgw"} v 0) v1 -- 0x7f9684070ff0 con 0x7f968415cc80 2023-10-06T03:30:30.242 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.237+0000 7f96817fa700 1 -- 172.21.15.110:0/316689586 <== mon.0 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rgw"}]=0 v8) v1 ==== 90+0+0 (secure 0 0 0) 0x7f967c050d10 con 0x7f968415cc80 2023-10-06T03:30:30.246 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.241+0000 7f968a507700 1 -- 172.21.15.110:0/316689586 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f9670060550 msgr2=0x7f9670062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:30.247 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.241+0000 7f968a507700 1 --2- 172.21.15.110:0/316689586 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f9670060550 0x7f9670062a00 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f96780099b0 tx=0x7f9678017000 comp rx=0 tx=0).stop 2023-10-06T03:30:30.247 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.241+0000 7f968a507700 1 -- 172.21.15.110:0/316689586 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f968415cc80 msgr2=0x7f96840ae8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:30.247 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.241+0000 7f968a507700 1 --2- 172.21.15.110:0/316689586 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f968415cc80 0x7f96840ae8e0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f967c009e50 tx=0x7f967c00b040 comp rx=0 tx=0).stop 2023-10-06T03:30:30.247 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.241+0000 7f968a507700 1 -- 172.21.15.110:0/316689586 shutdown_connections 2023-10-06T03:30:30.247 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.241+0000 7f968a507700 1 --2- 172.21.15.110:0/316689586 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f9670060550 0x7f9670062a00 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:30.247 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.241+0000 7f968a507700 1 --2- 172.21.15.110:0/316689586 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f968415d050 0x7f9684155690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:30.247 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.241+0000 7f968a507700 1 --2- 172.21.15.110:0/316689586 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f968415cc80 0x7f96840ae8e0 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:30.247 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.241+0000 7f968a507700 1 --2- 172.21.15.110:0/316689586 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9684158990 0x7f96841691d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:30.247 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.241+0000 7f968a507700 1 -- 172.21.15.110:0/316689586 >> 172.21.15.110:0/316689586 conn(0x7f968406e760 msgr2=0x7f96840cb960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:30:30.247 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.241+0000 7f968a507700 1 -- 172.21.15.110:0/316689586 shutdown_connections 2023-10-06T03:30:30.248 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.241+0000 7f968a507700 1 -- 172.21.15.110:0/316689586 wait complete. 2023-10-06T03:30:30.262 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-06T03:30:30.367 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.361+0000 7fe997fff700 1 Processor -- start 2023-10-06T03:30:30.367 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.365+0000 7fe997fff700 1 -- start start 2023-10-06T03:30:30.369 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.365+0000 7fe997fff700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fe998158b00 0x7fe9980cca10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:30.369 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.365+0000 7fe997fff700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe9980ccf50 0x7fe99806b7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:30.369 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.365+0000 7fe997fff700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fe99806bd00 0x7fe9980c8160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:30.369 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.365+0000 7fe997fff700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe99806d3d0 con 0x7fe99806bd00 2023-10-06T03:30:30.369 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.365+0000 7fe997fff700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe9980abef0 con 0x7fe998158b00 2023-10-06T03:30:30.369 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.365+0000 7fe997fff700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe99815ca80 con 0x7fe9980ccf50 2023-10-06T03:30:30.370 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.365+0000 7fe9977fe700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fe99806bd00 0x7fe9980c8160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:30.370 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.365+0000 7fe9977fe700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fe99806bd00 0x7fe9980c8160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:40282/0 (socket says 172.21.15.110:40282) 2023-10-06T03:30:30.370 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.365+0000 7fe9967fc700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe9980ccf50 0x7fe99806b7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:30.370 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.365+0000 7fe9977fe700 1 -- 172.21.15.110:0/4050864363 learned_addr learned my addr 172.21.15.110:0/4050864363 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:30:30.370 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.365+0000 7fe996ffd700 1 --2- 172.21.15.110:0/4050864363 >> v2:172.21.15.192:3300/0 conn(0x7fe998158b00 0x7fe9980cca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:30.372 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.365+0000 7fe9977fe700 1 -- 172.21.15.110:0/4050864363 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe9980ccf50 msgr2=0x7fe99806b7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:30:30.372 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.365+0000 7fe9977fe700 1 --2- 172.21.15.110:0/4050864363 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe9980ccf50 0x7fe99806b7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:30.372 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.365+0000 7fe9977fe700 1 -- 172.21.15.110:0/4050864363 >> v2:172.21.15.192:3300/0 conn(0x7fe998158b00 msgr2=0x7fe9980cca10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:30:30.372 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.365+0000 7fe9977fe700 1 --2- 172.21.15.110:0/4050864363 >> v2:172.21.15.192:3300/0 conn(0x7fe998158b00 0x7fe9980cca10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:30.372 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.365+0000 7fe9977fe700 1 -- 172.21.15.110:0/4050864363 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9980cd380 con 0x7fe99806bd00 2023-10-06T03:30:30.372 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.365+0000 7fe996ffd700 1 --2- 172.21.15.110:0/4050864363 >> v2:172.21.15.192:3300/0 conn(0x7fe998158b00 0x7fe9980cca10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-06T03:30:30.372 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.365+0000 7fe9977fe700 1 --2- 172.21.15.110:0/4050864363 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe99806bd00 0x7fe9980c8160 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fe9980ca590 tx=0x7fe990009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=52d86fd933cdbb4d server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:30:30.372 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.365+0000 7fe995ffb700 1 -- 172.21.15.110:0/4050864363 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe99000e070 con 0x7fe99806bd00 2023-10-06T03:30:30.372 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.365+0000 7fe995ffb700 1 -- 172.21.15.110:0/4050864363 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe99000b930 con 0x7fe99806bd00 2023-10-06T03:30:30.373 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.365+0000 7fe995ffb700 1 -- 172.21.15.110:0/4050864363 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9900092e0 con 0x7fe99806bd00 2023-10-06T03:30:30.373 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.369+0000 7fe997fff700 1 -- 172.21.15.110:0/4050864363 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe99806bd00 msgr2=0x7fe9980c8160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:30.373 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.369+0000 7fe997fff700 1 --2- 172.21.15.110:0/4050864363 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe99806bd00 0x7fe9980c8160 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fe9980ca590 tx=0x7fe990009840 comp rx=0 tx=0).stop 2023-10-06T03:30:30.373 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.369+0000 7fe997fff700 1 -- 172.21.15.110:0/4050864363 shutdown_connections 2023-10-06T03:30:30.373 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.369+0000 7fe997fff700 1 --2- 172.21.15.110:0/4050864363 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe99806bd00 0x7fe9980c8160 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:30.373 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.369+0000 7fe997fff700 1 --2- 172.21.15.110:0/4050864363 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe9980ccf50 0x7fe99806b7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:30.373 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.369+0000 7fe997fff700 1 --2- 172.21.15.110:0/4050864363 >> v2:172.21.15.192:3300/0 conn(0x7fe998158b00 0x7fe9980cca10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:30.373 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.369+0000 7fe997fff700 1 -- 172.21.15.110:0/4050864363 >> 172.21.15.110:0/4050864363 conn(0x7fe9980670a0 msgr2=0x7fe9980674a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:30:30.373 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.369+0000 7fe997fff700 1 -- 172.21.15.110:0/4050864363 shutdown_connections 2023-10-06T03:30:30.373 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.369+0000 7fe997fff700 1 -- 172.21.15.110:0/4050864363 wait complete. 2023-10-06T03:30:30.375 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.369+0000 7fe997fff700 1 Processor -- start 2023-10-06T03:30:30.375 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.369+0000 7fe997fff700 1 -- start start 2023-10-06T03:30:30.378 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.373+0000 7fe997fff700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe99806bd00 0x7fe9980aa140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:30.379 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.373+0000 7fe997fff700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe9980ccf50 0x7fe998168e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:30.379 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.373+0000 7fe997fff700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe998158b00 0x7fe998157f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:30.379 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.373+0000 7fe997fff700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9980aa9b0 con 0x7fe9980ccf50 2023-10-06T03:30:30.379 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.373+0000 7fe997fff700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9980b08d0 con 0x7fe998158b00 2023-10-06T03:30:30.379 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.373+0000 7fe997fff700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9980b0bb0 con 0x7fe99806bd00 2023-10-06T03:30:30.379 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.373+0000 7fe9967fc700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe9980ccf50 0x7fe998168e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:30.379 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.373+0000 7fe9967fc700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe9980ccf50 0x7fe998168e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:40294/0 (socket says 172.21.15.110:40294) 2023-10-06T03:30:30.379 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.373+0000 7fe9967fc700 1 -- 172.21.15.110:0/2374535315 learned_addr learned my addr 172.21.15.110:0/2374535315 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:30:30.379 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.373+0000 7fe996ffd700 1 --2- 172.21.15.110:0/2374535315 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe99806bd00 0x7fe9980aa140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:30.379 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.373+0000 7fe9977fe700 1 --2- 172.21.15.110:0/2374535315 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe998158b00 0x7fe998157f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:30.379 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.373+0000 7fe9967fc700 1 -- 172.21.15.110:0/2374535315 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe99806bd00 msgr2=0x7fe9980aa140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:30.380 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.373+0000 7fe9967fc700 1 --2- 172.21.15.110:0/2374535315 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe99806bd00 0x7fe9980aa140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:30.380 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.373+0000 7fe9967fc700 1 -- 172.21.15.110:0/2374535315 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe998158b00 msgr2=0x7fe998157f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:30.380 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.373+0000 7fe9967fc700 1 --2- 172.21.15.110:0/2374535315 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe998158b00 0x7fe998157f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:30.380 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.373+0000 7fe9967fc700 1 -- 172.21.15.110:0/2374535315 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe98c009420 con 0x7fe9980ccf50 2023-10-06T03:30:30.380 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.373+0000 7fe9967fc700 1 --2- 172.21.15.110:0/2374535315 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe9980ccf50 0x7fe998168e30 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7fe98800ec80 tx=0x7fe9880091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:30:30.380 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.373+0000 7fe97effd700 1 -- 172.21.15.110:0/2374535315 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe98800c850 con 0x7fe9980ccf50 2023-10-06T03:30:30.380 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.373+0000 7fe97effd700 1 -- 172.21.15.110:0/2374535315 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe98800c9b0 con 0x7fe9980ccf50 2023-10-06T03:30:30.380 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.373+0000 7fe997fff700 1 -- 172.21.15.110:0/2374535315 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe990009ae0 con 0x7fe9980ccf50 2023-10-06T03:30:30.380 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.373+0000 7fe97effd700 1 -- 172.21.15.110:0/2374535315 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe98800cd50 con 0x7fe9980ccf50 2023-10-06T03:30:30.380 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.373+0000 7fe997fff700 1 -- 172.21.15.110:0/2374535315 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9980b1170 con 0x7fe9980ccf50 2023-10-06T03:30:30.381 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.377+0000 7fe997fff700 1 -- 172.21.15.110:0/2374535315 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe998070ff0 con 0x7fe9980ccf50 2023-10-06T03:30:30.381 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.377+0000 7fe97effd700 1 -- 172.21.15.110:0/2374535315 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7fe988013680 con 0x7fe9980ccf50 2023-10-06T03:30:30.382 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.377+0000 7fe97effd700 1 --2- 172.21.15.110:0/2374535315 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fe984060550 0x7fe984062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:30.382 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.377+0000 7fe97effd700 1 -- 172.21.15.110:0/2374535315 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 5526+0+0 (secure 0 0 0) 0x7fe98807c920 con 0x7fe9980ccf50 2023-10-06T03:30:30.382 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.377+0000 7fe996ffd700 1 --2- 172.21.15.110:0/2374535315 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fe984060550 0x7fe984062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:30.383 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.377+0000 7fe996ffd700 1 --2- 172.21.15.110:0/2374535315 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fe984060550 0x7fe984062a00 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fe98c009f40 tx=0x7fe98c0091a0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:30:30.390 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.385+0000 7fe97effd700 1 -- 172.21.15.110:0/2374535315 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fe98804cfc0 con 0x7fe9980ccf50 2023-10-06T03:30:30.553 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.549+0000 7fe997fff700 1 -- 172.21.15.110:0/2374535315 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe9980c7d10 con 0x7fe9980ccf50 2023-10-06T03:30:30.554 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.549+0000 7fe97effd700 1 -- 172.21.15.110:0/2374535315 <== mon.0 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v38) v1 ==== 74+0+13903 (secure 0 0 0) 0x7fe9880505e0 con 0x7fe9980ccf50 2023-10-06T03:30:30.555 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-10-06T03:30:30.555 INFO:teuthology.orchestra.run.smithi110.stdout:{"epoch":38,"fsid":"93b8f6ac-4c92-4b45-b8e4-b59617024aa2","created":"2023-10-06T03:29:10.519583+0000","modified":"2023-10-06T03:30:27.602993+0000","last_up_change":"2023-10-06T03:29:21.960511+0000","last_in_change":"2023-10-06T03:29:12.560436+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-06T03:29:20.703743+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2023-10-06T03:29:28.307281+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"5866eb6c-2adc-46e4-be7a-e5bcda0f4065","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6824","nonce":33394},{"type":"v1","addr":"172.21.15.110:6825","nonce":33394}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6826","nonce":33394},{"type":"v1","addr":"172.21.15.110:6827","nonce":33394}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6830","nonce":33394},{"type":"v1","addr":"172.21.15.110:6831","nonce":33394}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6828","nonce":33394},{"type":"v1","addr":"172.21.15.110:6829","nonce":33394}]},"public_addr":"172.21.15.110:6825/33394","cluster_addr":"172.21.15.110:6827/33394","heartbeat_back_addr":"172.21.15.110:6831/33394","heartbeat_front_addr":"172.21.15.110:6829/33394","state":["exists","up"]},{"osd":1,"uuid":"a38de118-062b-4a05-9782-2ceddf6ff4df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6800","nonce":33396},{"type":"v1","addr":"172.21.15.110:6801","nonce":33396}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6802","nonce":33396},{"type":"v1","addr":"172.21.15.110:6803","nonce":33396}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6806","nonce":33396},{"type":"v1","addr":"172.21.15.110:6807","nonce":33396}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6804","nonce":33396},{"type":"v1","addr":"172.21.15.110:6805","nonce":33396}]},"public_addr":"172.21.15.110:6801/33396","cluster_addr":"172.21.15.110:6803/33396","heartbeat_back_addr":"172.21.15.110:6807/33396","heartbeat_front_addr":"172.21.15.110:6805/33396","state":["exists","up"]},{"osd":2,"uuid":"56e39be2-3407-4b69-877e-cf16617b06d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6816","nonce":33397},{"type":"v1","addr":"172.21.15.110:6817","nonce":33397}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6818","nonce":33397},{"type":"v1","addr":"172.21.15.110:6819","nonce":33397}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6822","nonce":33397},{"type":"v1","addr":"172.21.15.110:6823","nonce":33397}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6820","nonce":33397},{"type":"v1","addr":"172.21.15.110:6821","nonce":33397}]},"public_addr":"172.21.15.110:6817/33397","cluster_addr":"172.21.15.110:6819/33397","heartbeat_back_addr":"172.21.15.110:6823/33397","heartbeat_front_addr":"172.21.15.110:6821/33397","state":["exists","up"]},{"osd":3,"uuid":"01c82d13-c1d4-4bde-b55e-3e0089b40115","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6808","nonce":33395},{"type":"v1","addr":"172.21.15.110:6809","nonce":33395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6810","nonce":33395},{"type":"v1","addr":"172.21.15.110:6811","nonce":33395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6814","nonce":33395},{"type":"v1","addr":"172.21.15.110:6815","nonce":33395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6812","nonce":33395},{"type":"v1","addr":"172.21.15.110:6813","nonce":33395}]},"public_addr":"172.21.15.110:6809/33395","cluster_addr":"172.21.15.110:6811/33395","heartbeat_back_addr":"172.21.15.110:6815/33395","heartbeat_front_addr":"172.21.15.110:6813/33395","state":["exists","up"]},{"osd":4,"uuid":"86983430-5fa3-4d17-998f-03e2cd843237","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":33683},{"type":"v1","addr":"172.21.15.192:6817","nonce":33683}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":33683},{"type":"v1","addr":"172.21.15.192:6819","nonce":33683}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":33683},{"type":"v1","addr":"172.21.15.192:6823","nonce":33683}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6820","nonce":33683},{"type":"v1","addr":"172.21.15.192:6821","nonce":33683}]},"public_addr":"172.21.15.192:6817/33683","cluster_addr":"172.21.15.192:6819/33683","heartbeat_back_addr":"172.21.15.192:6823/33683","heartbeat_front_addr":"172.21.15.192:6821/33683","state":["exists","up"]},{"osd":5,"uuid":"6945257f-821d-4619-a63f-ebe857eba6f4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":33684},{"type":"v1","addr":"172.21.15.192:6801","nonce":33684}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":33684},{"type":"v1","addr":"172.21.15.192:6803","nonce":33684}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":33684},{"type":"v1","addr":"172.21.15.192:6807","nonce":33684}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6804","nonce":33684},{"type":"v1","addr":"172.21.15.192:6805","nonce":33684}]},"public_addr":"172.21.15.192:6801/33684","cluster_addr":"172.21.15.192:6803/33684","heartbeat_back_addr":"172.21.15.192:6807/33684","heartbeat_front_addr":"172.21.15.192:6805/33684","state":["exists","up"]},{"osd":6,"uuid":"7854d1ae-7060-4780-8f97-73cea5e7a12f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6824","nonce":33686},{"type":"v1","addr":"172.21.15.192:6825","nonce":33686}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":33686},{"type":"v1","addr":"172.21.15.192:6827","nonce":33686}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":33686},{"type":"v1","addr":"172.21.15.192:6831","nonce":33686}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6828","nonce":33686},{"type":"v1","addr":"172.21.15.192:6829","nonce":33686}]},"public_addr":"172.21.15.192:6825/33686","cluster_addr":"172.21.15.192:6827/33686","heartbeat_back_addr":"172.21.15.192:6831/33686","heartbeat_front_addr":"172.21.15.192:6829/33686","state":["exists","up"]},{"osd":7,"uuid":"f1ebdfd5-b859-482b-9f97-c18c4a26f445","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6808","nonce":33685},{"type":"v1","addr":"172.21.15.192:6809","nonce":33685}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":33685},{"type":"v1","addr":"172.21.15.192:6811","nonce":33685}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":33685},{"type":"v1","addr":"172.21.15.192:6815","nonce":33685}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6812","nonce":33685},{"type":"v1","addr":"172.21.15.192:6813","nonce":33685}]},"public_addr":"172.21.15.192:6809/33685","cluster_addr":"172.21.15.192:6811/33685","heartbeat_back_addr":"172.21.15.192:6815/33685","heartbeat_front_addr":"172.21.15.192:6813/33685","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T03:29:18.002048+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T03:29:17.478576+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T03:29:18.034944+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T03:29:17.628513+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T03:29:18.047338+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T03:29:17.772349+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T03:29:17.883958+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T03:29:18.151838+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-06T03:30:30.557 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.553+0000 7fe997fff700 1 -- 172.21.15.110:0/2374535315 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fe984060550 msgr2=0x7fe984062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:30.557 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.553+0000 7fe997fff700 1 --2- 172.21.15.110:0/2374535315 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fe984060550 0x7fe984062a00 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fe98c009f40 tx=0x7fe98c0091a0 comp rx=0 tx=0).stop 2023-10-06T03:30:30.557 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.553+0000 7fe997fff700 1 -- 172.21.15.110:0/2374535315 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe9980ccf50 msgr2=0x7fe998168e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:30.557 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.553+0000 7fe997fff700 1 --2- 172.21.15.110:0/2374535315 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe9980ccf50 0x7fe998168e30 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7fe98800ec80 tx=0x7fe9880091a0 comp rx=0 tx=0).stop 2023-10-06T03:30:30.558 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.553+0000 7fe997fff700 1 -- 172.21.15.110:0/2374535315 shutdown_connections 2023-10-06T03:30:30.558 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.553+0000 7fe997fff700 1 --2- 172.21.15.110:0/2374535315 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fe984060550 0x7fe984062a00 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:30.558 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.553+0000 7fe997fff700 1 --2- 172.21.15.110:0/2374535315 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe998158b00 0x7fe998157f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:30.558 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.553+0000 7fe997fff700 1 --2- 172.21.15.110:0/2374535315 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe9980ccf50 0x7fe998168e30 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:30.558 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.553+0000 7fe997fff700 1 --2- 172.21.15.110:0/2374535315 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fe99806bd00 0x7fe9980aa140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:30.558 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.553+0000 7fe997fff700 1 -- 172.21.15.110:0/2374535315 >> 172.21.15.110:0/2374535315 conn(0x7fe9980670a0 msgr2=0x7fe9980acb20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:30:30.558 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.553+0000 7fe997fff700 1 -- 172.21.15.110:0/2374535315 shutdown_connections 2023-10-06T03:30:30.558 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.553+0000 7fe997fff700 1 -- 172.21.15.110:0/2374535315 wait complete. 2023-10-06T03:30:30.569 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-06T03:29:20.703743+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-10-06T03:29:28.307281+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}] 2023-10-06T03:30:30.569 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-06T03:30:30.679 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.673+0000 7fcbcafb3700 1 Processor -- start 2023-10-06T03:30:30.679 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.673+0000 7fcbcafb3700 1 -- start start 2023-10-06T03:30:30.681 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.677+0000 7fcbcafb3700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fcbc4156fe0 0x7fcbc4159fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:30.682 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.677+0000 7fcbcafb3700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fcbc40c6ba0 0x7fcbc415a4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:30.682 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.677+0000 7fcbcafb3700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcbc40cbdf0 con 0x7fcbc4155c00 2023-10-06T03:30:30.682 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.677+0000 7fcbcafb3700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcbc40c7960 con 0x7fcbc4156fe0 2023-10-06T03:30:30.682 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.677+0000 7fcbcafb3700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcbc4159b20 con 0x7fcbc40c6ba0 2023-10-06T03:30:30.682 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.677+0000 7fcbc9fb1700 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7fcbc4155c00 0x7fcbc406b7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:56636/0 (socket says 172.21.15.110:56636) 2023-10-06T03:30:30.683 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.677+0000 7fcbc9fb1700 1 -- 172.21.15.110:0/761930796 learned_addr learned my addr 172.21.15.110:0/761930796 (peer_addr_for_me v1:172.21.15.110:0/0) 2023-10-06T03:30:30.688 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.677+0000 7fcbc97b0700 1 --2- 172.21.15.110:0/761930796 >> v2:172.21.15.192:3300/0 conn(0x7fcbc4156fe0 0x7fcbc4159fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:30.688 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.677+0000 7fcbc8faf700 1 -- 172.21.15.110:0/761930796 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2182115199 0 0) 0x7fcbc40cbdf0 con 0x7fcbc4155c00 2023-10-06T03:30:30.688 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.677+0000 7fcbc8faf700 1 -- 172.21.15.110:0/761930796 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcbb0003570 con 0x7fcbc4155c00 2023-10-06T03:30:30.688 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.677+0000 7fcbca7b2700 1 --2- 172.21.15.110:0/761930796 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fcbc40c6ba0 0x7fcbc415a4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:30.688 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.677+0000 7fcbc8faf700 1 -- 172.21.15.110:0/761930796 <== mon.0 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 603990405 0 0) 0x7fcbb0003570 con 0x7fcbc4155c00 2023-10-06T03:30:30.688 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.677+0000 7fcbc8faf700 1 -- 172.21.15.110:0/761930796 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fcbc40c6ba0 msgr2=0x7fcbc415a4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:30:30.688 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.677+0000 7fcbc8faf700 1 --2- 172.21.15.110:0/761930796 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fcbc40c6ba0 0x7fcbc415a4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:30.689 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.677+0000 7fcbc8faf700 1 -- 172.21.15.110:0/761930796 >> v2:172.21.15.192:3300/0 conn(0x7fcbc4156fe0 msgr2=0x7fcbc4159fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:30:30.689 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.677+0000 7fcbc8faf700 1 --2- 172.21.15.110:0/761930796 >> v2:172.21.15.192:3300/0 conn(0x7fcbc4156fe0 0x7fcbc4159fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:30.689 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.677+0000 7fcbc8faf700 1 -- 172.21.15.110:0/761930796 --> v1:172.21.15.110:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbc40cba00 con 0x7fcbc4155c00 2023-10-06T03:30:30.689 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.677+0000 7fcbc8faf700 1 -- 172.21.15.110:0/761930796 <== mon.0 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 118603703 0 0) 0x7fcbc0001470 con 0x7fcbc4155c00 2023-10-06T03:30:30.689 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.677+0000 7fcbc8faf700 1 -- 172.21.15.110:0/761930796 >> v1:172.21.15.110:6789/0 conn(0x7fcbc4155c00 legacy=0x7fcbc406b7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:30.689 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.677+0000 7fcbc8faf700 1 --2- 172.21.15.110:0/761930796 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcbb0003a60 0x7fcbb0003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:30.689 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.677+0000 7fcbc8faf700 1 --2- 172.21.15.110:0/761930796 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fcbc40c6ba0 0x7fcbb0004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:30.689 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.677+0000 7fcbc8faf700 1 --2- 172.21.15.110:0/761930796 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcbb0004d50 0x7fcbb00051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:30.689 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.677+0000 7fcbc8faf700 1 -- 172.21.15.110:0/761930796 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbc4159b20 con 0x7fcbb0003a60 2023-10-06T03:30:30.689 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.677+0000 7fcbc8faf700 1 -- 172.21.15.110:0/761930796 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbb000bb40 con 0x7fcbb0004d50 2023-10-06T03:30:30.689 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.677+0000 7fcbc8faf700 1 -- 172.21.15.110:0/761930796 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcbb000bd00 con 0x7fcbc40c6ba0 2023-10-06T03:30:30.689 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.677+0000 7fcbc9fb1700 1 --2- 172.21.15.110:0/761930796 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcbb0003a60 0x7fcbb0003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:30.690 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.677+0000 7fcbca7b2700 1 --2- 172.21.15.110:0/761930796 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcbb0004d50 0x7fcbb00051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:30.690 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.677+0000 7fcbc97b0700 1 --2- 172.21.15.110:0/761930796 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fcbc40c6ba0 0x7fcbb0004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:30.690 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.677+0000 7fcbc9fb1700 1 -- 172.21.15.110:0/761930796 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fcbc40c6ba0 msgr2=0x7fcbb0004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:30:30.690 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.677+0000 7fcbc9fb1700 1 --2- 172.21.15.110:0/761930796 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fcbc40c6ba0 0x7fcbb0004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:30.690 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.677+0000 7fcbc9fb1700 1 -- 172.21.15.110:0/761930796 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcbb0004d50 msgr2=0x7fcbb00051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:30:30.690 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.677+0000 7fcbc9fb1700 1 --2- 172.21.15.110:0/761930796 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcbb0004d50 0x7fcbb00051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:30.690 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.677+0000 7fcbc9fb1700 1 -- 172.21.15.110:0/761930796 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbb0003570 con 0x7fcbb0003a60 2023-10-06T03:30:30.690 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.681+0000 7fcbc9fb1700 1 --2- 172.21.15.110:0/761930796 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcbb0003a60 0x7fcbb0003e70 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7fcbc000c3c0 tx=0x7fcbc0001e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55cfe6dcb13d3181 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:30:30.690 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.681+0000 7fcbc8faf700 1 -- 172.21.15.110:0/761930796 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbc000cc90 con 0x7fcbb0003a60 2023-10-06T03:30:30.690 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.681+0000 7fcbc8faf700 1 -- 172.21.15.110:0/761930796 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcbc000ce10 con 0x7fcbb0003a60 2023-10-06T03:30:30.690 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.681+0000 7fcbc8faf700 1 -- 172.21.15.110:0/761930796 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbc0005b70 con 0x7fcbb0003a60 2023-10-06T03:30:30.691 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.681+0000 7fcbcafb3700 1 -- 172.21.15.110:0/761930796 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcbb0003a60 msgr2=0x7fcbb0003e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:30.691 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.681+0000 7fcbcafb3700 1 --2- 172.21.15.110:0/761930796 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcbb0003a60 0x7fcbb0003e70 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7fcbc000c3c0 tx=0x7fcbc0001e90 comp rx=0 tx=0).stop 2023-10-06T03:30:30.691 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.681+0000 7fcbca7b2700 1 -- 172.21.15.110:0/761930796 reap_dead start 2023-10-06T03:30:30.691 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.681+0000 7fcbcafb3700 1 -- 172.21.15.110:0/761930796 shutdown_connections 2023-10-06T03:30:30.691 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.681+0000 7fcbcafb3700 1 -- 172.21.15.110:0/761930796 >> 172.21.15.110:0/761930796 conn(0x7fcbc4067b70 msgr2=0x7fcbc40cc560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:30:30.691 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.681+0000 7fcbcafb3700 1 -- 172.21.15.110:0/761930796 shutdown_connections 2023-10-06T03:30:30.691 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.681+0000 7fcbcafb3700 1 -- 172.21.15.110:0/761930796 wait complete. 2023-10-06T03:30:30.691 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.681+0000 7fcbcafb3700 1 Processor -- start 2023-10-06T03:30:30.691 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.681+0000 7fcbcafb3700 1 -- start start 2023-10-06T03:30:30.693 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.685+0000 7fcbcafb3700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcbb0003a60 0x7fcbc40c5890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:30.694 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.685+0000 7fcbcafb3700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fcbc40c5dd0 0x7fcbc414c8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:30.694 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.685+0000 7fcbcafb3700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcbc40c3680 0x7fcbc40b0390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:30.694 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.685+0000 7fcbcafb3700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbc40af380 con 0x7fcbc40c3680 2023-10-06T03:30:30.694 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.685+0000 7fcbcafb3700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbc40af5d0 con 0x7fcbb0003a60 2023-10-06T03:30:30.694 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.685+0000 7fcbcafb3700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcbc40af8b0 con 0x7fcbc40c5dd0 2023-10-06T03:30:30.694 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.685+0000 7fcbca7b2700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcbc40c3680 0x7fcbc40b0390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:30.694 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.685+0000 7fcbca7b2700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcbc40c3680 0x7fcbc40b0390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:40304/0 (socket says 172.21.15.110:40304) 2023-10-06T03:30:30.694 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.685+0000 7fcbca7b2700 1 -- 172.21.15.110:0/4201557570 learned_addr learned my addr 172.21.15.110:0/4201557570 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:30:30.694 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.685+0000 7fcbc9fb1700 1 --2- 172.21.15.110:0/4201557570 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcbb0003a60 0x7fcbc40c5890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:30.694 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.685+0000 7fcbc97b0700 1 --2- 172.21.15.110:0/4201557570 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fcbc40c5dd0 0x7fcbc414c8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:30.695 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.685+0000 7fcbca7b2700 1 -- 172.21.15.110:0/4201557570 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fcbc40c5dd0 msgr2=0x7fcbc414c8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:30.695 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.685+0000 7fcbca7b2700 1 --2- 172.21.15.110:0/4201557570 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fcbc40c5dd0 0x7fcbc414c8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:30.695 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.685+0000 7fcbca7b2700 1 -- 172.21.15.110:0/4201557570 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcbb0003a60 msgr2=0x7fcbc40c5890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:30.695 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.685+0000 7fcbca7b2700 1 --2- 172.21.15.110:0/4201557570 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcbb0003a60 0x7fcbc40c5890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:30.695 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.685+0000 7fcbca7b2700 1 -- 172.21.15.110:0/4201557570 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbc40afb90 con 0x7fcbc40c3680 2023-10-06T03:30:30.695 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.685+0000 7fcbca7b2700 1 --2- 172.21.15.110:0/4201557570 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcbc40c3680 0x7fcbc40b0390 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7fcbc41573b0 tx=0x7fcbbc009710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:30:30.695 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.685+0000 7fcbb9ffb700 1 -- 172.21.15.110:0/4201557570 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbbc009b10 con 0x7fcbc40c3680 2023-10-06T03:30:30.695 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.685+0000 7fcbb9ffb700 1 -- 172.21.15.110:0/4201557570 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcbbc002bd0 con 0x7fcbc40c3680 2023-10-06T03:30:30.695 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.685+0000 7fcbcafb3700 1 -- 172.21.15.110:0/4201557570 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcbc40afe70 con 0x7fcbc40c3680 2023-10-06T03:30:30.696 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.685+0000 7fcbb9ffb700 1 -- 172.21.15.110:0/4201557570 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbbc015420 con 0x7fcbc40c3680 2023-10-06T03:30:30.696 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.685+0000 7fcbcafb3700 1 -- 172.21.15.110:0/4201557570 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcbc40c3d00 con 0x7fcbc40c3680 2023-10-06T03:30:30.696 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.689+0000 7fcbcafb3700 1 -- 172.21.15.110:0/4201557570 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcbc40b0060 con 0x7fcbc40c3680 2023-10-06T03:30:30.696 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.689+0000 7fcbb9ffb700 1 -- 172.21.15.110:0/4201557570 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7fcbbc003040 con 0x7fcbc40c3680 2023-10-06T03:30:30.696 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.689+0000 7fcbb9ffb700 1 --2- 172.21.15.110:0/4201557570 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fcba8060550 0x7fcba8062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:30.696 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.689+0000 7fcbb9ffb700 1 -- 172.21.15.110:0/4201557570 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 5526+0+0 (secure 0 0 0) 0x7fcbbc07cbc0 con 0x7fcbc40c3680 2023-10-06T03:30:30.696 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.689+0000 7fcbc9fb1700 1 --2- 172.21.15.110:0/4201557570 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fcba8060550 0x7fcba8062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:30.696 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.689+0000 7fcbc9fb1700 1 --2- 172.21.15.110:0/4201557570 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fcba8060550 0x7fcba8062a00 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fcbc0001ed0 tx=0x7fcbc0015000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:30:30.699 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.693+0000 7fcbb9ffb700 1 -- 172.21.15.110:0/4201557570 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fcbbc04d260 con 0x7fcbc40c3680 2023-10-06T03:30:30.853 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.849+0000 7fcbcafb3700 1 -- 172.21.15.110:0/4201557570 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fcbc4070ff0 con 0x7fcbc40c3680 2023-10-06T03:30:30.853 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.849+0000 7fcbb9ffb700 1 -- 172.21.15.110:0/4201557570 <== mon.0 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v38) v1 ==== 93+0+10 (secure 0 0 0) 0x7fcbbc050880 con 0x7fcbc40c3680 2023-10-06T03:30:30.854 INFO:teuthology.orchestra.run.smithi110.stdout:pg_num: 1 2023-10-06T03:30:30.856 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.853+0000 7fcbcafb3700 1 -- 172.21.15.110:0/4201557570 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fcba8060550 msgr2=0x7fcba8062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:30.856 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.853+0000 7fcbcafb3700 1 --2- 172.21.15.110:0/4201557570 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fcba8060550 0x7fcba8062a00 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fcbc0001ed0 tx=0x7fcbc0015000 comp rx=0 tx=0).stop 2023-10-06T03:30:30.857 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.853+0000 7fcbcafb3700 1 -- 172.21.15.110:0/4201557570 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcbc40c3680 msgr2=0x7fcbc40b0390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:30.857 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.853+0000 7fcbcafb3700 1 --2- 172.21.15.110:0/4201557570 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcbc40c3680 0x7fcbc40b0390 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7fcbc41573b0 tx=0x7fcbbc009710 comp rx=0 tx=0).stop 2023-10-06T03:30:30.857 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.853+0000 7fcbcafb3700 1 -- 172.21.15.110:0/4201557570 shutdown_connections 2023-10-06T03:30:30.857 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.853+0000 7fcbcafb3700 1 --2- 172.21.15.110:0/4201557570 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fcba8060550 0x7fcba8062a00 secure :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fcbc0001ed0 tx=0x7fcbc0015000 comp rx=0 tx=0).stop 2023-10-06T03:30:30.857 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.853+0000 7fcbcafb3700 1 --2- 172.21.15.110:0/4201557570 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcbc40c3680 0x7fcbc40b0390 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:30.857 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.853+0000 7fcbcafb3700 1 --2- 172.21.15.110:0/4201557570 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fcbc40c5dd0 0x7fcbc414c8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:30.858 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.853+0000 7fcbcafb3700 1 --2- 172.21.15.110:0/4201557570 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcbb0003a60 0x7fcbc40c5890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:30.858 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.853+0000 7fcbcafb3700 1 -- 172.21.15.110:0/4201557570 >> 172.21.15.110:0/4201557570 conn(0x7fcbc4067b70 msgr2=0x7fcbc406c7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:30:30.858 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.853+0000 7fcbcafb3700 1 -- 172.21.15.110:0/4201557570 shutdown_connections 2023-10-06T03:30:30.858 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.853+0000 7fcbcafb3700 1 -- 172.21.15.110:0/4201557570 wait complete. 2023-10-06T03:30:30.867 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-10-06T03:30:30.981 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.977+0000 7f073b59e700 1 Processor -- start 2023-10-06T03:30:30.981 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.977+0000 7f073b59e700 1 -- start start 2023-10-06T03:30:30.989 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.977+0000 7f073b59e700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f073c06b7c0 0x7f073c0ce170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:30.990 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.977+0000 7f073b59e700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f073c0cfa00 con 0x7f073c06c2a0 2023-10-06T03:30:30.990 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.977+0000 7f073b59e700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f073c0cedf0 con 0x7f073c0ce6b0 2023-10-06T03:30:30.990 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.977+0000 7f073b59e700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f073c0cb1c0 con 0x7f073c06b7c0 2023-10-06T03:30:30.990 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.977+0000 7f073ad9d700 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f073c06c2a0 0x7f073c162ac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:54568/0 (socket says 172.21.15.110:54568) 2023-10-06T03:30:30.990 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.977+0000 7f073ad9d700 1 -- 172.21.15.110:0/2877681938 learned_addr learned my addr 172.21.15.110:0/2877681938 (peer_addr_for_me v1:172.21.15.110:0/0) 2023-10-06T03:30:30.990 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.981+0000 7f073a59c700 1 --2- 172.21.15.110:0/2877681938 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f073c06b7c0 0x7f073c0ce170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:30.991 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.981+0000 7f073959a700 1 -- 172.21.15.110:0/2877681938 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1676006379 0 0) 0x7f073c0cfa00 con 0x7f073c06c2a0 2023-10-06T03:30:30.991 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.981+0000 7f073959a700 1 -- 172.21.15.110:0/2877681938 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0728003570 con 0x7f073c06c2a0 2023-10-06T03:30:30.991 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.981+0000 7f073959a700 1 -- 172.21.15.110:0/2877681938 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4271495190 0 0) 0x7f073c0cedf0 con 0x7f073c0ce6b0 2023-10-06T03:30:30.991 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.981+0000 7f073959a700 1 -- 172.21.15.110:0/2877681938 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f073c0cfa00 con 0x7f073c0ce6b0 2023-10-06T03:30:30.992 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.981+0000 7f073959a700 1 -- 172.21.15.110:0/2877681938 <== mon.0 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 145422224 0 0) 0x7f0728003570 con 0x7f073c06c2a0 2023-10-06T03:30:30.992 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.981+0000 7f073959a700 1 -- 172.21.15.110:0/2877681938 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f073c06b7c0 msgr2=0x7f073c0ce170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:30:30.992 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.981+0000 7f073959a700 1 --2- 172.21.15.110:0/2877681938 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f073c06b7c0 0x7f073c0ce170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:30.992 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.981+0000 7f073959a700 1 -- 172.21.15.110:0/2877681938 >> v1:172.21.15.192:6789/0 conn(0x7f073c0ce6b0 legacy=0x7f073c06bb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:30.993 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.981+0000 7f073959a700 1 -- 172.21.15.110:0/2877681938 --> v1:172.21.15.110:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f073c0cf0d0 con 0x7f073c06c2a0 2023-10-06T03:30:30.993 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.981+0000 7f073959a700 1 -- 172.21.15.110:0/2877681938 <== mon.0 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 118603703 0 0) 0x7f07340017f0 con 0x7f073c06c2a0 2023-10-06T03:30:30.993 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.981+0000 7f073959a700 1 -- 172.21.15.110:0/2877681938 >> v1:172.21.15.110:6789/0 conn(0x7f073c06c2a0 legacy=0x7f073c162ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:30.993 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.981+0000 7f073959a700 1 --2- 172.21.15.110:0/2877681938 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0728003c20 0x7f0728004010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:30.993 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.981+0000 7f073959a700 1 --2- 172.21.15.110:0/2877681938 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0728004810 0x7f0728004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:30.993 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.981+0000 7f073959a700 1 --2- 172.21.15.110:0/2877681938 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f07280051c0 0x7f072800ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:30.994 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.981+0000 7f073959a700 1 -- 172.21.15.110:0/2877681938 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f073c0cb1c0 con 0x7f0728003c20 2023-10-06T03:30:30.994 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.981+0000 7f073959a700 1 -- 172.21.15.110:0/2877681938 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f072800c020 con 0x7f07280051c0 2023-10-06T03:30:30.994 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.981+0000 7f073959a700 1 -- 172.21.15.110:0/2877681938 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f072800c1e0 con 0x7f0728004810 2023-10-06T03:30:30.994 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.981+0000 7f073a59c700 1 --2- 172.21.15.110:0/2877681938 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f073c06b7c0 0x7f073c0ce170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-06T03:30:30.994 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.981+0000 7f073ad9d700 1 --2- 172.21.15.110:0/2877681938 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f07280051c0 0x7f072800ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:30.995 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.981+0000 7f0739d9b700 1 --2- 172.21.15.110:0/2877681938 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0728004810 0x7f0728004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:30.995 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.981+0000 7f073a59c700 1 --2- 172.21.15.110:0/2877681938 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0728003c20 0x7f0728004010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:30.995 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.981+0000 7f073ad9d700 1 -- 172.21.15.110:0/2877681938 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0728004810 msgr2=0x7f0728004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:30:30.995 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.981+0000 7f073ad9d700 1 --2- 172.21.15.110:0/2877681938 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0728004810 0x7f0728004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:30.995 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.981+0000 7f073ad9d700 1 -- 172.21.15.110:0/2877681938 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0728003c20 msgr2=0x7f0728004010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:30:30.995 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.981+0000 7f073ad9d700 1 --2- 172.21.15.110:0/2877681938 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0728003c20 0x7f0728004010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:30.996 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.981+0000 7f073ad9d700 1 -- 172.21.15.110:0/2877681938 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0728003570 con 0x7f07280051c0 2023-10-06T03:30:30.996 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.981+0000 7f073a59c700 1 --2- 172.21.15.110:0/2877681938 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0728003c20 0x7f0728004010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-06T03:30:30.996 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.981+0000 7f073ad9d700 1 --2- 172.21.15.110:0/2877681938 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f07280051c0 0x7f072800ba60 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f073400a8d0 tx=0x7f0734002190 comp rx=0 tx=0).ready entity=mon.1 client_cookie=51923d47e9a2e915 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:30:30.996 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.981+0000 7f073959a700 1 -- 172.21.15.110:0/2877681938 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07340070d0 con 0x7f07280051c0 2023-10-06T03:30:30.996 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.981+0000 7f073959a700 1 -- 172.21.15.110:0/2877681938 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0734007230 con 0x7f07280051c0 2023-10-06T03:30:30.996 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.981+0000 7f073959a700 1 -- 172.21.15.110:0/2877681938 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07340075f0 con 0x7f07280051c0 2023-10-06T03:30:30.996 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.981+0000 7f073b59e700 1 -- 172.21.15.110:0/2877681938 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f07280051c0 msgr2=0x7f072800ba60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:30.996 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.981+0000 7f073b59e700 1 --2- 172.21.15.110:0/2877681938 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f07280051c0 0x7f072800ba60 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f073400a8d0 tx=0x7f0734002190 comp rx=0 tx=0).stop 2023-10-06T03:30:30.996 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.981+0000 7f073ad9d700 1 -- 172.21.15.110:0/2877681938 reap_dead start 2023-10-06T03:30:30.997 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.981+0000 7f073b59e700 1 -- 172.21.15.110:0/2877681938 shutdown_connections 2023-10-06T03:30:30.997 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.981+0000 7f073b59e700 1 -- 172.21.15.110:0/2877681938 >> 172.21.15.110:0/2877681938 conn(0x7f073c0cfea0 msgr2=0x7f073c0d02a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:30:30.997 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.985+0000 7f073b59e700 1 -- 172.21.15.110:0/2877681938 shutdown_connections 2023-10-06T03:30:30.997 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.985+0000 7f073b59e700 1 -- 172.21.15.110:0/2877681938 wait complete. 2023-10-06T03:30:30.997 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.985+0000 7f073b59e700 1 Processor -- start 2023-10-06T03:30:30.997 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.985+0000 7f073b59e700 1 -- start start 2023-10-06T03:30:31.000 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.985+0000 7f073b59e700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f07280051c0 0x7f073c15b070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:31.000 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.985+0000 7f073b59e700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f073c1467e0 0x7f073c15dba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:31.000 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.985+0000 7f073b59e700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f073c146c30 0x7f073c15e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:31.000 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.985+0000 7f073b59e700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f073c159560 con 0x7f07280051c0 2023-10-06T03:30:31.000 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.985+0000 7f073b59e700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f073c159840 con 0x7f073c146c30 2023-10-06T03:30:31.000 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.985+0000 7f073b59e700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f073c159b20 con 0x7f073c1467e0 2023-10-06T03:30:31.000 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.985+0000 7f073a59c700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f07280051c0 0x7f073c15b070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:31.000 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.985+0000 7f073a59c700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f07280051c0 0x7f073c15b070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:47994/0 (socket says 172.21.15.110:47994) 2023-10-06T03:30:31.000 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.985+0000 7f073a59c700 1 -- 172.21.15.110:0/2719743864 learned_addr learned my addr 172.21.15.110:0/2719743864 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:30:31.000 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.985+0000 7f073ad9d700 1 --2- 172.21.15.110:0/2719743864 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f073c146c30 0x7f073c15e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:31.001 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.985+0000 7f0739d9b700 1 --2- 172.21.15.110:0/2719743864 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f073c1467e0 0x7f073c15dba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:31.001 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.989+0000 7f073a59c700 1 -- 172.21.15.110:0/2719743864 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f073c1467e0 msgr2=0x7f073c15dba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:31.001 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.989+0000 7f073a59c700 1 --2- 172.21.15.110:0/2719743864 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f073c1467e0 0x7f073c15dba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:31.001 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.989+0000 7f073a59c700 1 -- 172.21.15.110:0/2719743864 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f073c146c30 msgr2=0x7f073c15e0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:31.001 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.989+0000 7f073a59c700 1 --2- 172.21.15.110:0/2719743864 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f073c146c30 0x7f073c15e0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:31.001 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.989+0000 7f073a59c700 1 -- 172.21.15.110:0/2719743864 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f073c159e00 con 0x7f07280051c0 2023-10-06T03:30:31.001 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.989+0000 7f073ad9d700 1 --2- 172.21.15.110:0/2719743864 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f073c146c30 0x7f073c15e0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-06T03:30:31.001 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.989+0000 7f073a59c700 1 --2- 172.21.15.110:0/2719743864 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f07280051c0 0x7f073c15b070 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f072c009fd0 tx=0x7f072c0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:30:31.001 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.989+0000 7f07327fc700 1 -- 172.21.15.110:0/2719743864 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f072c011ed0 con 0x7f07280051c0 2023-10-06T03:30:31.001 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.989+0000 7f07327fc700 1 -- 172.21.15.110:0/2719743864 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f072c00fcd0 con 0x7f07280051c0 2023-10-06T03:30:31.001 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.989+0000 7f073b59e700 1 -- 172.21.15.110:0/2719743864 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f073c15a0e0 con 0x7f07280051c0 2023-10-06T03:30:31.002 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.989+0000 7f07327fc700 1 -- 172.21.15.110:0/2719743864 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f072c01d690 con 0x7f07280051c0 2023-10-06T03:30:31.002 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.989+0000 7f073b59e700 1 -- 172.21.15.110:0/2719743864 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f073c0c8270 con 0x7f07280051c0 2023-10-06T03:30:31.002 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.989+0000 7f073b59e700 1 -- 172.21.15.110:0/2719743864 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f073c169010 con 0x7f07280051c0 2023-10-06T03:30:31.002 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.989+0000 7f07327fc700 1 -- 172.21.15.110:0/2719743864 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7f072c01dac0 con 0x7f07280051c0 2023-10-06T03:30:31.002 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.989+0000 7f07327fc700 1 --2- 172.21.15.110:0/2719743864 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f071c060580 0x7f071c062a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:31.002 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.993+0000 7f07327fc700 1 -- 172.21.15.110:0/2719743864 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 5526+0+0 (secure 0 0 0) 0x7f072c018070 con 0x7f07280051c0 2023-10-06T03:30:31.002 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.993+0000 7f0739d9b700 1 --2- 172.21.15.110:0/2719743864 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f071c060580 0x7f071c062a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:31.002 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.993+0000 7f0739d9b700 1 --2- 172.21.15.110:0/2719743864 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f071c060580 0x7f071c062a30 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f0724000c00 tx=0x7f0724009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:30:31.003 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:30.997+0000 7f07327fc700 1 -- 172.21.15.110:0/2719743864 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f072c04ddf0 con 0x7f07280051c0 2023-10-06T03:30:31.166 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:31.161+0000 7f073b59e700 1 -- 172.21.15.110:0/2719743864 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f073c070ff0 con 0x7f07280051c0 2023-10-06T03:30:31.167 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:31.165+0000 7f07327fc700 1 -- 172.21.15.110:0/2719743864 <== mon.0 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v38) v1 ==== 92+0+10 (secure 0 0 0) 0x7f072c051410 con 0x7f07280051c0 2023-10-06T03:30:31.167 INFO:teuthology.orchestra.run.smithi110.stdout:pg_num: 8 2023-10-06T03:30:31.170 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:31.165+0000 7f073b59e700 1 -- 172.21.15.110:0/2719743864 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f071c060580 msgr2=0x7f071c062a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:31.170 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:31.165+0000 7f073b59e700 1 --2- 172.21.15.110:0/2719743864 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f071c060580 0x7f071c062a30 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f0724000c00 tx=0x7f0724009040 comp rx=0 tx=0).stop 2023-10-06T03:30:31.170 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:31.165+0000 7f073b59e700 1 -- 172.21.15.110:0/2719743864 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f07280051c0 msgr2=0x7f073c15b070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:31.171 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:31.165+0000 7f073b59e700 1 --2- 172.21.15.110:0/2719743864 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f07280051c0 0x7f073c15b070 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f072c009fd0 tx=0x7f072c0091a0 comp rx=0 tx=0).stop 2023-10-06T03:30:31.171 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:31.165+0000 7f073b59e700 1 -- 172.21.15.110:0/2719743864 shutdown_connections 2023-10-06T03:30:31.171 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:31.165+0000 7f073b59e700 1 --2- 172.21.15.110:0/2719743864 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f071c060580 0x7f071c062a30 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:31.171 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:31.165+0000 7f073b59e700 1 --2- 172.21.15.110:0/2719743864 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f073c146c30 0x7f073c15e0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:31.171 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:31.165+0000 7f073b59e700 1 --2- 172.21.15.110:0/2719743864 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f073c1467e0 0x7f073c15dba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:31.171 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:31.165+0000 7f073b59e700 1 --2- 172.21.15.110:0/2719743864 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f07280051c0 0x7f073c15b070 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:31.171 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:31.165+0000 7f073b59e700 1 -- 172.21.15.110:0/2719743864 >> 172.21.15.110:0/2719743864 conn(0x7f073c0cfea0 msgr2=0x7f073c153660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:30:31.171 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:31.165+0000 7f073b59e700 1 -- 172.21.15.110:0/2719743864 shutdown_connections 2023-10-06T03:30:31.172 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:31.165+0000 7f073b59e700 1 -- 172.21.15.110:0/2719743864 wait complete. 2023-10-06T03:30:31.181 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2023-10-06T03:30:31.285 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:31.281+0000 7fcec5fe6700 1 Processor -- start 2023-10-06T03:30:31.286 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:31.281+0000 7fcec5fe6700 1 -- start start 2023-10-06T03:30:31.286 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:31.281+0000 7fcec5fe6700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fcec006f340 0x7fcec00adcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:31.287 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:31.281+0000 7fcec5fe6700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcec00ca500 con 0x7fcec00c63f0 2023-10-06T03:30:31.287 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:31.281+0000 7fcec5fe6700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcec00ca7e0 con 0x7fcec00ca020 2023-10-06T03:30:31.287 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:31.281+0000 7fcec5fe6700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcec0156440 con 0x7fcec006f340 2023-10-06T03:30:31.287 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:31.281+0000 7fcebffff700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fcec006f340 0x7fcec00adcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:31.287 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:31.281+0000 7fcec57e5700 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7fcec00c63f0 0x7fcec00c6870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:54580/0 (socket says 172.21.15.110:54580) 2023-10-06T03:30:31.287 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:31.281+0000 7fcec57e5700 1 -- 172.21.15.110:0/3069844188 learned_addr learned my addr 172.21.15.110:0/3069844188 (peer_addr_for_me v1:172.21.15.110:0/0) 2023-10-06T03:30:31.288 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:31.281+0000 7fcebffff700 1 -- 172.21.15.110:0/3069844188 >> v1:172.21.15.192:6789/0 conn(0x7fcec00ca020 legacy=0x7fcec01592c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:31.288 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:31.281+0000 7fcebffff700 1 -- 172.21.15.110:0/3069844188 >> v1:172.21.15.110:6789/0 conn(0x7fcec00c63f0 legacy=0x7fcec00c6870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:31.288 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:31.281+0000 7fcebffff700 1 -- 172.21.15.110:0/3069844188 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcec00caac0 con 0x7fcec006f340 2023-10-06T03:30:31.288 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:31.285+0000 7fcebffff700 1 --2- 172.21.15.110:0/3069844188 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcec006f340 0x7fcec00adcd0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fceb0009e60 tx=0x7fceb0009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a6953101ff36e8a4 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:30:31.288 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:31.285+0000 7fcebf7fe700 1 -- 172.21.15.110:0/3069844188 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fceb000e070 con 0x7fcec006f340 2023-10-06T03:30:31.288 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:31.285+0000 7fcebf7fe700 1 -- 172.21.15.110:0/3069844188 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fceb000bc30 con 0x7fcec006f340 2023-10-06T03:30:31.288 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:31.285+0000 7fcebf7fe700 1 -- 172.21.15.110:0/3069844188 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fceb0012430 con 0x7fcec006f340 2023-10-06T03:30:31.289 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:31.285+0000 7fcec5fe6700 1 -- 172.21.15.110:0/3069844188 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcec006f340 msgr2=0x7fcec00adcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:31.290 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:31.285+0000 7fcec5fe6700 1 --2- 172.21.15.110:0/3069844188 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcec006f340 0x7fcec00adcd0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fceb0009e60 tx=0x7fceb0009780 comp rx=0 tx=0).stop 2023-10-06T03:30:31.290 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:31.285+0000 7fcec5fe6700 1 -- 172.21.15.110:0/3069844188 shutdown_connections 2023-10-06T03:30:31.290 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:31.285+0000 7fcec5fe6700 1 --2- 172.21.15.110:0/3069844188 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcec006f340 0x7fcec00adcd0 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:31.290 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:31.285+0000 7fcec5fe6700 1 -- 172.21.15.110:0/3069844188 >> 172.21.15.110:0/3069844188 conn(0x7fcec00cc470 msgr2=0x7fcec00c8410 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:30:31.290 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:31.285+0000 7fcec5fe6700 1 -- 172.21.15.110:0/3069844188 shutdown_connections 2023-10-06T03:30:31.290 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:31.285+0000 7fcec5fe6700 1 -- 172.21.15.110:0/3069844188 wait complete. 2023-10-06T03:30:31.292 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:31.289+0000 7fcec5fe6700 1 Processor -- start 2023-10-06T03:30:31.292 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:31.289+0000 7fcec5fe6700 1 -- start start 2023-10-06T03:30:31.295 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:31.289+0000 7fcec5fe6700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fcec006f340 0x7fcec01521b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:31.295 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:31.289+0000 7fcec5fe6700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcec00c63f0 0x7fcec01549d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:31.295 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:31.289+0000 7fcec5fe6700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcec00ca020 0x7fcec00c29b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:31.295 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:31.289+0000 7fcec5fe6700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcec01674f0 con 0x7fcec00c63f0 2023-10-06T03:30:31.296 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:31.289+0000 7fcec5fe6700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcec01526f0 con 0x7fcec00ca020 2023-10-06T03:30:31.296 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:31.289+0000 7fcec5fe6700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcec016af90 con 0x7fcec006f340 2023-10-06T03:30:31.296 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:31.289+0000 7fcebffff700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcec00c63f0 0x7fcec01549d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:31.296 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:31.289+0000 7fcebffff700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcec00c63f0 0x7fcec01549d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:48016/0 (socket says 172.21.15.110:48016) 2023-10-06T03:30:31.296 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:31.289+0000 7fcebffff700 1 -- 172.21.15.110:0/3688190593 learned_addr learned my addr 172.21.15.110:0/3688190593 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:30:31.296 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:31.289+0000 7fcec4fe4700 1 --2- 172.21.15.110:0/3688190593 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fcec006f340 0x7fcec01521b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:31.296 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:31.289+0000 7fcec57e5700 1 --2- 172.21.15.110:0/3688190593 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcec00ca020 0x7fcec00c29b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:31.296 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:31.289+0000 7fcebffff700 1 -- 172.21.15.110:0/3688190593 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fcec006f340 msgr2=0x7fcec01521b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:31.296 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:31.289+0000 7fcebffff700 1 --2- 172.21.15.110:0/3688190593 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fcec006f340 0x7fcec01521b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:31.296 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:31.289+0000 7fcebffff700 1 -- 172.21.15.110:0/3688190593 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcec00ca020 msgr2=0x7fcec00c29b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:31.296 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:31.289+0000 7fcebffff700 1 --2- 172.21.15.110:0/3688190593 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcec00ca020 0x7fcec00c29b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:31.297 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:31.289+0000 7fcebffff700 1 -- 172.21.15.110:0/3688190593 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fceb0009a20 con 0x7fcec00c63f0 2023-10-06T03:30:31.297 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:31.293+0000 7fcebffff700 1 --2- 172.21.15.110:0/3688190593 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcec00c63f0 0x7fcec01549d0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fceb0013000 tx=0x7fceb0019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:30:31.297 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:31.293+0000 7fcebcff9700 1 -- 172.21.15.110:0/3688190593 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fceb0001090 con 0x7fcec00c63f0 2023-10-06T03:30:31.298 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:31.293+0000 7fcebcff9700 1 -- 172.21.15.110:0/3688190593 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fceb0027470 con 0x7fcec00c63f0 2023-10-06T03:30:31.298 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:31.293+0000 7fcec5fe6700 1 -- 172.21.15.110:0/3688190593 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcec016b270 con 0x7fcec00c63f0 2023-10-06T03:30:31.298 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:31.293+0000 7fcebcff9700 1 -- 172.21.15.110:0/3688190593 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fceb0027810 con 0x7fcec00c63f0 2023-10-06T03:30:31.298 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:31.293+0000 7fcec5fe6700 1 -- 172.21.15.110:0/3688190593 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcec016b830 con 0x7fcec00c63f0 2023-10-06T03:30:31.298 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:31.293+0000 7fcec5fe6700 1 -- 172.21.15.110:0/3688190593 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcec0167710 con 0x7fcec00c63f0 2023-10-06T03:30:31.299 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:31.293+0000 7fcebcff9700 1 -- 172.21.15.110:0/3688190593 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7fceb0024070 con 0x7fcec00c63f0 2023-10-06T03:30:31.299 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:31.293+0000 7fcebcff9700 1 --2- 172.21.15.110:0/3688190593 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fceac060550 0x7fceac062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:31.299 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:31.293+0000 7fcebcff9700 1 -- 172.21.15.110:0/3688190593 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 5526+0+0 (secure 0 0 0) 0x7fceb007ed40 con 0x7fcec00c63f0 2023-10-06T03:30:31.299 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:31.293+0000 7fcec4fe4700 1 --2- 172.21.15.110:0/3688190593 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fceac060550 0x7fceac062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:31.299 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:31.297+0000 7fcec4fe4700 1 --2- 172.21.15.110:0/3688190593 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fceac060550 0x7fceac062a00 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fcec00bf110 tx=0x7fceb4009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:30:31.303 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:31.301+0000 7fcebcff9700 1 -- 172.21.15.110:0/3688190593 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fcec0167710 con 0x7fcec00c63f0 2023-10-06T03:30:31.466 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:31.461+0000 7fcec5fe6700 1 -- 172.21.15.110:0/3688190593 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7fcec0167710 con 0x7fcec00c63f0 2023-10-06T03:30:31.467 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:31.461+0000 7fcebcff9700 1 -- 172.21.15.110:0/3688190593 <== mon.0 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v38) v1 ==== 84+0+126 (secure 0 0 0) 0x7fcec0167710 con 0x7fcec00c63f0 2023-10-06T03:30:31.467 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-10-06T03:30:31.467 INFO:teuthology.orchestra.run.smithi110.stdout:[ 2023-10-06T03:30:31.467 INFO:teuthology.orchestra.run.smithi110.stdout: { 2023-10-06T03:30:31.467 INFO:teuthology.orchestra.run.smithi110.stdout: "poolnum": 1, 2023-10-06T03:30:31.467 INFO:teuthology.orchestra.run.smithi110.stdout: "poolname": ".mgr" 2023-10-06T03:30:31.468 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2023-10-06T03:30:31.468 INFO:teuthology.orchestra.run.smithi110.stdout: { 2023-10-06T03:30:31.468 INFO:teuthology.orchestra.run.smithi110.stdout: "poolnum": 2, 2023-10-06T03:30:31.468 INFO:teuthology.orchestra.run.smithi110.stdout: "poolname": "rbd" 2023-10-06T03:30:31.468 INFO:teuthology.orchestra.run.smithi110.stdout: } 2023-10-06T03:30:31.468 INFO:teuthology.orchestra.run.smithi110.stdout:] 2023-10-06T03:30:31.469 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:31.465+0000 7fcec5fe6700 1 -- 172.21.15.110:0/3688190593 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fceac060550 msgr2=0x7fceac062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:31.470 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:31.465+0000 7fcec5fe6700 1 --2- 172.21.15.110:0/3688190593 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fceac060550 0x7fceac062a00 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fcec00bf110 tx=0x7fceb4009040 comp rx=0 tx=0).stop 2023-10-06T03:30:31.470 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:31.465+0000 7fcec5fe6700 1 -- 172.21.15.110:0/3688190593 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcec00c63f0 msgr2=0x7fcec01549d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:31.470 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:31.465+0000 7fcec5fe6700 1 --2- 172.21.15.110:0/3688190593 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcec00c63f0 0x7fcec01549d0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fceb0013000 tx=0x7fceb0019000 comp rx=0 tx=0).stop 2023-10-06T03:30:31.470 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:31.465+0000 7fcec5fe6700 1 -- 172.21.15.110:0/3688190593 shutdown_connections 2023-10-06T03:30:31.470 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:31.465+0000 7fcec5fe6700 1 --2- 172.21.15.110:0/3688190593 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fceac060550 0x7fceac062a00 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:31.470 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:31.465+0000 7fcec5fe6700 1 --2- 172.21.15.110:0/3688190593 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcec00ca020 0x7fcec00c29b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:31.470 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:31.465+0000 7fcec5fe6700 1 --2- 172.21.15.110:0/3688190593 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcec00c63f0 0x7fcec01549d0 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:31.470 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:31.465+0000 7fcec5fe6700 1 --2- 172.21.15.110:0/3688190593 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fcec006f340 0x7fcec01521b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:31.470 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:31.465+0000 7fcec5fe6700 1 -- 172.21.15.110:0/3688190593 >> 172.21.15.110:0/3688190593 conn(0x7fcec00cc470 msgr2=0x7fcec006dc60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:30:31.471 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:31.465+0000 7fcec5fe6700 1 -- 172.21.15.110:0/3688190593 shutdown_connections 2023-10-06T03:30:31.471 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:31.465+0000 7fcec5fe6700 1 -- 172.21.15.110:0/3688190593 wait complete. 2023-10-06T03:30:31.481 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2023-10-06T03:30:31.481 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2023-10-06T03:30:31.593 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:31.589+0000 7f494a70a700 1 Processor -- start 2023-10-06T03:30:31.593 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:31.589+0000 7f494a70a700 1 -- start start 2023-10-06T03:30:31.595 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:31.589+0000 7f494a70a700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f49440c9160 0x7f494406d4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:31.595 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:31.589+0000 7f494a70a700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f49440b1890 0x7f49440a5d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:31.595 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:31.589+0000 7f494a70a700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49440a5870 con 0x7f49440a44c0 2023-10-06T03:30:31.595 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:31.589+0000 7f494a70a700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f49441560c0 con 0x7f49440b1890 2023-10-06T03:30:31.595 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:31.589+0000 7f494a70a700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49440a4c60 con 0x7f49440c9160 2023-10-06T03:30:31.595 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:31.589+0000 7f4948f07700 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f49440a44c0 0x7f49440b1180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:54594/0 (socket says 172.21.15.110:54594) 2023-10-06T03:30:31.595 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:31.589+0000 7f4948f07700 1 -- 172.21.15.110:0/1198902450 learned_addr learned my addr 172.21.15.110:0/1198902450 (peer_addr_for_me v1:172.21.15.110:0/0) 2023-10-06T03:30:31.595 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:31.589+0000 7f4949f09700 1 --2- 172.21.15.110:0/1198902450 >> v2:172.21.15.192:3300/0 conn(0x7f49440b1890 0x7f49440a5d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:31.595 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:31.589+0000 7f4949708700 1 --2- 172.21.15.110:0/1198902450 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f49440c9160 0x7f494406d4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:31.596 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:31.589+0000 7f4943fff700 1 -- 172.21.15.110:0/1198902450 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 664093043 0 0) 0x7f49440a5870 con 0x7f49440a44c0 2023-10-06T03:30:31.597 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:31.589+0000 7f4943fff700 1 -- 172.21.15.110:0/1198902450 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f492c003570 con 0x7f49440a44c0 2023-10-06T03:30:31.597 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:31.593+0000 7f4949f09700 1 -- 172.21.15.110:0/1198902450 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f49440c9160 msgr2=0x7f494406d4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:30:31.597 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:31.593+0000 7f4949f09700 1 --2- 172.21.15.110:0/1198902450 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f49440c9160 0x7f494406d4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:31.597 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:31.593+0000 7f4949f09700 1 -- 172.21.15.110:0/1198902450 >> v1:172.21.15.110:6789/0 conn(0x7f49440a44c0 legacy=0x7f49440b1180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:31.597 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:31.593+0000 7f4949f09700 1 -- 172.21.15.110:0/1198902450 --> v2:172.21.15.192:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49440a4f40 con 0x7f49440b1890 2023-10-06T03:30:31.597 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:31.593+0000 7f4949f09700 1 --2- 172.21.15.110:0/1198902450 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f49440b1890 0x7f49440a5d00 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f493c009910 tx=0x7f493c0091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3678b2e4826f5eb9 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:30:31.597 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:31.593+0000 7f4943fff700 1 -- 172.21.15.110:0/1198902450 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f493c011630 con 0x7f49440b1890 2023-10-06T03:30:31.598 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:31.593+0000 7f4943fff700 1 -- 172.21.15.110:0/1198902450 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f493c011790 con 0x7f49440b1890 2023-10-06T03:30:31.598 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:31.593+0000 7f4943fff700 1 -- 172.21.15.110:0/1198902450 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f493c011b30 con 0x7f49440b1890 2023-10-06T03:30:31.598 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:31.593+0000 7f494a70a700 1 -- 172.21.15.110:0/1198902450 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f49440b1890 msgr2=0x7f49440a5d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:31.598 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:31.593+0000 7f494a70a700 1 --2- 172.21.15.110:0/1198902450 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f49440b1890 0x7f49440a5d00 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f493c009910 tx=0x7f493c0091a0 comp rx=0 tx=0).stop 2023-10-06T03:30:31.599 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:31.593+0000 7f494a70a700 1 -- 172.21.15.110:0/1198902450 shutdown_connections 2023-10-06T03:30:31.599 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:31.593+0000 7f494a70a700 1 --2- 172.21.15.110:0/1198902450 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f49440b1890 0x7f49440a5d00 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:31.599 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:31.593+0000 7f494a70a700 1 --2- 172.21.15.110:0/1198902450 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f49440c9160 0x7f494406d4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:31.599 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:31.593+0000 7f494a70a700 1 -- 172.21.15.110:0/1198902450 >> 172.21.15.110:0/1198902450 conn(0x7f49440670a0 msgr2=0x7f4944067470 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:30:31.599 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:31.593+0000 7f494a70a700 1 -- 172.21.15.110:0/1198902450 shutdown_connections 2023-10-06T03:30:31.599 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:31.593+0000 7f494a70a700 1 -- 172.21.15.110:0/1198902450 wait complete. 2023-10-06T03:30:31.599 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:31.593+0000 7f494a70a700 1 Processor -- start 2023-10-06T03:30:31.600 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:31.597+0000 7f494a70a700 1 -- start start 2023-10-06T03:30:31.601 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:31.597+0000 7f494a70a700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f49440a44c0 0x7f49441490c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:31.601 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:31.597+0000 7f494a70a700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f49440b1890 0x7f49440ae9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:31.601 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:31.597+0000 7f494a70a700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f49440c9160 0x7f49440aef30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:31.601 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:31.597+0000 7f494a70a700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49441465c0 con 0x7f49440a44c0 2023-10-06T03:30:31.601 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:31.597+0000 7f494a70a700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49440c8060 con 0x7f49440b1890 2023-10-06T03:30:31.601 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:31.597+0000 7f494a70a700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49440c7450 con 0x7f49440c9160 2023-10-06T03:30:31.602 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:31.597+0000 7f4949708700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f49440a44c0 0x7f49441490c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:31.602 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:31.597+0000 7f4949708700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f49440a44c0 0x7f49441490c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:48024/0 (socket says 172.21.15.110:48024) 2023-10-06T03:30:31.602 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:31.597+0000 7f4949708700 1 -- 172.21.15.110:0/3767631249 learned_addr learned my addr 172.21.15.110:0/3767631249 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:30:31.602 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:31.597+0000 7f4948f07700 1 --2- 172.21.15.110:0/3767631249 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f49440b1890 0x7f49440ae9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:31.602 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:31.597+0000 7f4949f09700 1 --2- 172.21.15.110:0/3767631249 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f49440c9160 0x7f49440aef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:31.602 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:31.597+0000 7f4949708700 1 -- 172.21.15.110:0/3767631249 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f49440c9160 msgr2=0x7f49440aef30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:31.602 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:31.597+0000 7f4949708700 1 --2- 172.21.15.110:0/3767631249 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f49440c9160 0x7f49440aef30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:31.602 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:31.597+0000 7f4949708700 1 -- 172.21.15.110:0/3767631249 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f49440b1890 msgr2=0x7f49440ae9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:31.602 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:31.597+0000 7f4949708700 1 --2- 172.21.15.110:0/3767631249 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f49440b1890 0x7f49440ae9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:31.602 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:31.597+0000 7f4949708700 1 -- 172.21.15.110:0/3767631249 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f493c009420 con 0x7f49440a44c0 2023-10-06T03:30:31.602 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:31.597+0000 7f4948f07700 1 --2- 172.21.15.110:0/3767631249 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f49440b1890 0x7f49440ae9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-06T03:30:31.603 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:31.597+0000 7f4949708700 1 --2- 172.21.15.110:0/3767631249 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f49440a44c0 0x7f49441490c0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f493400e930 tx=0x7f49340091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:30:31.603 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:31.597+0000 7f49417fa700 1 -- 172.21.15.110:0/3767631249 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f493400eed0 con 0x7f49440a44c0 2023-10-06T03:30:31.604 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:31.597+0000 7f49417fa700 1 -- 172.21.15.110:0/3767631249 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f493400c8e0 con 0x7f49440a44c0 2023-10-06T03:30:31.604 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:31.597+0000 7f494a70a700 1 -- 172.21.15.110:0/3767631249 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49440c7730 con 0x7f49440a44c0 2023-10-06T03:30:31.604 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:31.597+0000 7f49417fa700 1 -- 172.21.15.110:0/3767631249 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f493400cc80 con 0x7f49440a44c0 2023-10-06T03:30:31.604 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:31.597+0000 7f494a70a700 1 -- 172.21.15.110:0/3767631249 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49440c7cf0 con 0x7f49440a44c0 2023-10-06T03:30:31.604 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:31.597+0000 7f494a70a700 1 -- 172.21.15.110:0/3767631249 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4944070ff0 con 0x7f49440a44c0 2023-10-06T03:30:31.604 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:31.597+0000 7f49417fa700 1 -- 172.21.15.110:0/3767631249 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7f4934013680 con 0x7f49440a44c0 2023-10-06T03:30:31.604 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:31.601+0000 7f49417fa700 1 --2- 172.21.15.110:0/3767631249 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f4930060580 0x7f4930062a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:31.604 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:31.601+0000 7f49417fa700 1 -- 172.21.15.110:0/3767631249 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 5526+0+0 (secure 0 0 0) 0x7f493407c9d0 con 0x7f49440a44c0 2023-10-06T03:30:31.605 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:31.601+0000 7f4948f07700 1 --2- 172.21.15.110:0/3767631249 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f4930060580 0x7f4930062a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:31.605 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:31.601+0000 7f4948f07700 1 --2- 172.21.15.110:0/3767631249 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f4930060580 0x7f4930062a30 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f49440c87e0 tx=0x7f4938009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:30:31.608 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:31.605+0000 7f49417fa700 1 -- 172.21.15.110:0/3767631249 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f493404d050 con 0x7f49440a44c0 2023-10-06T03:30:31.770 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:31.765+0000 7f494a70a700 1 -- 172.21.15.110:0/3767631249 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f49440a5220 con 0x7f49440a44c0 2023-10-06T03:30:32.636 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:32.633+0000 7f49417fa700 1 -- 172.21.15.110:0/3767631249 <== mon.0 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v39) v1 ==== 138+0+0 (secure 0 0 0) 0x7f4934050670 con 0x7f49440a44c0 2023-10-06T03:30:32.642 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:32.637+0000 7f494a70a700 1 -- 172.21.15.110:0/3767631249 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f4930060580 msgr2=0x7f4930062a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:32.643 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:32.637+0000 7f494a70a700 1 --2- 172.21.15.110:0/3767631249 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f4930060580 0x7f4930062a30 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f49440c87e0 tx=0x7f4938009040 comp rx=0 tx=0).stop 2023-10-06T03:30:32.643 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:32.637+0000 7f494a70a700 1 -- 172.21.15.110:0/3767631249 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f49440a44c0 msgr2=0x7f49441490c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:32.643 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:32.637+0000 7f494a70a700 1 --2- 172.21.15.110:0/3767631249 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f49440a44c0 0x7f49441490c0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f493400e930 tx=0x7f49340091a0 comp rx=0 tx=0).stop 2023-10-06T03:30:32.643 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:32.637+0000 7f494a70a700 1 -- 172.21.15.110:0/3767631249 shutdown_connections 2023-10-06T03:30:32.643 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:32.637+0000 7f494a70a700 1 --2- 172.21.15.110:0/3767631249 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f4930060580 0x7f4930062a30 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:32.643 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:32.637+0000 7f494a70a700 1 --2- 172.21.15.110:0/3767631249 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f49440c9160 0x7f49440aef30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:32.643 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:32.637+0000 7f494a70a700 1 --2- 172.21.15.110:0/3767631249 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f49440b1890 0x7f49440ae9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:32.643 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:32.637+0000 7f494a70a700 1 --2- 172.21.15.110:0/3767631249 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f49440a44c0 0x7f49441490c0 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:32.643 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:32.637+0000 7f494a70a700 1 -- 172.21.15.110:0/3767631249 >> 172.21.15.110:0/3767631249 conn(0x7f49440670a0 msgr2=0x7f49440acfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:30:32.644 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:32.637+0000 7f494a70a700 1 -- 172.21.15.110:0/3767631249 shutdown_connections 2023-10-06T03:30:32.644 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:32.637+0000 7f494a70a700 1 -- 172.21.15.110:0/3767631249 wait complete. 2023-10-06T03:30:32.644 INFO:teuthology.orchestra.run.smithi110.stderr:pool 'cephfs_metadata' created 2023-10-06T03:30:32.655 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2023-10-06T03:30:32.770 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:32.765+0000 7fdbfded7700 1 Processor -- start 2023-10-06T03:30:32.770 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:32.765+0000 7fdbfded7700 1 -- start start 2023-10-06T03:30:32.771 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:32.765+0000 7fdbfded7700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fdbf80aef40 0x7fdbf80ce200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:32.771 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:32.765+0000 7fdbfded7700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fdbf815d210 0x7fdbf80afa60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:32.771 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:32.765+0000 7fdbfded7700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdbf80cf5f0 con 0x7fdbf80cb1b0 2023-10-06T03:30:32.772 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:32.765+0000 7fdbfded7700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdbf80ce740 con 0x7fdbf80aef40 2023-10-06T03:30:32.772 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:32.765+0000 7fdbfded7700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdbf80ce9e0 con 0x7fdbf815d210 2023-10-06T03:30:32.772 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:32.769+0000 7fdbfced5700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fdbf80aef40 0x7fdbf80ce200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:32.772 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:32.769+0000 7fdbfced5700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fdbf80aef40 0x7fdbf80ce200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:48036/0 (socket says 172.21.15.110:48036) 2023-10-06T03:30:32.772 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:32.769+0000 7fdbfced5700 1 -- 172.21.15.110:0/129244833 learned_addr learned my addr 172.21.15.110:0/129244833 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:30:32.772 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:32.769+0000 7fdbfd6d6700 1 --2- 172.21.15.110:0/129244833 >> v2:172.21.15.192:3300/0 conn(0x7fdbf815d210 0x7fdbf80afa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:32.773 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:32.769+0000 7fdbfced5700 1 -- 172.21.15.110:0/129244833 >> v2:172.21.15.192:3300/0 conn(0x7fdbf815d210 msgr2=0x7fdbf80afa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:30:32.773 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:32.769+0000 7fdbfced5700 1 --2- 172.21.15.110:0/129244833 >> v2:172.21.15.192:3300/0 conn(0x7fdbf815d210 0x7fdbf80afa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:32.773 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:32.769+0000 7fdbfced5700 1 -- 172.21.15.110:0/129244833 >> v1:172.21.15.192:6789/0 conn(0x7fdbf80cb1b0 legacy=0x7fdbf815cb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:32.773 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:32.769+0000 7fdbfced5700 1 -- 172.21.15.110:0/129244833 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbf80cecc0 con 0x7fdbf80aef40 2023-10-06T03:30:32.773 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:32.769+0000 7fdbfced5700 1 --2- 172.21.15.110:0/129244833 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdbf80aef40 0x7fdbf80ce200 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7fdbf80cb780 tx=0x7fdbe8009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c9e67d5d4edeb76 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:30:32.773 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:32.769+0000 7fdbf77fe700 1 -- 172.21.15.110:0/129244833 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbe800e070 con 0x7fdbf80aef40 2023-10-06T03:30:32.774 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:32.769+0000 7fdbf77fe700 1 -- 172.21.15.110:0/129244833 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdbe800ba20 con 0x7fdbf80aef40 2023-10-06T03:30:32.774 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:32.769+0000 7fdbf77fe700 1 -- 172.21.15.110:0/129244833 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbe80092e0 con 0x7fdbf80aef40 2023-10-06T03:30:32.775 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:32.769+0000 7fdbfded7700 1 -- 172.21.15.110:0/129244833 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdbf80aef40 msgr2=0x7fdbf80ce200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:32.775 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:32.769+0000 7fdbfded7700 1 --2- 172.21.15.110:0/129244833 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdbf80aef40 0x7fdbf80ce200 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7fdbf80cb780 tx=0x7fdbe8009840 comp rx=0 tx=0).stop 2023-10-06T03:30:32.775 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:32.769+0000 7fdbfded7700 1 -- 172.21.15.110:0/129244833 shutdown_connections 2023-10-06T03:30:32.775 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:32.769+0000 7fdbfded7700 1 --2- 172.21.15.110:0/129244833 >> v2:172.21.15.192:3300/0 conn(0x7fdbf815d210 0x7fdbf80afa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:32.775 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:32.769+0000 7fdbfded7700 1 --2- 172.21.15.110:0/129244833 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdbf80aef40 0x7fdbf80ce200 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:32.775 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:32.769+0000 7fdbfded7700 1 -- 172.21.15.110:0/129244833 >> 172.21.15.110:0/129244833 conn(0x7fdbf80cfec0 msgr2=0x7fdbf80d02c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:30:32.775 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:32.769+0000 7fdbfded7700 1 -- 172.21.15.110:0/129244833 shutdown_connections 2023-10-06T03:30:32.775 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:32.769+0000 7fdbfded7700 1 -- 172.21.15.110:0/129244833 wait complete. 2023-10-06T03:30:32.776 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:32.773+0000 7fdbfded7700 1 Processor -- start 2023-10-06T03:30:32.776 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:32.773+0000 7fdbfded7700 1 -- start start 2023-10-06T03:30:32.778 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:32.773+0000 7fdbfded7700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdbf80aef40 0x7fdbf8156f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:32.778 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:32.773+0000 7fdbfded7700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fdbf80cb1b0 0x7fdbf80c6f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:32.779 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:32.773+0000 7fdbfded7700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdbf815d210 0x7fdbf80c74b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:32.779 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:32.773+0000 7fdbfded7700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbf80ae560 con 0x7fdbf815d210 2023-10-06T03:30:32.779 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:32.773+0000 7fdbfded7700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbf80c7ae0 con 0x7fdbf80aef40 2023-10-06T03:30:32.779 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:32.773+0000 7fdbfded7700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdbf80c7dc0 con 0x7fdbf80cb1b0 2023-10-06T03:30:32.779 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:32.773+0000 7fdbfced5700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdbf80aef40 0x7fdbf8156f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:32.779 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:32.773+0000 7fdbfd6d6700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdbf815d210 0x7fdbf80c74b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:32.779 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:32.773+0000 7fdbfced5700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdbf80aef40 0x7fdbf8156f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.110:48544/0 (socket says 172.21.15.110:48544) 2023-10-06T03:30:32.779 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:32.773+0000 7fdbfced5700 1 -- 172.21.15.110:0/2367673218 learned_addr learned my addr 172.21.15.110:0/2367673218 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:30:32.779 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:32.773+0000 7fdbf7fff700 1 --2- 172.21.15.110:0/2367673218 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fdbf80cb1b0 0x7fdbf80c6f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:32.779 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:32.773+0000 7fdbfd6d6700 1 -- 172.21.15.110:0/2367673218 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fdbf80cb1b0 msgr2=0x7fdbf80c6f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:32.779 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:32.773+0000 7fdbfd6d6700 1 --2- 172.21.15.110:0/2367673218 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fdbf80cb1b0 0x7fdbf80c6f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:32.780 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:32.773+0000 7fdbfd6d6700 1 -- 172.21.15.110:0/2367673218 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdbf80aef40 msgr2=0x7fdbf8156f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:32.780 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:32.773+0000 7fdbfd6d6700 1 --2- 172.21.15.110:0/2367673218 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdbf80aef40 0x7fdbf8156f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:32.780 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:32.773+0000 7fdbfd6d6700 1 -- 172.21.15.110:0/2367673218 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbe8009ae0 con 0x7fdbf815d210 2023-10-06T03:30:32.781 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:32.773+0000 7fdbfd6d6700 1 --2- 172.21.15.110:0/2367673218 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdbf815d210 0x7fdbf80c74b0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7fdbf000e8d0 tx=0x7fdbf00091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:30:32.781 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:32.773+0000 7fdbf4ff9700 1 -- 172.21.15.110:0/2367673218 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbf0013680 con 0x7fdbf815d210 2023-10-06T03:30:32.781 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:32.777+0000 7fdbf4ff9700 1 -- 172.21.15.110:0/2367673218 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdbf00137e0 con 0x7fdbf815d210 2023-10-06T03:30:32.781 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:32.777+0000 7fdbf4ff9700 1 -- 172.21.15.110:0/2367673218 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbf0013ba0 con 0x7fdbf815d210 2023-10-06T03:30:32.781 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:32.777+0000 7fdbfded7700 1 -- 172.21.15.110:0/2367673218 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdbf80c80a0 con 0x7fdbf815d210 2023-10-06T03:30:32.781 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:32.777+0000 7fdbfded7700 1 -- 172.21.15.110:0/2367673218 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdbf80c8660 con 0x7fdbf815d210 2023-10-06T03:30:32.781 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:32.777+0000 7fdbfded7700 1 -- 172.21.15.110:0/2367673218 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdbf8070ff0 con 0x7fdbf815d210 2023-10-06T03:30:32.782 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:32.777+0000 7fdbf4ff9700 1 -- 172.21.15.110:0/2367673218 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7fdbf001c450 con 0x7fdbf815d210 2023-10-06T03:30:32.783 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:32.781+0000 7fdbf4ff9700 1 --2- 172.21.15.110:0/2367673218 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fdbd8060550 0x7fdbd8062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:32.783 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:32.781+0000 7fdbf4ff9700 1 -- 172.21.15.110:0/2367673218 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 5896+0+0 (secure 0 0 0) 0x7fdbf007cb60 con 0x7fdbf815d210 2023-10-06T03:30:32.783 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:32.781+0000 7fdbfced5700 1 --2- 172.21.15.110:0/2367673218 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fdbd8060550 0x7fdbd8062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:32.784 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:32.781+0000 7fdbfced5700 1 --2- 172.21.15.110:0/2367673218 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fdbd8060550 0x7fdbd8062a00 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fdbf8088890 tx=0x7fdbe8018000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:30:32.789 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:32.785+0000 7fdbf4ff9700 1 -- 172.21.15.110:0/2367673218 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fdbf004d090 con 0x7fdbf815d210 2023-10-06T03:30:32.952 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:32.949+0000 7fdbfded7700 1 -- 172.21.15.110:0/2367673218 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7fdbf80afff0 con 0x7fdbf815d210 2023-10-06T03:30:33.654 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:33.649+0000 7fdbf4ff9700 1 -- 172.21.15.110:0/2367673218 <== mon.0 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v40) v1 ==== 170+0+0 (secure 0 0 0) 0x7fdbf00506b0 con 0x7fdbf815d210 2023-10-06T03:30:33.671 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:33.665+0000 7fdbfded7700 1 -- 172.21.15.110:0/2367673218 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fdbd8060550 msgr2=0x7fdbd8062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:33.671 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:33.665+0000 7fdbfded7700 1 --2- 172.21.15.110:0/2367673218 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fdbd8060550 0x7fdbd8062a00 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fdbf8088890 tx=0x7fdbe8018000 comp rx=0 tx=0).stop 2023-10-06T03:30:33.671 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:33.665+0000 7fdbfded7700 1 -- 172.21.15.110:0/2367673218 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdbf815d210 msgr2=0x7fdbf80c74b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:33.672 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:33.665+0000 7fdbfded7700 1 --2- 172.21.15.110:0/2367673218 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdbf815d210 0x7fdbf80c74b0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7fdbf000e8d0 tx=0x7fdbf00091a0 comp rx=0 tx=0).stop 2023-10-06T03:30:33.672 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:33.669+0000 7fdbfded7700 1 -- 172.21.15.110:0/2367673218 shutdown_connections 2023-10-06T03:30:33.672 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:33.669+0000 7fdbfded7700 1 --2- 172.21.15.110:0/2367673218 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fdbd8060550 0x7fdbd8062a00 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:33.672 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:33.669+0000 7fdbfded7700 1 --2- 172.21.15.110:0/2367673218 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdbf815d210 0x7fdbf80c74b0 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:33.673 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:33.669+0000 7fdbfded7700 1 --2- 172.21.15.110:0/2367673218 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fdbf80cb1b0 0x7fdbf80c6f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:33.673 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:33.669+0000 7fdbfded7700 1 --2- 172.21.15.110:0/2367673218 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdbf80aef40 0x7fdbf8156f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:33.673 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:33.669+0000 7fdbfded7700 1 -- 172.21.15.110:0/2367673218 >> 172.21.15.110:0/2367673218 conn(0x7fdbf80cfec0 msgr2=0x7fdbf80b1370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:30:33.673 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:33.669+0000 7fdbfded7700 1 -- 172.21.15.110:0/2367673218 shutdown_connections 2023-10-06T03:30:33.673 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:33.669+0000 7fdbfded7700 1 -- 172.21.15.110:0/2367673218 wait complete. 2023-10-06T03:30:33.673 INFO:teuthology.orchestra.run.smithi110.stderr:pool 'cephfs_data' created 2023-10-06T03:30:33.684 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2023-10-06T03:30:33.798 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:33.793+0000 7fc363442700 1 Processor -- start 2023-10-06T03:30:33.798 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:33.793+0000 7fc363442700 1 -- start start 2023-10-06T03:30:33.800 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:33.793+0000 7fc363442700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc35c15d270 0x7fc35c15d640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:33.801 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:33.793+0000 7fc363442700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fc35c0c8780 0x7fc35c0c9ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:33.801 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:33.797+0000 7fc363442700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fc35c0cc980 0x7fc35c068ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:33.801 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:33.797+0000 7fc363442700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc35c0c9850 con 0x7fc35c0c8780 2023-10-06T03:30:33.801 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:33.797+0000 7fc363442700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc35c0c8c40 con 0x7fc35c0cc980 2023-10-06T03:30:33.801 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:33.797+0000 7fc363442700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc35c0c8f20 con 0x7fc35c15d270 2023-10-06T03:30:33.801 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:33.797+0000 7fc361c3f700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fc35c0c8780 0x7fc35c0c9ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:33.801 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:33.797+0000 7fc361c3f700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fc35c0c8780 0x7fc35c0c9ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:48042/0 (socket says 172.21.15.110:48042) 2023-10-06T03:30:33.801 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:33.797+0000 7fc361c3f700 1 -- 172.21.15.110:0/3827296002 learned_addr learned my addr 172.21.15.110:0/3827296002 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:30:33.801 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:33.797+0000 7fc362440700 1 --2- 172.21.15.110:0/3827296002 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc35c15d270 0x7fc35c15d640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:33.801 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:33.797+0000 7fc362c41700 1 --2- 172.21.15.110:0/3827296002 >> v2:172.21.15.192:3300/0 conn(0x7fc35c0cc980 0x7fc35c068ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:33.802 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:33.797+0000 7fc361c3f700 1 -- 172.21.15.110:0/3827296002 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc35c15d270 msgr2=0x7fc35c15d640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:30:33.802 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:33.797+0000 7fc361c3f700 1 --2- 172.21.15.110:0/3827296002 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc35c15d270 0x7fc35c15d640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:33.802 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:33.797+0000 7fc361c3f700 1 -- 172.21.15.110:0/3827296002 >> v2:172.21.15.192:3300/0 conn(0x7fc35c0cc980 msgr2=0x7fc35c068ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:30:33.802 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:33.797+0000 7fc361c3f700 1 --2- 172.21.15.110:0/3827296002 >> v2:172.21.15.192:3300/0 conn(0x7fc35c0cc980 0x7fc35c068ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:33.802 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:33.797+0000 7fc361c3f700 1 -- 172.21.15.110:0/3827296002 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc35c0c9200 con 0x7fc35c0c8780 2023-10-06T03:30:33.802 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:33.797+0000 7fc362440700 1 --2- 172.21.15.110:0/3827296002 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc35c15d270 0x7fc35c15d640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-06T03:30:33.802 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:33.797+0000 7fc362c41700 1 --2- 172.21.15.110:0/3827296002 >> v2:172.21.15.192:3300/0 conn(0x7fc35c0cc980 0x7fc35c068ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-06T03:30:33.802 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:33.797+0000 7fc361c3f700 1 --2- 172.21.15.110:0/3827296002 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc35c0c8780 0x7fc35c0c9ce0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7fc35c0c4d20 tx=0x7fc34c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc0cec31f89db121 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:30:33.802 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:33.797+0000 7fc36143e700 1 -- 172.21.15.110:0/3827296002 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc34c00b860 con 0x7fc35c0c8780 2023-10-06T03:30:33.802 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:33.797+0000 7fc36143e700 1 -- 172.21.15.110:0/3827296002 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc34c00b9c0 con 0x7fc35c0c8780 2023-10-06T03:30:33.802 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:33.797+0000 7fc36143e700 1 -- 172.21.15.110:0/3827296002 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc34c00bd80 con 0x7fc35c0c8780 2023-10-06T03:30:33.803 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:33.797+0000 7fc363442700 1 -- 172.21.15.110:0/3827296002 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc35c0c8780 msgr2=0x7fc35c0c9ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:33.803 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:33.797+0000 7fc363442700 1 --2- 172.21.15.110:0/3827296002 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc35c0c8780 0x7fc35c0c9ce0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7fc35c0c4d20 tx=0x7fc34c009250 comp rx=0 tx=0).stop 2023-10-06T03:30:33.803 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:33.797+0000 7fc363442700 1 -- 172.21.15.110:0/3827296002 shutdown_connections 2023-10-06T03:30:33.803 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:33.797+0000 7fc363442700 1 --2- 172.21.15.110:0/3827296002 >> v2:172.21.15.192:3300/0 conn(0x7fc35c0cc980 0x7fc35c068ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:33.804 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:33.797+0000 7fc363442700 1 --2- 172.21.15.110:0/3827296002 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc35c0c8780 0x7fc35c0c9ce0 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:33.804 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:33.797+0000 7fc363442700 1 --2- 172.21.15.110:0/3827296002 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc35c15d270 0x7fc35c15d640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:33.804 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:33.797+0000 7fc363442700 1 -- 172.21.15.110:0/3827296002 >> 172.21.15.110:0/3827296002 conn(0x7fc35c068230 msgr2=0x7fc35c0cdcf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:30:33.804 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:33.797+0000 7fc363442700 1 -- 172.21.15.110:0/3827296002 shutdown_connections 2023-10-06T03:30:33.804 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:33.797+0000 7fc363442700 1 -- 172.21.15.110:0/3827296002 wait complete. 2023-10-06T03:30:33.804 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:33.801+0000 7fc363442700 1 Processor -- start 2023-10-06T03:30:33.804 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:33.801+0000 7fc363442700 1 -- start start 2023-10-06T03:30:33.807 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:33.801+0000 7fc363442700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc35c0c8780 0x7fc35c145190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:33.807 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:33.801+0000 7fc363442700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc35c0cc980 0x7fc35c156480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:33.807 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:33.801+0000 7fc363442700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc35c15d270 0x7fc35c0cae00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:33.807 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:33.801+0000 7fc363442700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc35c1456d0 con 0x7fc35c15d270 2023-10-06T03:30:33.807 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:33.801+0000 7fc363442700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc35c152710 con 0x7fc35c0c8780 2023-10-06T03:30:33.807 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:33.801+0000 7fc363442700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc35c1529f0 con 0x7fc35c0cc980 2023-10-06T03:30:33.807 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:33.801+0000 7fc362c41700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc35c15d270 0x7fc35c0cae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:33.807 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:33.801+0000 7fc362c41700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc35c15d270 0x7fc35c0cae00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:48056/0 (socket says 172.21.15.110:48056) 2023-10-06T03:30:33.807 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:33.801+0000 7fc362440700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc35c0c8780 0x7fc35c145190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:33.807 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:33.801+0000 7fc362c41700 1 -- 172.21.15.110:0/3516047877 learned_addr learned my addr 172.21.15.110:0/3516047877 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:30:33.808 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:33.801+0000 7fc361c3f700 1 --2- 172.21.15.110:0/3516047877 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc35c0cc980 0x7fc35c156480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:33.808 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:33.801+0000 7fc362c41700 1 -- 172.21.15.110:0/3516047877 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc35c0cc980 msgr2=0x7fc35c156480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:33.808 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:33.801+0000 7fc362c41700 1 --2- 172.21.15.110:0/3516047877 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc35c0cc980 0x7fc35c156480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:33.808 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:33.801+0000 7fc362c41700 1 -- 172.21.15.110:0/3516047877 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc35c0c8780 msgr2=0x7fc35c145190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:33.808 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:33.801+0000 7fc362c41700 1 --2- 172.21.15.110:0/3516047877 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc35c0c8780 0x7fc35c145190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:33.808 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:33.801+0000 7fc362c41700 1 -- 172.21.15.110:0/3516047877 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc34c0094f0 con 0x7fc35c15d270 2023-10-06T03:30:33.808 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:33.801+0000 7fc362c41700 1 --2- 172.21.15.110:0/3516047877 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc35c15d270 0x7fc35c0cae00 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7fc35400d180 tx=0x7fc354009a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:30:33.809 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:33.805+0000 7fc3527fc700 1 -- 172.21.15.110:0/3516047877 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc35400dcc0 con 0x7fc35c15d270 2023-10-06T03:30:33.809 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:33.805+0000 7fc3527fc700 1 -- 172.21.15.110:0/3516047877 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc35400f040 con 0x7fc35c15d270 2023-10-06T03:30:33.809 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:33.805+0000 7fc363442700 1 -- 172.21.15.110:0/3516047877 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc35c152cd0 con 0x7fc35c15d270 2023-10-06T03:30:33.809 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:33.805+0000 7fc3527fc700 1 -- 172.21.15.110:0/3516047877 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc354015690 con 0x7fc35c15d270 2023-10-06T03:30:33.809 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:33.805+0000 7fc363442700 1 -- 172.21.15.110:0/3516047877 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc35c153290 con 0x7fc35c15d270 2023-10-06T03:30:33.810 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:33.805+0000 7fc363442700 1 -- 172.21.15.110:0/3516047877 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc35c153480 con 0x7fc35c15d270 2023-10-06T03:30:33.810 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:33.805+0000 7fc3527fc700 1 -- 172.21.15.110:0/3516047877 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7fc35401e3b0 con 0x7fc35c15d270 2023-10-06T03:30:33.810 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:33.805+0000 7fc3527fc700 1 --2- 172.21.15.110:0/3516047877 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fc340060550 0x7fc340062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:33.810 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:33.805+0000 7fc3527fc700 1 -- 172.21.15.110:0/3516047877 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6278+0+0 (secure 0 0 0) 0x7fc35407cf30 con 0x7fc35c15d270 2023-10-06T03:30:33.811 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:33.805+0000 7fc362440700 1 --2- 172.21.15.110:0/3516047877 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fc340060550 0x7fc340062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:33.811 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:33.809+0000 7fc362440700 1 --2- 172.21.15.110:0/3516047877 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fc340060550 0x7fc340062a00 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fc358006fd0 tx=0x7fc358008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:30:33.818 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:33.813+0000 7fc3527fc700 1 -- 172.21.15.110:0/3516047877 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fc35404d2e0 con 0x7fc35c15d270 2023-10-06T03:30:34.014 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:34.009+0000 7fc363442700 1 -- 172.21.15.110:0/3516047877 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7fc35c070ff0 con 0x7fc35c15d270 2023-10-06T03:30:34.677 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:34.672+0000 7fc3527fc700 1 -- 172.21.15.110:0/3516047877 <== mon.0 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 6 and data pool 7 v13) v1 ==== 172+0+0 (secure 0 0 0) 0x7fc35401a0c0 con 0x7fc35c15d270 2023-10-06T03:30:34.684 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:34.680+0000 7fc363442700 1 -- 172.21.15.110:0/3516047877 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fc340060550 msgr2=0x7fc340062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:34.684 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:34.680+0000 7fc363442700 1 --2- 172.21.15.110:0/3516047877 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fc340060550 0x7fc340062a00 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fc358006fd0 tx=0x7fc358008040 comp rx=0 tx=0).stop 2023-10-06T03:30:34.684 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:34.680+0000 7fc363442700 1 -- 172.21.15.110:0/3516047877 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc35c15d270 msgr2=0x7fc35c0cae00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:34.684 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:34.680+0000 7fc363442700 1 --2- 172.21.15.110:0/3516047877 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc35c15d270 0x7fc35c0cae00 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7fc35400d180 tx=0x7fc354009a50 comp rx=0 tx=0).stop 2023-10-06T03:30:34.689 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:34.684+0000 7fc363442700 1 -- 172.21.15.110:0/3516047877 shutdown_connections 2023-10-06T03:30:34.689 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:34.684+0000 7fc363442700 1 --2- 172.21.15.110:0/3516047877 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fc340060550 0x7fc340062a00 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:34.689 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:34.684+0000 7fc363442700 1 --2- 172.21.15.110:0/3516047877 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc35c15d270 0x7fc35c0cae00 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:34.689 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:34.684+0000 7fc363442700 1 --2- 172.21.15.110:0/3516047877 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc35c0cc980 0x7fc35c156480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:34.689 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:34.684+0000 7fc363442700 1 --2- 172.21.15.110:0/3516047877 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc35c0c8780 0x7fc35c145190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:34.689 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:34.684+0000 7fc363442700 1 -- 172.21.15.110:0/3516047877 >> 172.21.15.110:0/3516047877 conn(0x7fc35c068230 msgr2=0x7fc35c0ca630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:30:34.689 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:34.684+0000 7fc363442700 1 -- 172.21.15.110:0/3516047877 shutdown_connections 2023-10-06T03:30:34.689 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:34.684+0000 7fc363442700 1 -- 172.21.15.110:0/3516047877 wait complete. 2023-10-06T03:30:34.690 INFO:teuthology.orchestra.run.smithi110.stderr:new fs with metadata pool 6 and data pool 7 2023-10-06T03:30:34.703 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-06T03:30:34.800 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:34.796+0000 7f9566f78700 1 Processor -- start 2023-10-06T03:30:34.800 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:34.796+0000 7f9566f78700 1 -- start start 2023-10-06T03:30:34.801 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:34.796+0000 7f9566f78700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9560156b80 0x7f95600c68d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:34.801 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:34.796+0000 7f9566f78700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95600c7f30 con 0x7f95600c6e10 2023-10-06T03:30:34.801 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:34.796+0000 7f9566f78700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95600c7320 con 0x7f9560156f50 2023-10-06T03:30:34.801 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:34.796+0000 7f9566f78700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95600c7600 con 0x7f9560156b80 2023-10-06T03:30:34.801 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:34.796+0000 7f9566777700 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f95600c6e10 0x7f95600ad560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:54600/0 (socket says 172.21.15.110:54600) 2023-10-06T03:30:34.801 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:34.796+0000 7f9566777700 1 -- 172.21.15.110:0/2806587505 learned_addr learned my addr 172.21.15.110:0/2806587505 (peer_addr_for_me v1:172.21.15.110:0/0) 2023-10-06T03:30:34.803 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:34.796+0000 7f9565f76700 1 --2- 172.21.15.110:0/2806587505 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9560156b80 0x7f95600c68d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:34.803 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:34.796+0000 7f9564f74700 1 -- 172.21.15.110:0/2806587505 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3353142380 0 0) 0x7f95600c7f30 con 0x7f95600c6e10 2023-10-06T03:30:34.803 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:34.796+0000 7f9564f74700 1 -- 172.21.15.110:0/2806587505 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f954c003570 con 0x7f95600c6e10 2023-10-06T03:30:34.804 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:34.796+0000 7f9564f74700 1 -- 172.21.15.110:0/2806587505 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 235099125 0 0) 0x7f95600c7320 con 0x7f9560156f50 2023-10-06T03:30:34.804 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:34.796+0000 7f9564f74700 1 -- 172.21.15.110:0/2806587505 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f95600c7f30 con 0x7f9560156f50 2023-10-06T03:30:34.804 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:34.796+0000 7f9564f74700 1 -- 172.21.15.110:0/2806587505 <== mon.0 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2078004338 0 0) 0x7f954c003570 con 0x7f95600c6e10 2023-10-06T03:30:34.804 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:34.796+0000 7f9564f74700 1 -- 172.21.15.110:0/2806587505 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9560156b80 msgr2=0x7f95600c68d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:30:34.804 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:34.796+0000 7f9564f74700 1 --2- 172.21.15.110:0/2806587505 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9560156b80 0x7f95600c68d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:34.804 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:34.796+0000 7f9564f74700 1 -- 172.21.15.110:0/2806587505 >> v1:172.21.15.192:6789/0 conn(0x7f9560156f50 legacy=0x7f95600b0e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:34.804 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:34.796+0000 7f9564f74700 1 -- 172.21.15.110:0/2806587505 --> v1:172.21.15.110:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95600c78e0 con 0x7f95600c6e10 2023-10-06T03:30:34.805 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:34.796+0000 7f9564f74700 1 -- 172.21.15.110:0/2806587505 <== mon.0 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 118603703 0 0) 0x7f9558001470 con 0x7f95600c6e10 2023-10-06T03:30:34.805 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:34.796+0000 7f9564f74700 1 -- 172.21.15.110:0/2806587505 >> v1:172.21.15.110:6789/0 conn(0x7f95600c6e10 legacy=0x7f95600ad560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:34.805 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:34.796+0000 7f9564f74700 1 --2- 172.21.15.110:0/2806587505 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9560156b80 0x7f954c003c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:34.805 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:34.796+0000 7f9564f74700 1 --2- 172.21.15.110:0/2806587505 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f954c004180 0x7f954c004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:34.805 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:34.796+0000 7f9564f74700 1 --2- 172.21.15.110:0/2806587505 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f954c004d50 0x7f954c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:34.805 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:34.796+0000 7f9564f74700 1 -- 172.21.15.110:0/2806587505 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95600c7600 con 0x7f954c004d50 2023-10-06T03:30:34.805 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:34.796+0000 7f9564f74700 1 -- 172.21.15.110:0/2806587505 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f954c00bba0 con 0x7f954c004180 2023-10-06T03:30:34.805 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:34.796+0000 7f9564f74700 1 -- 172.21.15.110:0/2806587505 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f954c00bd60 con 0x7f9560156b80 2023-10-06T03:30:34.805 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:34.796+0000 7f9566777700 1 --2- 172.21.15.110:0/2806587505 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f954c004d50 0x7f954c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:34.806 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:34.796+0000 7f9565f76700 1 --2- 172.21.15.110:0/2806587505 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9560156b80 0x7f954c003c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:34.806 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:34.796+0000 7f9565775700 1 --2- 172.21.15.110:0/2806587505 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f954c004180 0x7f954c004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:34.806 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:34.796+0000 7f9566777700 1 -- 172.21.15.110:0/2806587505 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9560156b80 msgr2=0x7f954c003c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:30:34.806 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:34.796+0000 7f9566777700 1 --2- 172.21.15.110:0/2806587505 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9560156b80 0x7f954c003c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:34.806 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:34.796+0000 7f9566777700 1 -- 172.21.15.110:0/2806587505 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f954c004180 msgr2=0x7f954c004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:30:34.806 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:34.796+0000 7f9566777700 1 --2- 172.21.15.110:0/2806587505 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f954c004180 0x7f954c004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:34.806 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:34.796+0000 7f9566777700 1 -- 172.21.15.110:0/2806587505 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f954c003570 con 0x7f954c004d50 2023-10-06T03:30:34.806 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:34.796+0000 7f9566777700 1 --2- 172.21.15.110:0/2806587505 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f954c004d50 0x7f954c0051c0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f955800c2a0 tx=0x7f95600c7ea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca83253ba5f995ea server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:30:34.806 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:34.800+0000 7f9564f74700 1 -- 172.21.15.110:0/2806587505 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f955800cb70 con 0x7f954c004d50 2023-10-06T03:30:34.806 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:34.800+0000 7f9564f74700 1 -- 172.21.15.110:0/2806587505 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f955800ccf0 con 0x7f954c004d50 2023-10-06T03:30:34.806 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:34.800+0000 7f9564f74700 1 -- 172.21.15.110:0/2806587505 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9558005b70 con 0x7f954c004d50 2023-10-06T03:30:34.808 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:34.800+0000 7f9566f78700 1 -- 172.21.15.110:0/2806587505 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f954c004d50 msgr2=0x7f954c0051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:34.808 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:34.800+0000 7f9566f78700 1 --2- 172.21.15.110:0/2806587505 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f954c004d50 0x7f954c0051c0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f955800c2a0 tx=0x7f95600c7ea0 comp rx=0 tx=0).stop 2023-10-06T03:30:34.809 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:34.800+0000 7f9566777700 1 -- 172.21.15.110:0/2806587505 reap_dead start 2023-10-06T03:30:34.809 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:34.800+0000 7f9566f78700 1 -- 172.21.15.110:0/2806587505 shutdown_connections 2023-10-06T03:30:34.809 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:34.800+0000 7f9566f78700 1 -- 172.21.15.110:0/2806587505 >> 172.21.15.110:0/2806587505 conn(0x7f95600cc470 msgr2=0x7f95600c8410 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:30:34.809 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:34.800+0000 7f9566f78700 1 -- 172.21.15.110:0/2806587505 shutdown_connections 2023-10-06T03:30:34.809 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:34.800+0000 7f9566f78700 1 -- 172.21.15.110:0/2806587505 wait complete. 2023-10-06T03:30:34.809 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:34.800+0000 7f9566f78700 1 Processor -- start 2023-10-06T03:30:34.810 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:34.800+0000 7f9566f78700 1 -- start start 2023-10-06T03:30:34.810 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:34.800+0000 7f9566f78700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9560153a50 0x7f956006a470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:34.810 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:34.800+0000 7f9566f78700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9560153e20 0x7f956006a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:34.810 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:34.800+0000 7f9566f78700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f95600c42b0 0x7f95600c4730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:34.810 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:34.800+0000 7f9566f78700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95600bf4f0 con 0x7f95600c42b0 2023-10-06T03:30:34.810 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:34.800+0000 7f9566f78700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95600bf7d0 con 0x7f9560153e20 2023-10-06T03:30:34.810 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:34.800+0000 7f9566f78700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95600bfab0 con 0x7f9560153a50 2023-10-06T03:30:34.810 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:34.800+0000 7f9566777700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f95600c42b0 0x7f95600c4730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:34.810 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:34.800+0000 7f9566777700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f95600c42b0 0x7f95600c4730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:48080/0 (socket says 172.21.15.110:48080) 2023-10-06T03:30:34.810 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:34.800+0000 7f9566777700 1 -- 172.21.15.110:0/1444012863 learned_addr learned my addr 172.21.15.110:0/1444012863 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:30:34.810 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:34.800+0000 7f9565775700 1 --2- 172.21.15.110:0/1444012863 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9560153e20 0x7f956006a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:34.811 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:34.800+0000 7f9565f76700 1 --2- 172.21.15.110:0/1444012863 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9560153a50 0x7f956006a470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:34.811 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:34.800+0000 7f9566777700 1 -- 172.21.15.110:0/1444012863 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9560153a50 msgr2=0x7f956006a470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:34.811 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:34.800+0000 7f9566777700 1 --2- 172.21.15.110:0/1444012863 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9560153a50 0x7f956006a470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:34.811 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:34.800+0000 7f9566777700 1 -- 172.21.15.110:0/1444012863 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9560153e20 msgr2=0x7f956006a9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:34.811 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:34.800+0000 7f9566777700 1 --2- 172.21.15.110:0/1444012863 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9560153e20 0x7f956006a9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:34.811 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:34.800+0000 7f9566777700 1 -- 172.21.15.110:0/1444012863 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95600bfd90 con 0x7f95600c42b0 2023-10-06T03:30:34.811 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:34.804+0000 7f9565f76700 1 --2- 172.21.15.110:0/1444012863 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9560153a50 0x7f956006a470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-06T03:30:34.811 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:34.804+0000 7f9565775700 1 --2- 172.21.15.110:0/1444012863 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9560153e20 0x7f956006a9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-06T03:30:34.811 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:34.804+0000 7f9566777700 1 --2- 172.21.15.110:0/1444012863 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f95600c42b0 0x7f95600c4730 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f9558007c90 tx=0x7f9558015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:30:34.811 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:34.804+0000 7f9555ffb700 1 -- 172.21.15.110:0/1444012863 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f955800ce60 con 0x7f95600c42b0 2023-10-06T03:30:34.811 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:34.804+0000 7f9555ffb700 1 -- 172.21.15.110:0/1444012863 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9558004e10 con 0x7f95600c42b0 2023-10-06T03:30:34.811 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:34.804+0000 7f9566f78700 1 -- 172.21.15.110:0/1444012863 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95600c0070 con 0x7f95600c42b0 2023-10-06T03:30:34.812 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:34.804+0000 7f9555ffb700 1 -- 172.21.15.110:0/1444012863 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95580072e0 con 0x7f95600c42b0 2023-10-06T03:30:34.812 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:34.804+0000 7f9566f78700 1 -- 172.21.15.110:0/1444012863 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f95600c29b0 con 0x7f95600c42b0 2023-10-06T03:30:34.812 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:34.804+0000 7f9566f78700 1 -- 172.21.15.110:0/1444012863 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9560070ff0 con 0x7f95600c42b0 2023-10-06T03:30:34.812 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:34.804+0000 7f9555ffb700 1 -- 172.21.15.110:0/1444012863 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7f9558007710 con 0x7f95600c42b0 2023-10-06T03:30:34.813 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:34.804+0000 7f9555ffb700 1 --2- 172.21.15.110:0/1444012863 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f9544060580 0x7f9544062a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:34.813 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:34.804+0000 7f9555ffb700 1 -- 172.21.15.110:0/1444012863 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6378+0+0 (secure 0 0 0) 0x7f955807ec00 con 0x7f95600c42b0 2023-10-06T03:30:34.813 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:34.804+0000 7f9565f76700 1 --2- 172.21.15.110:0/1444012863 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f9544060580 0x7f9544062a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:34.813 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:34.808+0000 7f9565f76700 1 --2- 172.21.15.110:0/1444012863 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f9544060580 0x7f9544062a30 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f9550009d50 tx=0x7f9550008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:30:34.816 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:34.812+0000 7f9555ffb700 1 -- 172.21.15.110:0/1444012863 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f955804eeb0 con 0x7f95600c42b0 2023-10-06T03:30:34.978 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:34.972+0000 7f9566f78700 1 -- 172.21.15.110:0/1444012863 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f95600c7bc0 con 0x7f95600c42b0 2023-10-06T03:30:34.980 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:34.976+0000 7f9555ffb700 1 -- 172.21.15.110:0/1444012863 <== mon.0 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v41) v1 ==== 74+0+17132 (secure 0 0 0) 0x7f9558017020 con 0x7f95600c42b0 2023-10-06T03:30:34.980 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-10-06T03:30:34.980 INFO:teuthology.orchestra.run.smithi110.stdout:{"epoch":41,"fsid":"93b8f6ac-4c92-4b45-b8e4-b59617024aa2","created":"2023-10-06T03:29:10.519583+0000","modified":"2023-10-06T03:30:34.013838+0000","last_up_change":"2023-10-06T03:29:21.960511+0000","last_in_change":"2023-10-06T03:29:12.560436+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-06T03:29:20.703743+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2023-10-06T03:29:28.307281+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2023-10-06T03:30:31.807897+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"41","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":7,"pool_name":"cephfs_data","create_time":"2023-10-06T03:30:32.991334+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"41","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"5866eb6c-2adc-46e4-be7a-e5bcda0f4065","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6824","nonce":33394},{"type":"v1","addr":"172.21.15.110:6825","nonce":33394}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6826","nonce":33394},{"type":"v1","addr":"172.21.15.110:6827","nonce":33394}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6830","nonce":33394},{"type":"v1","addr":"172.21.15.110:6831","nonce":33394}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6828","nonce":33394},{"type":"v1","addr":"172.21.15.110:6829","nonce":33394}]},"public_addr":"172.21.15.110:6825/33394","cluster_addr":"172.21.15.110:6827/33394","heartbeat_back_addr":"172.21.15.110:6831/33394","heartbeat_front_addr":"172.21.15.110:6829/33394","state":["exists","up"]},{"osd":1,"uuid":"a38de118-062b-4a05-9782-2ceddf6ff4df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6800","nonce":33396},{"type":"v1","addr":"172.21.15.110:6801","nonce":33396}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6802","nonce":33396},{"type":"v1","addr":"172.21.15.110:6803","nonce":33396}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6806","nonce":33396},{"type":"v1","addr":"172.21.15.110:6807","nonce":33396}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6804","nonce":33396},{"type":"v1","addr":"172.21.15.110:6805","nonce":33396}]},"public_addr":"172.21.15.110:6801/33396","cluster_addr":"172.21.15.110:6803/33396","heartbeat_back_addr":"172.21.15.110:6807/33396","heartbeat_front_addr":"172.21.15.110:6805/33396","state":["exists","up"]},{"osd":2,"uuid":"56e39be2-3407-4b69-877e-cf16617b06d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6816","nonce":33397},{"type":"v1","addr":"172.21.15.110:6817","nonce":33397}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6818","nonce":33397},{"type":"v1","addr":"172.21.15.110:6819","nonce":33397}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6822","nonce":33397},{"type":"v1","addr":"172.21.15.110:6823","nonce":33397}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6820","nonce":33397},{"type":"v1","addr":"172.21.15.110:6821","nonce":33397}]},"public_addr":"172.21.15.110:6817/33397","cluster_addr":"172.21.15.110:6819/33397","heartbeat_back_addr":"172.21.15.110:6823/33397","heartbeat_front_addr":"172.21.15.110:6821/33397","state":["exists","up"]},{"osd":3,"uuid":"01c82d13-c1d4-4bde-b55e-3e0089b40115","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6808","nonce":33395},{"type":"v1","addr":"172.21.15.110:6809","nonce":33395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6810","nonce":33395},{"type":"v1","addr":"172.21.15.110:6811","nonce":33395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6814","nonce":33395},{"type":"v1","addr":"172.21.15.110:6815","nonce":33395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6812","nonce":33395},{"type":"v1","addr":"172.21.15.110:6813","nonce":33395}]},"public_addr":"172.21.15.110:6809/33395","cluster_addr":"172.21.15.110:6811/33395","heartbeat_back_addr":"172.21.15.110:6815/33395","heartbeat_front_addr":"172.21.15.110:6813/33395","state":["exists","up"]},{"osd":4,"uuid":"86983430-5fa3-4d17-998f-03e2cd843237","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":33683},{"type":"v1","addr":"172.21.15.192:6817","nonce":33683}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":33683},{"type":"v1","addr":"172.21.15.192:6819","nonce":33683}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":33683},{"type":"v1","addr":"172.21.15.192:6823","nonce":33683}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6820","nonce":33683},{"type":"v1","addr":"172.21.15.192:6821","nonce":33683}]},"public_addr":"172.21.15.192:6817/33683","cluster_addr":"172.21.15.192:6819/33683","heartbeat_back_addr":"172.21.15.192:6823/33683","heartbeat_front_addr":"172.21.15.192:6821/33683","state":["exists","up"]},{"osd":5,"uuid":"6945257f-821d-4619-a63f-ebe857eba6f4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":33684},{"type":"v1","addr":"172.21.15.192:6801","nonce":33684}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":33684},{"type":"v1","addr":"172.21.15.192:6803","nonce":33684}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":33684},{"type":"v1","addr":"172.21.15.192:6807","nonce":33684}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6804","nonce":33684},{"type":"v1","addr":"172.21.15.192:6805","nonce":33684}]},"public_addr":"172.21.15.192:6801/33684","cluster_addr":"172.21.15.192:6803/33684","heartbeat_back_addr":"172.21.15.192:6807/33684","heartbeat_front_addr":"172.21.15.192:6805/33684","state":["exists","up"]},{"osd":6,"uuid":"7854d1ae-7060-4780-8f97-73cea5e7a12f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6824","nonce":33686},{"type":"v1","addr":"172.21.15.192:6825","nonce":33686}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":33686},{"type":"v1","addr":"172.21.15.192:6827","nonce":33686}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":33686},{"type":"v1","addr":"172.21.15.192:6831","nonce":33686}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6828","nonce":33686},{"type":"v1","addr":"172.21.15.192:6829","nonce":33686}]},"public_addr":"172.21.15.192:6825/33686","cluster_addr":"172.21.15.192:6827/33686","heartbeat_back_addr":"172.21.15.192:6831/33686","heartbeat_front_addr":"172.21.15.192:6829/33686","state":["exists","up"]},{"osd":7,"uuid":"f1ebdfd5-b859-482b-9f97-c18c4a26f445","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6808","nonce":33685},{"type":"v1","addr":"172.21.15.192:6809","nonce":33685}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":33685},{"type":"v1","addr":"172.21.15.192:6811","nonce":33685}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":33685},{"type":"v1","addr":"172.21.15.192:6815","nonce":33685}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6812","nonce":33685},{"type":"v1","addr":"172.21.15.192:6813","nonce":33685}]},"public_addr":"172.21.15.192:6809/33685","cluster_addr":"172.21.15.192:6811/33685","heartbeat_back_addr":"172.21.15.192:6815/33685","heartbeat_front_addr":"172.21.15.192:6813/33685","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T03:29:18.002048+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T03:29:17.478576+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T03:29:18.034944+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T03:29:17.628513+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T03:29:18.047338+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T03:29:17.772349+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T03:29:17.883958+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T03:29:18.151838+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-06T03:30:34.982 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:34.976+0000 7f9566f78700 1 -- 172.21.15.110:0/1444012863 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f9544060580 msgr2=0x7f9544062a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:34.983 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:34.976+0000 7f9566f78700 1 --2- 172.21.15.110:0/1444012863 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f9544060580 0x7f9544062a30 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f9550009d50 tx=0x7f9550008040 comp rx=0 tx=0).stop 2023-10-06T03:30:34.983 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:34.976+0000 7f9566f78700 1 -- 172.21.15.110:0/1444012863 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f95600c42b0 msgr2=0x7f95600c4730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:34.983 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:34.976+0000 7f9566f78700 1 --2- 172.21.15.110:0/1444012863 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f95600c42b0 0x7f95600c4730 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f9558007c90 tx=0x7f9558015000 comp rx=0 tx=0).stop 2023-10-06T03:30:34.983 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:34.976+0000 7f9566f78700 1 -- 172.21.15.110:0/1444012863 shutdown_connections 2023-10-06T03:30:34.983 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:34.976+0000 7f9566f78700 1 --2- 172.21.15.110:0/1444012863 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f9544060580 0x7f9544062a30 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:34.983 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:34.976+0000 7f9566f78700 1 --2- 172.21.15.110:0/1444012863 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f95600c42b0 0x7f95600c4730 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:34.983 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:34.976+0000 7f9566f78700 1 --2- 172.21.15.110:0/1444012863 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9560153e20 0x7f956006a9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:34.983 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:34.976+0000 7f9566f78700 1 --2- 172.21.15.110:0/1444012863 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9560153a50 0x7f956006a470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:34.983 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:34.976+0000 7f9566f78700 1 -- 172.21.15.110:0/1444012863 >> 172.21.15.110:0/1444012863 conn(0x7f95600cc470 msgr2=0x7f95600c94d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:30:34.983 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:34.976+0000 7f9566f78700 1 -- 172.21.15.110:0/1444012863 shutdown_connections 2023-10-06T03:30:34.983 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:34.976+0000 7f9566f78700 1 -- 172.21.15.110:0/1444012863 wait complete. 2023-10-06T03:30:34.994 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-06T03:30:35.102 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:35.096+0000 7fd13252b700 1 Processor -- start 2023-10-06T03:30:35.103 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:35.096+0000 7fd13252b700 1 -- start start 2023-10-06T03:30:35.104 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:35.096+0000 7fd13252b700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd12c156f50 0x7fd12c0b0e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:35.104 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:35.100+0000 7fd13252b700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fd12c0b1380 0x7fd12c0ad560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:35.104 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:35.100+0000 7fd13252b700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd12c0c7f30 con 0x7fd12c156b80 2023-10-06T03:30:35.104 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:35.100+0000 7fd13252b700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd12c0c7320 con 0x7fd12c0b1380 2023-10-06T03:30:35.104 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:35.100+0000 7fd13252b700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd12c0c7600 con 0x7fd12c156f50 2023-10-06T03:30:35.104 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:35.100+0000 7fd131d2a700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fd12c0b1380 0x7fd12c0ad560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:35.104 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:35.100+0000 7fd131d2a700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fd12c0b1380 0x7fd12c0ad560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:48092/0 (socket says 172.21.15.110:48092) 2023-10-06T03:30:35.105 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:35.100+0000 7fd131d2a700 1 -- 172.21.15.110:0/2432079111 learned_addr learned my addr 172.21.15.110:0/2432079111 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:30:35.105 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:35.100+0000 7fd130d28700 1 --2- 172.21.15.110:0/2432079111 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd12c156f50 0x7fd12c0b0e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:35.105 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:35.100+0000 7fd131d2a700 1 -- 172.21.15.110:0/2432079111 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd12c156f50 msgr2=0x7fd12c0b0e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:30:35.105 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:35.100+0000 7fd131d2a700 1 --2- 172.21.15.110:0/2432079111 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd12c156f50 0x7fd12c0b0e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:35.105 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:35.100+0000 7fd131d2a700 1 -- 172.21.15.110:0/2432079111 >> v1:172.21.15.192:6789/0 conn(0x7fd12c156b80 legacy=0x7fd12c0c68d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:35.105 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:35.100+0000 7fd131d2a700 1 -- 172.21.15.110:0/2432079111 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd12c0c78e0 con 0x7fd12c0b1380 2023-10-06T03:30:35.105 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:35.100+0000 7fd131d2a700 1 --2- 172.21.15.110:0/2432079111 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd12c0b1380 0x7fd12c0ad560 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7fd124009f20 tx=0x7fd124009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aad0ce71ba027db6 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:30:35.106 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:35.100+0000 7fd123fff700 1 -- 172.21.15.110:0/2432079111 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd12400e070 con 0x7fd12c0b1380 2023-10-06T03:30:35.106 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:35.100+0000 7fd123fff700 1 -- 172.21.15.110:0/2432079111 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd12400ba80 con 0x7fd12c0b1380 2023-10-06T03:30:35.106 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:35.100+0000 7fd123fff700 1 -- 172.21.15.110:0/2432079111 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1240092e0 con 0x7fd12c0b1380 2023-10-06T03:30:35.106 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:35.100+0000 7fd13252b700 1 -- 172.21.15.110:0/2432079111 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd12c0b1380 msgr2=0x7fd12c0ad560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:35.106 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:35.100+0000 7fd13252b700 1 --2- 172.21.15.110:0/2432079111 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd12c0b1380 0x7fd12c0ad560 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7fd124009f20 tx=0x7fd124009840 comp rx=0 tx=0).stop 2023-10-06T03:30:35.106 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:35.100+0000 7fd13252b700 1 -- 172.21.15.110:0/2432079111 shutdown_connections 2023-10-06T03:30:35.106 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:35.100+0000 7fd13252b700 1 --2- 172.21.15.110:0/2432079111 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd12c0b1380 0x7fd12c0ad560 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:35.107 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:35.100+0000 7fd13252b700 1 --2- 172.21.15.110:0/2432079111 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd12c156f50 0x7fd12c0b0e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:35.107 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:35.100+0000 7fd13252b700 1 -- 172.21.15.110:0/2432079111 >> 172.21.15.110:0/2432079111 conn(0x7fd12c0cc470 msgr2=0x7fd12c0c8410 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:30:35.107 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:35.100+0000 7fd13252b700 1 -- 172.21.15.110:0/2432079111 shutdown_connections 2023-10-06T03:30:35.107 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:35.100+0000 7fd13252b700 1 -- 172.21.15.110:0/2432079111 wait complete. 2023-10-06T03:30:35.107 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:35.100+0000 7fd13252b700 1 Processor -- start 2023-10-06T03:30:35.107 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:35.100+0000 7fd13252b700 1 -- start start 2023-10-06T03:30:35.108 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:35.104+0000 7fd13252b700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd12c0b1380 0x7fd12c0c3370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:35.108 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:35.104+0000 7fd13252b700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd12c156b80 0x7fd12c0c4ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:35.108 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:35.104+0000 7fd13252b700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd12c156f50 0x7fd12c151110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:35.108 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:35.104+0000 7fd13252b700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd12c0c39a0 con 0x7fd12c0b1380 2023-10-06T03:30:35.109 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:35.104+0000 7fd13252b700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd12c0c3c80 con 0x7fd12c156b80 2023-10-06T03:30:35.109 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:35.104+0000 7fd13252b700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd12c0c3f60 con 0x7fd12c156f50 2023-10-06T03:30:35.109 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:35.104+0000 7fd131529700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd12c0b1380 0x7fd12c0c3370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:35.109 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:35.104+0000 7fd131529700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd12c0b1380 0x7fd12c0c3370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:48106/0 (socket says 172.21.15.110:48106) 2023-10-06T03:30:35.109 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:35.104+0000 7fd130d28700 1 --2- 172.21.15.110:0/3730894314 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd12c156b80 0x7fd12c0c4ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:35.109 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:35.104+0000 7fd131529700 1 -- 172.21.15.110:0/3730894314 learned_addr learned my addr 172.21.15.110:0/3730894314 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:30:35.109 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:35.104+0000 7fd131d2a700 1 --2- 172.21.15.110:0/3730894314 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd12c156f50 0x7fd12c151110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:35.110 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:35.104+0000 7fd131529700 1 -- 172.21.15.110:0/3730894314 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd12c156f50 msgr2=0x7fd12c151110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:35.110 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:35.104+0000 7fd131529700 1 --2- 172.21.15.110:0/3730894314 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd12c156f50 0x7fd12c151110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:35.110 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:35.104+0000 7fd131529700 1 -- 172.21.15.110:0/3730894314 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd12c156b80 msgr2=0x7fd12c0c4ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:35.110 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:35.104+0000 7fd131529700 1 --2- 172.21.15.110:0/3730894314 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd12c156b80 0x7fd12c0c4ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:35.110 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:35.104+0000 7fd131529700 1 -- 172.21.15.110:0/3730894314 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd124009ae0 con 0x7fd12c0b1380 2023-10-06T03:30:35.110 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:35.104+0000 7fd131529700 1 --2- 172.21.15.110:0/3730894314 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd12c0b1380 0x7fd12c0c3370 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7fd12800c590 tx=0x7fd12800c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:30:35.111 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:35.104+0000 7fd1217fa700 1 -- 172.21.15.110:0/3730894314 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd128011590 con 0x7fd12c0b1380 2023-10-06T03:30:35.111 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:35.104+0000 7fd1217fa700 1 -- 172.21.15.110:0/3730894314 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd128011710 con 0x7fd12c0b1380 2023-10-06T03:30:35.111 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:35.104+0000 7fd13252b700 1 -- 172.21.15.110:0/3730894314 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd12c0c4240 con 0x7fd12c0b1380 2023-10-06T03:30:35.111 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:35.104+0000 7fd1217fa700 1 -- 172.21.15.110:0/3730894314 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd128011ad0 con 0x7fd12c0b1380 2023-10-06T03:30:35.111 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:35.104+0000 7fd13252b700 1 -- 172.21.15.110:0/3730894314 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd12c0c4520 con 0x7fd12c0b1380 2023-10-06T03:30:35.111 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:35.104+0000 7fd13252b700 1 -- 172.21.15.110:0/3730894314 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd12c070ff0 con 0x7fd12c0b1380 2023-10-06T03:30:35.112 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:35.108+0000 7fd1217fa700 1 -- 172.21.15.110:0/3730894314 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7fd12801b3b0 con 0x7fd12c0b1380 2023-10-06T03:30:35.113 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:35.108+0000 7fd1217fa700 1 --2- 172.21.15.110:0/3730894314 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fd110060580 0x7fd110062a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:35.113 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:35.108+0000 7fd1217fa700 1 -- 172.21.15.110:0/3730894314 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6378+0+0 (secure 0 0 0) 0x7fd12807bed0 con 0x7fd12c0b1380 2023-10-06T03:30:35.113 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:35.108+0000 7fd130d28700 1 --2- 172.21.15.110:0/3730894314 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fd110060580 0x7fd110062a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:35.114 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:35.108+0000 7fd130d28700 1 --2- 172.21.15.110:0/3730894314 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fd110060580 0x7fd110062a30 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fd12c15a2d0 tx=0x7fd11c008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:30:35.119 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:35.116+0000 7fd1217fa700 1 -- 172.21.15.110:0/3730894314 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fd12804c220 con 0x7fd12c0b1380 2023-10-06T03:30:35.278 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:35.272+0000 7fd13252b700 1 -- 172.21.15.110:0/3730894314 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd12c0c7bc0 con 0x7fd12c0b1380 2023-10-06T03:30:35.280 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:35.276+0000 7fd1217fa700 1 -- 172.21.15.110:0/3730894314 <== mon.0 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v41) v1 ==== 74+0+17132 (secure 0 0 0) 0x7fd128017020 con 0x7fd12c0b1380 2023-10-06T03:30:35.281 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-10-06T03:30:35.281 INFO:teuthology.orchestra.run.smithi110.stdout:{"epoch":41,"fsid":"93b8f6ac-4c92-4b45-b8e4-b59617024aa2","created":"2023-10-06T03:29:10.519583+0000","modified":"2023-10-06T03:30:34.013838+0000","last_up_change":"2023-10-06T03:29:21.960511+0000","last_in_change":"2023-10-06T03:29:12.560436+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-06T03:29:20.703743+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2023-10-06T03:29:28.307281+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2023-10-06T03:30:31.807897+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"41","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":7,"pool_name":"cephfs_data","create_time":"2023-10-06T03:30:32.991334+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"41","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"5866eb6c-2adc-46e4-be7a-e5bcda0f4065","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6824","nonce":33394},{"type":"v1","addr":"172.21.15.110:6825","nonce":33394}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6826","nonce":33394},{"type":"v1","addr":"172.21.15.110:6827","nonce":33394}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6830","nonce":33394},{"type":"v1","addr":"172.21.15.110:6831","nonce":33394}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6828","nonce":33394},{"type":"v1","addr":"172.21.15.110:6829","nonce":33394}]},"public_addr":"172.21.15.110:6825/33394","cluster_addr":"172.21.15.110:6827/33394","heartbeat_back_addr":"172.21.15.110:6831/33394","heartbeat_front_addr":"172.21.15.110:6829/33394","state":["exists","up"]},{"osd":1,"uuid":"a38de118-062b-4a05-9782-2ceddf6ff4df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6800","nonce":33396},{"type":"v1","addr":"172.21.15.110:6801","nonce":33396}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6802","nonce":33396},{"type":"v1","addr":"172.21.15.110:6803","nonce":33396}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6806","nonce":33396},{"type":"v1","addr":"172.21.15.110:6807","nonce":33396}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6804","nonce":33396},{"type":"v1","addr":"172.21.15.110:6805","nonce":33396}]},"public_addr":"172.21.15.110:6801/33396","cluster_addr":"172.21.15.110:6803/33396","heartbeat_back_addr":"172.21.15.110:6807/33396","heartbeat_front_addr":"172.21.15.110:6805/33396","state":["exists","up"]},{"osd":2,"uuid":"56e39be2-3407-4b69-877e-cf16617b06d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6816","nonce":33397},{"type":"v1","addr":"172.21.15.110:6817","nonce":33397}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6818","nonce":33397},{"type":"v1","addr":"172.21.15.110:6819","nonce":33397}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6822","nonce":33397},{"type":"v1","addr":"172.21.15.110:6823","nonce":33397}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6820","nonce":33397},{"type":"v1","addr":"172.21.15.110:6821","nonce":33397}]},"public_addr":"172.21.15.110:6817/33397","cluster_addr":"172.21.15.110:6819/33397","heartbeat_back_addr":"172.21.15.110:6823/33397","heartbeat_front_addr":"172.21.15.110:6821/33397","state":["exists","up"]},{"osd":3,"uuid":"01c82d13-c1d4-4bde-b55e-3e0089b40115","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6808","nonce":33395},{"type":"v1","addr":"172.21.15.110:6809","nonce":33395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6810","nonce":33395},{"type":"v1","addr":"172.21.15.110:6811","nonce":33395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6814","nonce":33395},{"type":"v1","addr":"172.21.15.110:6815","nonce":33395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6812","nonce":33395},{"type":"v1","addr":"172.21.15.110:6813","nonce":33395}]},"public_addr":"172.21.15.110:6809/33395","cluster_addr":"172.21.15.110:6811/33395","heartbeat_back_addr":"172.21.15.110:6815/33395","heartbeat_front_addr":"172.21.15.110:6813/33395","state":["exists","up"]},{"osd":4,"uuid":"86983430-5fa3-4d17-998f-03e2cd843237","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":33683},{"type":"v1","addr":"172.21.15.192:6817","nonce":33683}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":33683},{"type":"v1","addr":"172.21.15.192:6819","nonce":33683}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":33683},{"type":"v1","addr":"172.21.15.192:6823","nonce":33683}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6820","nonce":33683},{"type":"v1","addr":"172.21.15.192:6821","nonce":33683}]},"public_addr":"172.21.15.192:6817/33683","cluster_addr":"172.21.15.192:6819/33683","heartbeat_back_addr":"172.21.15.192:6823/33683","heartbeat_front_addr":"172.21.15.192:6821/33683","state":["exists","up"]},{"osd":5,"uuid":"6945257f-821d-4619-a63f-ebe857eba6f4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":33684},{"type":"v1","addr":"172.21.15.192:6801","nonce":33684}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":33684},{"type":"v1","addr":"172.21.15.192:6803","nonce":33684}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":33684},{"type":"v1","addr":"172.21.15.192:6807","nonce":33684}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6804","nonce":33684},{"type":"v1","addr":"172.21.15.192:6805","nonce":33684}]},"public_addr":"172.21.15.192:6801/33684","cluster_addr":"172.21.15.192:6803/33684","heartbeat_back_addr":"172.21.15.192:6807/33684","heartbeat_front_addr":"172.21.15.192:6805/33684","state":["exists","up"]},{"osd":6,"uuid":"7854d1ae-7060-4780-8f97-73cea5e7a12f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6824","nonce":33686},{"type":"v1","addr":"172.21.15.192:6825","nonce":33686}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":33686},{"type":"v1","addr":"172.21.15.192:6827","nonce":33686}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":33686},{"type":"v1","addr":"172.21.15.192:6831","nonce":33686}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6828","nonce":33686},{"type":"v1","addr":"172.21.15.192:6829","nonce":33686}]},"public_addr":"172.21.15.192:6825/33686","cluster_addr":"172.21.15.192:6827/33686","heartbeat_back_addr":"172.21.15.192:6831/33686","heartbeat_front_addr":"172.21.15.192:6829/33686","state":["exists","up"]},{"osd":7,"uuid":"f1ebdfd5-b859-482b-9f97-c18c4a26f445","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6808","nonce":33685},{"type":"v1","addr":"172.21.15.192:6809","nonce":33685}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":33685},{"type":"v1","addr":"172.21.15.192:6811","nonce":33685}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":33685},{"type":"v1","addr":"172.21.15.192:6815","nonce":33685}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6812","nonce":33685},{"type":"v1","addr":"172.21.15.192:6813","nonce":33685}]},"public_addr":"172.21.15.192:6809/33685","cluster_addr":"172.21.15.192:6811/33685","heartbeat_back_addr":"172.21.15.192:6815/33685","heartbeat_front_addr":"172.21.15.192:6813/33685","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T03:29:18.002048+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T03:29:17.478576+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T03:29:18.034944+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T03:29:17.628513+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T03:29:18.047338+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T03:29:17.772349+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T03:29:17.883958+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T03:29:18.151838+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-06T03:30:35.283 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:35.276+0000 7fd13252b700 1 -- 172.21.15.110:0/3730894314 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fd110060580 msgr2=0x7fd110062a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:35.283 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:35.276+0000 7fd13252b700 1 --2- 172.21.15.110:0/3730894314 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fd110060580 0x7fd110062a30 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fd12c15a2d0 tx=0x7fd11c008040 comp rx=0 tx=0).stop 2023-10-06T03:30:35.283 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:35.276+0000 7fd13252b700 1 -- 172.21.15.110:0/3730894314 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd12c0b1380 msgr2=0x7fd12c0c3370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:35.283 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:35.276+0000 7fd13252b700 1 --2- 172.21.15.110:0/3730894314 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd12c0b1380 0x7fd12c0c3370 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7fd12800c590 tx=0x7fd12800c8a0 comp rx=0 tx=0).stop 2023-10-06T03:30:35.283 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:35.276+0000 7fd13252b700 1 -- 172.21.15.110:0/3730894314 shutdown_connections 2023-10-06T03:30:35.283 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:35.276+0000 7fd13252b700 1 --2- 172.21.15.110:0/3730894314 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fd110060580 0x7fd110062a30 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:35.283 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:35.276+0000 7fd13252b700 1 --2- 172.21.15.110:0/3730894314 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd12c156f50 0x7fd12c151110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:35.283 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:35.276+0000 7fd13252b700 1 --2- 172.21.15.110:0/3730894314 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd12c156b80 0x7fd12c0c4ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:35.283 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:35.276+0000 7fd13252b700 1 --2- 172.21.15.110:0/3730894314 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd12c0b1380 0x7fd12c0c3370 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:35.284 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:35.276+0000 7fd13252b700 1 -- 172.21.15.110:0/3730894314 >> 172.21.15.110:0/3730894314 conn(0x7fd12c0cc470 msgr2=0x7fd12c06a000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:30:35.284 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:35.276+0000 7fd13252b700 1 -- 172.21.15.110:0/3730894314 shutdown_connections 2023-10-06T03:30:35.284 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:35.276+0000 7fd13252b700 1 -- 172.21.15.110:0/3730894314 wait complete. 2023-10-06T03:30:35.295 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2023-10-06T03:30:35.410 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:35.404+0000 7f5600927700 1 Processor -- start 2023-10-06T03:30:35.410 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:35.404+0000 7f5600927700 1 -- start start 2023-10-06T03:30:35.411 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:35.404+0000 7f5600927700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f55fc155890 0x7f55fc0af3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:35.411 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:35.404+0000 7f5600927700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f55fc0af900 0x7f55fc0b0750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:35.411 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:35.404+0000 7f5600927700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55fc0aef30 con 0x7f55fc0b0c90 2023-10-06T03:30:35.411 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:35.404+0000 7f5600927700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f55fc0b10d0 con 0x7f55fc155890 2023-10-06T03:30:35.411 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:35.404+0000 7f5600927700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f55fc069790 con 0x7f55fc0af900 2023-10-06T03:30:35.411 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:35.404+0000 7f55fb59e700 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f55fc0b0c90 0x7f55fc06b7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:54608/0 (socket says 172.21.15.110:54608) 2023-10-06T03:30:35.411 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:35.404+0000 7f55fad9d700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f55fc155890 0x7f55fc0af3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:35.411 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:35.404+0000 7f55fb59e700 1 -- 172.21.15.110:0/495269743 learned_addr learned my addr 172.21.15.110:0/495269743 (peer_addr_for_me v1:172.21.15.110:0/0) 2023-10-06T03:30:35.412 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:35.404+0000 7f55fa59c700 1 --2- 172.21.15.110:0/495269743 >> v2:172.21.15.192:3300/0 conn(0x7f55fc0af900 0x7f55fc0b0750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:35.413 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:35.408+0000 7f55f9d9b700 1 -- 172.21.15.110:0/495269743 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3151069907 0 0) 0x7f55fc0aef30 con 0x7f55fc0b0c90 2023-10-06T03:30:35.413 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:35.408+0000 7f55f9d9b700 1 -- 172.21.15.110:0/495269743 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f55dc003570 con 0x7f55fc0b0c90 2023-10-06T03:30:35.413 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:35.408+0000 7f55fad9d700 1 -- 172.21.15.110:0/495269743 >> v2:172.21.15.192:3300/0 conn(0x7f55fc0af900 msgr2=0x7f55fc0b0750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:30:35.413 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:35.408+0000 7f55fad9d700 1 --2- 172.21.15.110:0/495269743 >> v2:172.21.15.192:3300/0 conn(0x7f55fc0af900 0x7f55fc0b0750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:35.413 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:35.408+0000 7f55fad9d700 1 -- 172.21.15.110:0/495269743 >> v1:172.21.15.110:6789/0 conn(0x7f55fc0b0c90 legacy=0x7f55fc06b7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:35.413 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:35.408+0000 7f55fad9d700 1 -- 172.21.15.110:0/495269743 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55fc156d70 con 0x7f55fc155890 2023-10-06T03:30:35.414 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:35.408+0000 7f55fad9d700 1 --2- 172.21.15.110:0/495269743 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f55fc155890 0x7f55fc0af3c0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f55fc069ab0 tx=0x7f55ec0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf383565899572d2 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:30:35.414 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:35.408+0000 7f55f9d9b700 1 -- 172.21.15.110:0/495269743 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55ec011630 con 0x7f55fc155890 2023-10-06T03:30:35.414 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:35.408+0000 7f55f9d9b700 1 -- 172.21.15.110:0/495269743 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55ec0117b0 con 0x7f55fc155890 2023-10-06T03:30:35.414 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:35.408+0000 7f55f9d9b700 1 -- 172.21.15.110:0/495269743 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55ec011b70 con 0x7f55fc155890 2023-10-06T03:30:35.414 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:35.408+0000 7f5600927700 1 -- 172.21.15.110:0/495269743 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f55fc155890 msgr2=0x7f55fc0af3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:35.414 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:35.408+0000 7f5600927700 1 --2- 172.21.15.110:0/495269743 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f55fc155890 0x7f55fc0af3c0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f55fc069ab0 tx=0x7f55ec0091a0 comp rx=0 tx=0).stop 2023-10-06T03:30:35.414 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:35.408+0000 7f5600927700 1 -- 172.21.15.110:0/495269743 shutdown_connections 2023-10-06T03:30:35.414 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:35.408+0000 7f5600927700 1 --2- 172.21.15.110:0/495269743 >> v2:172.21.15.192:3300/0 conn(0x7f55fc0af900 0x7f55fc0b0750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:35.414 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:35.408+0000 7f5600927700 1 --2- 172.21.15.110:0/495269743 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f55fc155890 0x7f55fc0af3c0 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:35.414 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:35.408+0000 7f5600927700 1 -- 172.21.15.110:0/495269743 >> 172.21.15.110:0/495269743 conn(0x7f55fc0cbac0 msgr2=0x7f55fc0cbec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:30:35.415 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:35.408+0000 7f5600927700 1 -- 172.21.15.110:0/495269743 shutdown_connections 2023-10-06T03:30:35.415 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:35.408+0000 7f5600927700 1 -- 172.21.15.110:0/495269743 wait complete. 2023-10-06T03:30:35.416 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:35.408+0000 7f5600927700 1 Processor -- start 2023-10-06T03:30:35.417 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:35.408+0000 7f5600927700 1 -- start start 2023-10-06T03:30:35.417 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:35.408+0000 7f5600927700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f55fc0af900 0x7f55fc0c2300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:35.417 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:35.408+0000 7f5600927700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f55fc0b0c90 0x7f55fc151200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:35.417 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:35.408+0000 7f5600927700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f55fc155890 0x7f55fc169cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:35.417 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:35.408+0000 7f5600927700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55fc16a200 con 0x7f55fc0b0c90 2023-10-06T03:30:35.417 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:35.408+0000 7f5600927700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55fc1524f0 con 0x7f55fc0af900 2023-10-06T03:30:35.417 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:35.408+0000 7f5600927700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55fc1518e0 con 0x7f55fc155890 2023-10-06T03:30:35.417 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:35.412+0000 7f55fa59c700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f55fc0b0c90 0x7f55fc151200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:35.417 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:35.412+0000 7f55fa59c700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f55fc0b0c90 0x7f55fc151200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:48116/0 (socket says 172.21.15.110:48116) 2023-10-06T03:30:35.417 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:35.412+0000 7f55fa59c700 1 -- 172.21.15.110:0/2623700303 learned_addr learned my addr 172.21.15.110:0/2623700303 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:30:35.417 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:35.412+0000 7f55fad9d700 1 --2- 172.21.15.110:0/2623700303 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f55fc0af900 0x7f55fc0c2300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:35.418 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:35.412+0000 7f55fb59e700 1 --2- 172.21.15.110:0/2623700303 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f55fc155890 0x7f55fc169cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:35.418 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:35.412+0000 7f55fa59c700 1 -- 172.21.15.110:0/2623700303 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f55fc155890 msgr2=0x7f55fc169cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:35.418 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:35.412+0000 7f55fa59c700 1 --2- 172.21.15.110:0/2623700303 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f55fc155890 0x7f55fc169cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:35.418 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:35.412+0000 7f55fa59c700 1 -- 172.21.15.110:0/2623700303 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f55fc0af900 msgr2=0x7f55fc0c2300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:35.418 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:35.412+0000 7f55fa59c700 1 --2- 172.21.15.110:0/2623700303 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f55fc0af900 0x7f55fc0c2300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:35.418 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:35.412+0000 7f55fa59c700 1 -- 172.21.15.110:0/2623700303 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55ec009420 con 0x7f55fc0b0c90 2023-10-06T03:30:35.418 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:35.412+0000 7f55fa59c700 1 --2- 172.21.15.110:0/2623700303 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f55fc0b0c90 0x7f55fc151200 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f55e400ec50 tx=0x7f55e40091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:30:35.418 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:35.412+0000 7f55f2ffd700 1 -- 172.21.15.110:0/2623700303 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55e400c850 con 0x7f55fc0b0c90 2023-10-06T03:30:35.418 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:35.412+0000 7f55f2ffd700 1 -- 172.21.15.110:0/2623700303 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55e400c9b0 con 0x7f55fc0b0c90 2023-10-06T03:30:35.418 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:35.412+0000 7f5600927700 1 -- 172.21.15.110:0/2623700303 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f55fc151bc0 con 0x7f55fc0b0c90 2023-10-06T03:30:35.418 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:35.412+0000 7f55f2ffd700 1 -- 172.21.15.110:0/2623700303 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55e400cd70 con 0x7f55fc0b0c90 2023-10-06T03:30:35.419 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:35.412+0000 7f5600927700 1 -- 172.21.15.110:0/2623700303 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f55fc152180 con 0x7f55fc0b0c90 2023-10-06T03:30:35.419 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:35.412+0000 7f5600927700 1 -- 172.21.15.110:0/2623700303 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f55fc0aec30 con 0x7f55fc0b0c90 2023-10-06T03:30:35.420 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:35.412+0000 7f55f2ffd700 1 -- 172.21.15.110:0/2623700303 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7f55e4013680 con 0x7f55fc0b0c90 2023-10-06T03:30:35.420 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:35.412+0000 7f55f2ffd700 1 --2- 172.21.15.110:0/2623700303 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f55e0060550 0x7f55e0062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:35.420 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:35.412+0000 7f55f2ffd700 1 -- 172.21.15.110:0/2623700303 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6378+0+0 (secure 0 0 0) 0x7f55e407d040 con 0x7f55fc0b0c90 2023-10-06T03:30:35.420 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:35.412+0000 7f55fad9d700 1 --2- 172.21.15.110:0/2623700303 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f55e0060550 0x7f55e0062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:35.420 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:35.416+0000 7f55fad9d700 1 --2- 172.21.15.110:0/2623700303 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f55e0060550 0x7f55e0062a00 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f55fc069ab0 tx=0x7f55ec017000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:30:35.427 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:35.420+0000 7f55f2ffd700 1 -- 172.21.15.110:0/2623700303 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f55fc0aec30 con 0x7f55fc0b0c90 2023-10-06T03:30:35.625 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:35.620+0000 7f5600927700 1 -- 172.21.15.110:0/2623700303 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f55fc0aec30 con 0x7f55fc0b0c90 2023-10-06T03:30:36.335 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:36.332+0000 7f55f2ffd700 1 -- 172.21.15.110:0/2623700303 <== mon.0 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v15) v1 ==== 118+0+0 (secure 0 0 0) 0x7f55fc0aec30 con 0x7f55fc0b0c90 2023-10-06T03:30:36.340 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:36.336+0000 7f5600927700 1 -- 172.21.15.110:0/2623700303 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f55e0060550 msgr2=0x7f55e0062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:36.340 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:36.336+0000 7f5600927700 1 --2- 172.21.15.110:0/2623700303 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f55e0060550 0x7f55e0062a00 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f55fc069ab0 tx=0x7f55ec017000 comp rx=0 tx=0).stop 2023-10-06T03:30:36.341 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:36.336+0000 7f5600927700 1 -- 172.21.15.110:0/2623700303 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f55fc0b0c90 msgr2=0x7f55fc151200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:36.341 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:36.336+0000 7f5600927700 1 --2- 172.21.15.110:0/2623700303 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f55fc0b0c90 0x7f55fc151200 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f55e400ec50 tx=0x7f55e40091a0 comp rx=0 tx=0).stop 2023-10-06T03:30:36.341 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:36.336+0000 7f5600927700 1 -- 172.21.15.110:0/2623700303 shutdown_connections 2023-10-06T03:30:36.341 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:36.336+0000 7f5600927700 1 --2- 172.21.15.110:0/2623700303 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f55e0060550 0x7f55e0062a00 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:36.341 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:36.336+0000 7f5600927700 1 --2- 172.21.15.110:0/2623700303 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f55fc155890 0x7f55fc169cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:36.341 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:36.336+0000 7f5600927700 1 --2- 172.21.15.110:0/2623700303 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f55fc0b0c90 0x7f55fc151200 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:36.341 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:36.336+0000 7f5600927700 1 --2- 172.21.15.110:0/2623700303 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f55fc0af900 0x7f55fc0c2300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:36.341 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:36.336+0000 7f5600927700 1 -- 172.21.15.110:0/2623700303 >> 172.21.15.110:0/2623700303 conn(0x7f55fc0cbac0 msgr2=0x7f55fc154620 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:30:36.341 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:36.336+0000 7f5600927700 1 -- 172.21.15.110:0/2623700303 shutdown_connections 2023-10-06T03:30:36.341 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:36.336+0000 7f5600927700 1 -- 172.21.15.110:0/2623700303 wait complete. 2023-10-06T03:30:36.356 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2023-10-06T03:30:36.474 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:36.468+0000 7fa96a0a6700 1 Processor -- start 2023-10-06T03:30:36.475 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:36.468+0000 7fa96a0a6700 1 -- start start 2023-10-06T03:30:36.477 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:36.472+0000 7fa96a0a6700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa96415d360 0x7fa9640af440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:36.477 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:36.472+0000 7fa96a0a6700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fa9640cfa40 0x7fa9640cb220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:36.477 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:36.472+0000 7fa96a0a6700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9640cf5b0 con 0x7fa9640af070 2023-10-06T03:30:36.478 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:36.472+0000 7fa96a0a6700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa9640ce9a0 con 0x7fa9640cfa40 2023-10-06T03:30:36.478 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:36.472+0000 7fa96a0a6700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9640cec80 con 0x7fa96415d360 2023-10-06T03:30:36.478 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:36.472+0000 7fa9637fe700 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7fa9640af070 0x7fa96415cc50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:54616/0 (socket says 172.21.15.110:54616) 2023-10-06T03:30:36.478 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:36.472+0000 7fa9637fe700 1 -- 172.21.15.110:0/994091526 learned_addr learned my addr 172.21.15.110:0/994091526 (peer_addr_for_me v1:172.21.15.110:0/0) 2023-10-06T03:30:36.478 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:36.472+0000 7fa962ffd700 1 --2- 172.21.15.110:0/994091526 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa96415d360 0x7fa9640af440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:36.478 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:36.472+0000 7fa963fff700 1 --2- 172.21.15.110:0/994091526 >> v2:172.21.15.192:3300/0 conn(0x7fa9640cfa40 0x7fa9640cb220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:36.478 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:36.472+0000 7fa9627fc700 1 -- 172.21.15.110:0/994091526 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 698564532 0 0) 0x7fa9640cf5b0 con 0x7fa9640af070 2023-10-06T03:30:36.478 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:36.472+0000 7fa9627fc700 1 -- 172.21.15.110:0/994091526 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa94c003570 con 0x7fa9640af070 2023-10-06T03:30:36.478 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:36.472+0000 7fa963fff700 1 -- 172.21.15.110:0/994091526 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa96415d360 msgr2=0x7fa9640af440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:30:36.479 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:36.472+0000 7fa963fff700 1 --2- 172.21.15.110:0/994091526 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa96415d360 0x7fa9640af440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:36.479 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:36.472+0000 7fa963fff700 1 -- 172.21.15.110:0/994091526 >> v1:172.21.15.110:6789/0 conn(0x7fa9640af070 legacy=0x7fa96415cc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:36.479 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:36.472+0000 7fa963fff700 1 -- 172.21.15.110:0/994091526 --> v2:172.21.15.192:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9640cef60 con 0x7fa9640cfa40 2023-10-06T03:30:36.479 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:36.472+0000 7fa962ffd700 1 --2- 172.21.15.110:0/994091526 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa96415d360 0x7fa9640af440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-06T03:30:36.479 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:36.472+0000 7fa963fff700 1 --2- 172.21.15.110:0/994091526 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa9640cfa40 0x7fa9640cb220 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fa95c009e60 tx=0x7fa95c009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f63c7aba204c767a server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:30:36.480 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:36.472+0000 7fa9627fc700 1 -- 172.21.15.110:0/994091526 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa95c00e070 con 0x7fa9640cfa40 2023-10-06T03:30:36.480 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:36.472+0000 7fa9627fc700 1 -- 172.21.15.110:0/994091526 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa95c00bab0 con 0x7fa9640cfa40 2023-10-06T03:30:36.481 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:36.472+0000 7fa9627fc700 1 -- 172.21.15.110:0/994091526 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa95c00be30 con 0x7fa9640cfa40 2023-10-06T03:30:36.481 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:36.472+0000 7fa96a0a6700 1 -- 172.21.15.110:0/994091526 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa9640cfa40 msgr2=0x7fa9640cb220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:36.481 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:36.472+0000 7fa96a0a6700 1 --2- 172.21.15.110:0/994091526 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa9640cfa40 0x7fa9640cb220 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fa95c009e60 tx=0x7fa95c009780 comp rx=0 tx=0).stop 2023-10-06T03:30:36.481 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:36.472+0000 7fa96a0a6700 1 -- 172.21.15.110:0/994091526 shutdown_connections 2023-10-06T03:30:36.481 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:36.472+0000 7fa96a0a6700 1 --2- 172.21.15.110:0/994091526 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa9640cfa40 0x7fa9640cb220 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:36.481 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:36.472+0000 7fa96a0a6700 1 --2- 172.21.15.110:0/994091526 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa96415d360 0x7fa9640af440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:36.481 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:36.472+0000 7fa96a0a6700 1 -- 172.21.15.110:0/994091526 >> 172.21.15.110:0/994091526 conn(0x7fa96406e840 msgr2=0x7fa9640d0010 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:30:36.481 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:36.472+0000 7fa96a0a6700 1 -- 172.21.15.110:0/994091526 shutdown_connections 2023-10-06T03:30:36.481 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:36.472+0000 7fa96a0a6700 1 -- 172.21.15.110:0/994091526 wait complete. 2023-10-06T03:30:36.483 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:36.476+0000 7fa96a0a6700 1 Processor -- start 2023-10-06T03:30:36.483 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:36.476+0000 7fa96a0a6700 1 -- start start 2023-10-06T03:30:36.483 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:36.476+0000 7fa96a0a6700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa9640af070 0x7fa964154c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:36.483 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:36.476+0000 7fa96a0a6700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa9640cfa40 0x7fa964159490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:36.483 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:36.476+0000 7fa96a0a6700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa96415d360 0x7fa9640c96b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:36.483 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:36.476+0000 7fa96a0a6700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9640b5a70 con 0x7fa9640af070 2023-10-06T03:30:36.483 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:36.476+0000 7fa96a0a6700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9640c83c0 con 0x7fa9640cfa40 2023-10-06T03:30:36.484 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:36.476+0000 7fa96a0a6700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9640c9220 con 0x7fa96415d360 2023-10-06T03:30:36.484 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:36.476+0000 7fa9637fe700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa9640af070 0x7fa964154c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:36.484 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:36.476+0000 7fa9637fe700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa9640af070 0x7fa964154c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:48130/0 (socket says 172.21.15.110:48130) 2023-10-06T03:30:36.484 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:36.476+0000 7fa9637fe700 1 -- 172.21.15.110:0/2638513575 learned_addr learned my addr 172.21.15.110:0/2638513575 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:30:36.484 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:36.476+0000 7fa962ffd700 1 --2- 172.21.15.110:0/2638513575 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa9640cfa40 0x7fa964159490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:36.484 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:36.476+0000 7fa963fff700 1 --2- 172.21.15.110:0/2638513575 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa96415d360 0x7fa9640c96b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:36.485 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:36.480+0000 7fa9637fe700 1 -- 172.21.15.110:0/2638513575 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa96415d360 msgr2=0x7fa9640c96b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:36.486 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:36.480+0000 7fa9637fe700 1 --2- 172.21.15.110:0/2638513575 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa96415d360 0x7fa9640c96b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:36.486 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:36.480+0000 7fa9637fe700 1 -- 172.21.15.110:0/2638513575 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa9640cfa40 msgr2=0x7fa964159490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:36.486 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:36.480+0000 7fa9637fe700 1 --2- 172.21.15.110:0/2638513575 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa9640cfa40 0x7fa964159490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:36.486 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:36.480+0000 7fa9637fe700 1 -- 172.21.15.110:0/2638513575 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa954009420 con 0x7fa9640af070 2023-10-06T03:30:36.486 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:36.480+0000 7fa9637fe700 1 --2- 172.21.15.110:0/2638513575 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa9640af070 0x7fa964154c60 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7fa95800c790 tx=0x7fa958007df0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:30:36.486 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:36.480+0000 7fa94b7fe700 1 -- 172.21.15.110:0/2638513575 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa958005b30 con 0x7fa9640af070 2023-10-06T03:30:36.486 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:36.480+0000 7fa94b7fe700 1 -- 172.21.15.110:0/2638513575 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa958005c90 con 0x7fa9640af070 2023-10-06T03:30:36.486 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:36.480+0000 7fa96a0a6700 1 -- 172.21.15.110:0/2638513575 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa95c009a20 con 0x7fa9640af070 2023-10-06T03:30:36.486 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:36.480+0000 7fa94b7fe700 1 -- 172.21.15.110:0/2638513575 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa958007320 con 0x7fa9640af070 2023-10-06T03:30:36.487 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:36.480+0000 7fa96a0a6700 1 -- 172.21.15.110:0/2638513575 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9640c8610 con 0x7fa9640af070 2023-10-06T03:30:36.487 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:36.480+0000 7fa96a0a6700 1 -- 172.21.15.110:0/2638513575 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa96406ff60 con 0x7fa9640af070 2023-10-06T03:30:36.488 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:36.480+0000 7fa94b7fe700 1 -- 172.21.15.110:0/2638513575 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7fa958007690 con 0x7fa9640af070 2023-10-06T03:30:36.488 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:36.480+0000 7fa94b7fe700 1 --2- 172.21.15.110:0/2638513575 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fa950060550 0x7fa950062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:36.488 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:36.480+0000 7fa94b7fe700 1 -- 172.21.15.110:0/2638513575 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6378+0+0 (secure 0 0 0) 0x7fa95807be40 con 0x7fa9640af070 2023-10-06T03:30:36.488 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:36.480+0000 7fa962ffd700 1 --2- 172.21.15.110:0/2638513575 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fa950060550 0x7fa950062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:36.488 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:36.484+0000 7fa962ffd700 1 --2- 172.21.15.110:0/2638513575 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fa950060550 0x7fa950062a00 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fa96415c490 tx=0x7fa9540091a0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:30:36.496 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:36.492+0000 7fa94b7fe700 1 -- 172.21.15.110:0/2638513575 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fa95804c190 con 0x7fa9640af070 2023-10-06T03:30:36.694 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:36.688+0000 7fa96a0a6700 1 -- 172.21.15.110:0/2638513575 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7fa9640cf240 con 0x7fa9640af070 2023-10-06T03:30:37.597 INFO:tasks.ceph.mds.a.smithi110.stderr:2023-10-06T03:30:37.592+0000 7fab283cf700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-06T03:30:37.681 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:37.676+0000 7fa94b7fe700 1 -- 172.21.15.110:0/2638513575 <== mon.0 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v16) v1 ==== 122+0+0 (secure 0 0 0) 0x7fa95804f7b0 con 0x7fa9640af070 2023-10-06T03:30:37.685 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:37.680+0000 7fa96a0a6700 1 -- 172.21.15.110:0/2638513575 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fa950060550 msgr2=0x7fa950062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:37.686 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:37.680+0000 7fa96a0a6700 1 --2- 172.21.15.110:0/2638513575 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fa950060550 0x7fa950062a00 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fa96415c490 tx=0x7fa9540091a0 comp rx=0 tx=0).stop 2023-10-06T03:30:37.686 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:37.680+0000 7fa96a0a6700 1 -- 172.21.15.110:0/2638513575 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa9640af070 msgr2=0x7fa964154c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:37.686 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:37.680+0000 7fa96a0a6700 1 --2- 172.21.15.110:0/2638513575 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa9640af070 0x7fa964154c60 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7fa95800c790 tx=0x7fa958007df0 comp rx=0 tx=0).stop 2023-10-06T03:30:37.686 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:37.680+0000 7fa96a0a6700 1 -- 172.21.15.110:0/2638513575 shutdown_connections 2023-10-06T03:30:37.686 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:37.680+0000 7fa96a0a6700 1 --2- 172.21.15.110:0/2638513575 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fa950060550 0x7fa950062a00 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:37.686 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:37.680+0000 7fa96a0a6700 1 --2- 172.21.15.110:0/2638513575 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fa96415d360 0x7fa9640c96b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:37.686 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:37.680+0000 7fa96a0a6700 1 --2- 172.21.15.110:0/2638513575 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa9640cfa40 0x7fa964159490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:37.686 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:37.680+0000 7fa96a0a6700 1 --2- 172.21.15.110:0/2638513575 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa9640af070 0x7fa964154c60 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:37.686 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:37.680+0000 7fa96a0a6700 1 -- 172.21.15.110:0/2638513575 >> 172.21.15.110:0/2638513575 conn(0x7fa96406e840 msgr2=0x7fa9640c6850 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:30:37.687 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:37.680+0000 7fa96a0a6700 1 -- 172.21.15.110:0/2638513575 shutdown_connections 2023-10-06T03:30:37.687 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:37.680+0000 7fa96a0a6700 1 -- 172.21.15.110:0/2638513575 wait complete. 2023-10-06T03:30:37.703 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-06T03:30:37.817 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:37.812+0000 7f767e072700 1 Processor -- start 2023-10-06T03:30:37.817 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:37.812+0000 7f767e072700 1 -- start start 2023-10-06T03:30:37.818 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:37.812+0000 7f767e072700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f7678158be0 0x7f7678158fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:37.818 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:37.812+0000 7f767e072700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f76780ca7f0 0x7f76780cac70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:37.818 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:37.812+0000 7f767e072700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76780cc1a0 con 0x7f76780b1ba0 2023-10-06T03:30:37.818 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:37.812+0000 7f767e072700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f76780cc630 con 0x7f7678158be0 2023-10-06T03:30:37.818 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:37.812+0000 7f767e072700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f76780cb590 con 0x7f76780ca7f0 2023-10-06T03:30:37.819 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:37.812+0000 7f767d070700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f7678158be0 0x7f7678158fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:37.820 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:37.812+0000 7f767c86f700 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f76780b1ba0 0x7f76780b1ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:54630/0 (socket says 172.21.15.110:54630) 2023-10-06T03:30:37.820 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:37.812+0000 7f767c86f700 1 -- 172.21.15.110:0/1040961119 learned_addr learned my addr 172.21.15.110:0/1040961119 (peer_addr_for_me v1:172.21.15.110:0/0) 2023-10-06T03:30:37.820 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:37.812+0000 7f767d871700 1 --2- 172.21.15.110:0/1040961119 >> v2:172.21.15.192:3300/0 conn(0x7f76780ca7f0 0x7f76780cac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:37.820 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:37.812+0000 7f766ffff700 1 -- 172.21.15.110:0/1040961119 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2987413401 0 0) 0x7f76780cc1a0 con 0x7f76780b1ba0 2023-10-06T03:30:37.820 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:37.812+0000 7f766ffff700 1 -- 172.21.15.110:0/1040961119 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7664003570 con 0x7f76780b1ba0 2023-10-06T03:30:37.820 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:37.812+0000 7f767d070700 1 -- 172.21.15.110:0/1040961119 >> v2:172.21.15.192:3300/0 conn(0x7f76780ca7f0 msgr2=0x7f76780cac70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:30:37.820 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:37.812+0000 7f767d070700 1 --2- 172.21.15.110:0/1040961119 >> v2:172.21.15.192:3300/0 conn(0x7f76780ca7f0 0x7f76780cac70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:37.820 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:37.812+0000 7f767d070700 1 -- 172.21.15.110:0/1040961119 >> v1:172.21.15.110:6789/0 conn(0x7f76780b1ba0 legacy=0x7f76780b1ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:37.820 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:37.812+0000 7f767d070700 1 -- 172.21.15.110:0/1040961119 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76780cb870 con 0x7f7678158be0 2023-10-06T03:30:37.820 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:37.816+0000 7f767d070700 1 --2- 172.21.15.110:0/1040961119 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7678158be0 0x7f7678158fb0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f76780cc110 tx=0x7f76680091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ec98dbf29cd7018 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:30:37.820 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:37.816+0000 7f766ffff700 1 -- 172.21.15.110:0/1040961119 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7668011630 con 0x7f7678158be0 2023-10-06T03:30:37.821 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:37.816+0000 7f766ffff700 1 -- 172.21.15.110:0/1040961119 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f76680117b0 con 0x7f7678158be0 2023-10-06T03:30:37.821 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:37.816+0000 7f766ffff700 1 -- 172.21.15.110:0/1040961119 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7668011b70 con 0x7f7678158be0 2023-10-06T03:30:37.822 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:37.816+0000 7f767e072700 1 -- 172.21.15.110:0/1040961119 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7678158be0 msgr2=0x7f7678158fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:37.822 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:37.816+0000 7f767e072700 1 --2- 172.21.15.110:0/1040961119 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7678158be0 0x7f7678158fb0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f76780cc110 tx=0x7f76680091a0 comp rx=0 tx=0).stop 2023-10-06T03:30:37.822 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:37.816+0000 7f767e072700 1 -- 172.21.15.110:0/1040961119 shutdown_connections 2023-10-06T03:30:37.822 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:37.816+0000 7f767e072700 1 --2- 172.21.15.110:0/1040961119 >> v2:172.21.15.192:3300/0 conn(0x7f76780ca7f0 0x7f76780cac70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:37.822 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:37.816+0000 7f767e072700 1 --2- 172.21.15.110:0/1040961119 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7678158be0 0x7f7678158fb0 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:37.822 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:37.816+0000 7f767e072700 1 -- 172.21.15.110:0/1040961119 >> 172.21.15.110:0/1040961119 conn(0x7f76780d08c0 msgr2=0x7f76780cc860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:30:37.822 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:37.816+0000 7f767e072700 1 -- 172.21.15.110:0/1040961119 shutdown_connections 2023-10-06T03:30:37.822 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:37.816+0000 7f767e072700 1 -- 172.21.15.110:0/1040961119 wait complete. 2023-10-06T03:30:37.822 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:37.816+0000 7f767e072700 1 Processor -- start 2023-10-06T03:30:37.823 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:37.816+0000 7f767e072700 1 -- start start 2023-10-06T03:30:37.825 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:37.820+0000 7f767e072700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f76780b1ba0 0x7f7678154650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:37.825 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:37.820+0000 7f767e072700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f76780ca7f0 0x7f76780c9460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:37.825 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:37.820+0000 7f767e072700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7678158be0 0x7f7678155ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:37.825 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:37.820+0000 7f767e072700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f767815a180 con 0x7f7678158be0 2023-10-06T03:30:37.825 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:37.820+0000 7f767e072700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76780c7d10 con 0x7f76780b1ba0 2023-10-06T03:30:37.825 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:37.820+0000 7f767e072700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76780c7ff0 con 0x7f76780ca7f0 2023-10-06T03:30:37.825 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:37.820+0000 7f767d871700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7678158be0 0x7f7678155ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:37.825 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:37.820+0000 7f767d871700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7678158be0 0x7f7678155ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:48144/0 (socket says 172.21.15.110:48144) 2023-10-06T03:30:37.825 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:37.820+0000 7f767d871700 1 -- 172.21.15.110:0/340256120 learned_addr learned my addr 172.21.15.110:0/340256120 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:30:37.825 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:37.820+0000 7f767d070700 1 --2- 172.21.15.110:0/340256120 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f76780b1ba0 0x7f7678154650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:37.826 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:37.820+0000 7f767c86f700 1 --2- 172.21.15.110:0/340256120 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f76780ca7f0 0x7f76780c9460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:37.826 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:37.820+0000 7f767d871700 1 -- 172.21.15.110:0/340256120 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f76780ca7f0 msgr2=0x7f76780c9460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:37.826 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:37.820+0000 7f767d871700 1 --2- 172.21.15.110:0/340256120 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f76780ca7f0 0x7f76780c9460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:37.826 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:37.820+0000 7f767d871700 1 -- 172.21.15.110:0/340256120 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f76780b1ba0 msgr2=0x7f7678154650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:37.826 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:37.820+0000 7f767d871700 1 --2- 172.21.15.110:0/340256120 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f76780b1ba0 0x7f7678154650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:37.826 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:37.820+0000 7f767d871700 1 -- 172.21.15.110:0/340256120 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7668009420 con 0x7f7678158be0 2023-10-06T03:30:37.826 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:37.820+0000 7f767d871700 1 --2- 172.21.15.110:0/340256120 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7678158be0 0x7f7678155ef0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f7670009e90 tx=0x7f767000b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:30:37.826 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:37.820+0000 7f766d7fa700 1 -- 172.21.15.110:0/340256120 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f767000cc20 con 0x7f7678158be0 2023-10-06T03:30:37.826 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:37.820+0000 7f766d7fa700 1 -- 172.21.15.110:0/340256120 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f767000f040 con 0x7f7678158be0 2023-10-06T03:30:37.826 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:37.820+0000 7f767e072700 1 -- 172.21.15.110:0/340256120 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76780c82d0 con 0x7f7678158be0 2023-10-06T03:30:37.826 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:37.820+0000 7f766d7fa700 1 -- 172.21.15.110:0/340256120 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7670014480 con 0x7f7678158be0 2023-10-06T03:30:37.826 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:37.820+0000 7f767e072700 1 -- 172.21.15.110:0/340256120 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7678154b90 con 0x7f7678158be0 2023-10-06T03:30:37.827 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:37.820+0000 7f767e072700 1 -- 172.21.15.110:0/340256120 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7678159f50 con 0x7f7678158be0 2023-10-06T03:30:37.827 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:37.824+0000 7f766d7fa700 1 -- 172.21.15.110:0/340256120 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7f7670014830 con 0x7f7678158be0 2023-10-06T03:30:37.828 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:37.824+0000 7f766d7fa700 1 --2- 172.21.15.110:0/340256120 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f765c060550 0x7f765c062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:37.828 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:37.824+0000 7f766d7fa700 1 -- 172.21.15.110:0/340256120 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6378+0+0 (secure 0 0 0) 0x7f767007d4b0 con 0x7f7678158be0 2023-10-06T03:30:37.828 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:37.824+0000 7f767d070700 1 --2- 172.21.15.110:0/340256120 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f765c060550 0x7f765c062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:37.829 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:37.824+0000 7f767d070700 1 --2- 172.21.15.110:0/340256120 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f765c060550 0x7f765c062a00 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f76780cc110 tx=0x7f7668017000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:30:37.833 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:37.828+0000 7f766d7fa700 1 -- 172.21.15.110:0/340256120 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f7678159f50 con 0x7f7678158be0 2023-10-06T03:30:37.883 DEBUG:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-06T03:30:37.885 DEBUG:teuthology.orchestra.run.smithi192:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-06T03:30:38.029 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:38.024+0000 7f767e072700 1 -- 172.21.15.110:0/340256120 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7678159f50 con 0x7f7678158be0 2023-10-06T03:30:38.030 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:38.024+0000 7f766d7fa700 1 -- 172.21.15.110:0/340256120 <== mon.0 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) v1 ==== 94+0+4603 (secure 0 0 0) 0x7f7678159f50 con 0x7f7678158be0 2023-10-06T03:30:38.030 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-10-06T03:30:38.031 INFO:teuthology.orchestra.run.smithi110.stdout:{"epoch":16,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5107,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6834/884349347","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6833","nonce":884349347},{"type":"v1","addr":"172.21.15.192:6834","nonce":884349347}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":5110,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6835/4077204152","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6832","nonce":4077204152},{"type":"v1","addr":"172.21.15.192:6835","nonce":4077204152}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":5160,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/504583843","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6834","nonce":504583843},{"type":"v1","addr":"172.21.15.110:6835","nonce":504583843}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-06T03:30:34.014644+0000","modified":"2023-10-06T03:30:37.662700+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5163},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5163":{"gid":5163,"name":"a","rank":0,"incarnation":13,"state":"up:active","state_seq":4,"addr":"172.21.15.110:6837/1379510962","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6836","nonce":1379510962},{"type":"v1","addr":"172.21.15.110:6837","nonce":1379510962}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":2}]} 2023-10-06T03:30:38.032 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:38.028+0000 7f767e072700 1 -- 172.21.15.110:0/340256120 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f765c060550 msgr2=0x7f765c062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:38.033 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:38.028+0000 7f767e072700 1 --2- 172.21.15.110:0/340256120 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f765c060550 0x7f765c062a00 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f76780cc110 tx=0x7f7668017000 comp rx=0 tx=0).stop 2023-10-06T03:30:38.033 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:38.028+0000 7f767e072700 1 -- 172.21.15.110:0/340256120 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7678158be0 msgr2=0x7f7678155ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:38.033 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:38.028+0000 7f767e072700 1 --2- 172.21.15.110:0/340256120 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7678158be0 0x7f7678155ef0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f7670009e90 tx=0x7f767000b040 comp rx=0 tx=0).stop 2023-10-06T03:30:38.033 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:38.028+0000 7f767e072700 1 -- 172.21.15.110:0/340256120 shutdown_connections 2023-10-06T03:30:38.033 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:38.028+0000 7f767e072700 1 --2- 172.21.15.110:0/340256120 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f765c060550 0x7f765c062a00 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:38.033 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:38.028+0000 7f767e072700 1 --2- 172.21.15.110:0/340256120 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7678158be0 0x7f7678155ef0 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:38.033 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:38.028+0000 7f767e072700 1 --2- 172.21.15.110:0/340256120 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f76780ca7f0 0x7f76780c9460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:38.033 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:38.028+0000 7f767e072700 1 --2- 172.21.15.110:0/340256120 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f76780b1ba0 0x7f7678154650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:38.033 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:38.028+0000 7f767e072700 1 -- 172.21.15.110:0/340256120 >> 172.21.15.110:0/340256120 conn(0x7f76780d08c0 msgr2=0x7f76780b40a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:30:38.033 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:38.028+0000 7f767e072700 1 -- 172.21.15.110:0/340256120 shutdown_connections 2023-10-06T03:30:38.034 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:38.028+0000 7f767e072700 1 -- 172.21.15.110:0/340256120 wait complete. 2023-10-06T03:30:38.034 INFO:teuthology.orchestra.run.smithi110.stderr:dumped fsmap epoch 16 2023-10-06T03:30:38.044 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-06T03:30:38.153 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:38.148+0000 7f9682a05700 1 Processor -- start 2023-10-06T03:30:38.154 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:38.148+0000 7f9682a05700 1 -- start start 2023-10-06T03:30:38.155 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:38.148+0000 7f9682a05700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f967c155da0 0x7f967c06cc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:38.155 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:38.148+0000 7f9682a05700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f967c159e80 0x7f967c0c5f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:38.155 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:38.148+0000 7f9682a05700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f967c0caad0 con 0x7f967c1559d0 2023-10-06T03:30:38.156 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:38.148+0000 7f9682a05700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f967c0ae3f0 con 0x7f967c159e80 2023-10-06T03:30:38.156 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:38.148+0000 7f9682a05700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f967c0b1060 con 0x7f967c155da0 2023-10-06T03:30:38.156 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:38.148+0000 7f9681202700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f967c155da0 0x7f967c06cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:38.156 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:38.148+0000 7f9682204700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f967c159e80 0x7f967c0c5f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:38.156 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:38.148+0000 7f9682204700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f967c159e80 0x7f967c0c5f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.110:48640/0 (socket says 172.21.15.110:48640) 2023-10-06T03:30:38.156 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:38.148+0000 7f9681202700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f967c155da0 0x7f967c06cc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3301/0 says I am v2:172.21.15.110:43236/0 (socket says 172.21.15.110:43236) 2023-10-06T03:30:38.156 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:38.148+0000 7f9682204700 1 -- 172.21.15.110:0/3565990837 learned_addr learned my addr 172.21.15.110:0/3565990837 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:30:38.156 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:38.148+0000 7f9681a03700 1 --1- >> v1:172.21.15.192:6789/0 conn(0x7f967c1559d0 0x7f967c159770 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.110:37158/0 (socket says 172.21.15.110:37158) 2023-10-06T03:30:38.156 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:38.152+0000 7f9682204700 1 -- 172.21.15.110:0/3565990837 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f967c155da0 msgr2=0x7f967c06cc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:30:38.157 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:38.152+0000 7f9682204700 1 --2- 172.21.15.110:0/3565990837 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f967c155da0 0x7f967c06cc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:38.157 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:38.152+0000 7f9682204700 1 -- 172.21.15.110:0/3565990837 >> v1:172.21.15.192:6789/0 conn(0x7f967c1559d0 legacy=0x7f967c159770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:38.157 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:38.152+0000 7f9680a01700 1 -- 172.21.15.110:0/3565990837 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 599603928 0 0) 0x7f967c0caad0 con 0x7f967c1559d0 2023-10-06T03:30:38.157 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:38.152+0000 7f9682204700 1 -- 172.21.15.110:0/3565990837 --> v2:172.21.15.192:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f967c0ca1a0 con 0x7f967c159e80 2023-10-06T03:30:38.158 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:38.152+0000 7f9682204700 1 --2- 172.21.15.110:0/3565990837 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f967c159e80 0x7f967c0c5f40 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f9674009f20 tx=0x7f9674009840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=21a4ffe128a5d3b server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:30:38.158 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:38.152+0000 7f9680a01700 1 -- 172.21.15.110:0/3565990837 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f967400e070 con 0x7f967c159e80 2023-10-06T03:30:38.158 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:38.152+0000 7f9680a01700 1 -- 172.21.15.110:0/3565990837 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f967400b9d0 con 0x7f967c159e80 2023-10-06T03:30:38.158 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:38.152+0000 7f9680a01700 1 -- 172.21.15.110:0/3565990837 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96740092e0 con 0x7f967c159e80 2023-10-06T03:30:38.159 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:38.152+0000 7f9682a05700 1 -- 172.21.15.110:0/3565990837 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f967c159e80 msgr2=0x7f967c0c5f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:38.159 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:38.152+0000 7f9682a05700 1 --2- 172.21.15.110:0/3565990837 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f967c159e80 0x7f967c0c5f40 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f9674009f20 tx=0x7f9674009840 comp rx=0 tx=0).stop 2023-10-06T03:30:38.159 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:38.152+0000 7f9682a05700 1 -- 172.21.15.110:0/3565990837 shutdown_connections 2023-10-06T03:30:38.159 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:38.152+0000 7f9682a05700 1 --2- 172.21.15.110:0/3565990837 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f967c159e80 0x7f967c0c5f40 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:38.159 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:38.152+0000 7f9682a05700 1 --2- 172.21.15.110:0/3565990837 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f967c155da0 0x7f967c06cc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:38.159 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:38.152+0000 7f9682a05700 1 -- 172.21.15.110:0/3565990837 >> 172.21.15.110:0/3565990837 conn(0x7f967c0cc470 msgr2=0x7f967c0c8410 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:30:38.159 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:38.152+0000 7f9682a05700 1 -- 172.21.15.110:0/3565990837 shutdown_connections 2023-10-06T03:30:38.159 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:38.152+0000 7f9682a05700 1 -- 172.21.15.110:0/3565990837 wait complete. 2023-10-06T03:30:38.159 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:38.152+0000 7f9682a05700 1 Processor -- start 2023-10-06T03:30:38.160 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:38.152+0000 7f9682a05700 1 -- start start 2023-10-06T03:30:38.161 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:38.152+0000 7f9682a05700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f967c1559d0 0x7f967c0c3a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:38.162 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:38.152+0000 7f9682a05700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f967c155da0 0x7f967c16ae30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:38.162 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:38.152+0000 7f9682a05700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f967c159e80 0x7f967c0ad680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:38.162 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:38.156+0000 7f9682a05700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f967c1438b0 con 0x7f967c1559d0 2023-10-06T03:30:38.162 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:38.156+0000 7f9682a05700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f967c0c4d70 con 0x7f967c155da0 2023-10-06T03:30:38.162 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:38.156+0000 7f9682a05700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f967c0adbc0 con 0x7f967c159e80 2023-10-06T03:30:38.162 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:38.156+0000 7f9681a03700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f967c1559d0 0x7f967c0c3a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:38.162 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:38.156+0000 7f9681a03700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f967c1559d0 0x7f967c0c3a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:48152/0 (socket says 172.21.15.110:48152) 2023-10-06T03:30:38.162 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:38.156+0000 7f9681202700 1 --2- 172.21.15.110:0/2158563431 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f967c155da0 0x7f967c16ae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:38.163 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:38.156+0000 7f9681a03700 1 -- 172.21.15.110:0/2158563431 learned_addr learned my addr 172.21.15.110:0/2158563431 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:30:38.163 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:38.156+0000 7f9682204700 1 --2- 172.21.15.110:0/2158563431 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f967c159e80 0x7f967c0ad680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:38.163 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:38.156+0000 7f9681a03700 1 -- 172.21.15.110:0/2158563431 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f967c159e80 msgr2=0x7f967c0ad680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:38.163 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:38.156+0000 7f9681a03700 1 --2- 172.21.15.110:0/2158563431 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f967c159e80 0x7f967c0ad680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:38.163 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:38.156+0000 7f9681a03700 1 -- 172.21.15.110:0/2158563431 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f967c155da0 msgr2=0x7f967c16ae30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:38.163 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:38.156+0000 7f9681a03700 1 --2- 172.21.15.110:0/2158563431 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f967c155da0 0x7f967c16ae30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:38.163 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:38.156+0000 7f9681a03700 1 -- 172.21.15.110:0/2158563431 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9674009ae0 con 0x7f967c1559d0 2023-10-06T03:30:38.163 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:38.156+0000 7f9681202700 1 --2- 172.21.15.110:0/2158563431 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f967c155da0 0x7f967c16ae30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-06T03:30:38.163 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:38.156+0000 7f9681a03700 1 --2- 172.21.15.110:0/2158563431 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f967c1559d0 0x7f967c0c3a60 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f96700011e0 tx=0x7f9670007e10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:30:38.163 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:38.156+0000 7f9679ffb700 1 -- 172.21.15.110:0/2158563431 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9670005a70 con 0x7f967c1559d0 2023-10-06T03:30:38.164 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:38.156+0000 7f9679ffb700 1 -- 172.21.15.110:0/2158563431 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9670005bd0 con 0x7f967c1559d0 2023-10-06T03:30:38.164 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:38.156+0000 7f9679ffb700 1 -- 172.21.15.110:0/2158563431 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96700070d0 con 0x7f967c1559d0 2023-10-06T03:30:38.164 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:38.156+0000 7f9682a05700 1 -- 172.21.15.110:0/2158563431 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f967c0c4160 con 0x7f967c1559d0 2023-10-06T03:30:38.164 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:38.156+0000 7f9682a05700 1 -- 172.21.15.110:0/2158563431 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f967c0c4720 con 0x7f967c1559d0 2023-10-06T03:30:38.165 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:38.156+0000 7f9682a05700 1 -- 172.21.15.110:0/2158563431 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f967c070ff0 con 0x7f967c1559d0 2023-10-06T03:30:38.165 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:38.156+0000 7f9679ffb700 1 -- 172.21.15.110:0/2158563431 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7f9670007440 con 0x7f967c1559d0 2023-10-06T03:30:38.165 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:38.160+0000 7f9679ffb700 1 --2- 172.21.15.110:0/2158563431 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f9668060550 0x7f9668062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:38.165 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:38.160+0000 7f9681202700 1 --2- 172.21.15.110:0/2158563431 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f9668060550 0x7f9668062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:38.165 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:38.160+0000 7f9679ffb700 1 -- 172.21.15.110:0/2158563431 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6378+0+0 (secure 0 0 0) 0x7f967007bfe0 con 0x7f967c1559d0 2023-10-06T03:30:38.165 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:38.160+0000 7f9681202700 1 --2- 172.21.15.110:0/2158563431 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f9668060550 0x7f9668062a00 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f967c16b4c0 tx=0x7f966c0091a0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:30:38.172 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:38.168+0000 7f9679ffb700 1 -- 172.21.15.110:0/2158563431 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f967004c2b0 con 0x7f967c1559d0 2023-10-06T03:30:38.333 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:38.328+0000 7f9682a05700 1 -- 172.21.15.110:0/2158563431 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f967c0ca480 con 0x7f967c1559d0 2023-10-06T03:30:38.334 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:38.328+0000 7f9679ffb700 1 -- 172.21.15.110:0/2158563431 <== mon.0 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v42) v1 ==== 74+0+17119 (secure 0 0 0) 0x7f9670017090 con 0x7f967c1559d0 2023-10-06T03:30:38.335 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-10-06T03:30:38.335 INFO:teuthology.orchestra.run.smithi110.stdout:{"epoch":42,"fsid":"93b8f6ac-4c92-4b45-b8e4-b59617024aa2","created":"2023-10-06T03:29:10.519583+0000","modified":"2023-10-06T03:30:35.644753+0000","last_up_change":"2023-10-06T03:29:21.960511+0000","last_in_change":"2023-10-06T03:29:12.560436+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-06T03:29:20.703743+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2023-10-06T03:29:28.307281+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2023-10-06T03:30:31.807897+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"41","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":7,"pool_name":"cephfs_data","create_time":"2023-10-06T03:30:32.991334+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"42","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"5866eb6c-2adc-46e4-be7a-e5bcda0f4065","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6824","nonce":33394},{"type":"v1","addr":"172.21.15.110:6825","nonce":33394}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6826","nonce":33394},{"type":"v1","addr":"172.21.15.110:6827","nonce":33394}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6830","nonce":33394},{"type":"v1","addr":"172.21.15.110:6831","nonce":33394}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6828","nonce":33394},{"type":"v1","addr":"172.21.15.110:6829","nonce":33394}]},"public_addr":"172.21.15.110:6825/33394","cluster_addr":"172.21.15.110:6827/33394","heartbeat_back_addr":"172.21.15.110:6831/33394","heartbeat_front_addr":"172.21.15.110:6829/33394","state":["exists","up"]},{"osd":1,"uuid":"a38de118-062b-4a05-9782-2ceddf6ff4df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6800","nonce":33396},{"type":"v1","addr":"172.21.15.110:6801","nonce":33396}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6802","nonce":33396},{"type":"v1","addr":"172.21.15.110:6803","nonce":33396}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6806","nonce":33396},{"type":"v1","addr":"172.21.15.110:6807","nonce":33396}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6804","nonce":33396},{"type":"v1","addr":"172.21.15.110:6805","nonce":33396}]},"public_addr":"172.21.15.110:6801/33396","cluster_addr":"172.21.15.110:6803/33396","heartbeat_back_addr":"172.21.15.110:6807/33396","heartbeat_front_addr":"172.21.15.110:6805/33396","state":["exists","up"]},{"osd":2,"uuid":"56e39be2-3407-4b69-877e-cf16617b06d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6816","nonce":33397},{"type":"v1","addr":"172.21.15.110:6817","nonce":33397}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6818","nonce":33397},{"type":"v1","addr":"172.21.15.110:6819","nonce":33397}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6822","nonce":33397},{"type":"v1","addr":"172.21.15.110:6823","nonce":33397}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6820","nonce":33397},{"type":"v1","addr":"172.21.15.110:6821","nonce":33397}]},"public_addr":"172.21.15.110:6817/33397","cluster_addr":"172.21.15.110:6819/33397","heartbeat_back_addr":"172.21.15.110:6823/33397","heartbeat_front_addr":"172.21.15.110:6821/33397","state":["exists","up"]},{"osd":3,"uuid":"01c82d13-c1d4-4bde-b55e-3e0089b40115","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6808","nonce":33395},{"type":"v1","addr":"172.21.15.110:6809","nonce":33395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6810","nonce":33395},{"type":"v1","addr":"172.21.15.110:6811","nonce":33395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6814","nonce":33395},{"type":"v1","addr":"172.21.15.110:6815","nonce":33395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6812","nonce":33395},{"type":"v1","addr":"172.21.15.110:6813","nonce":33395}]},"public_addr":"172.21.15.110:6809/33395","cluster_addr":"172.21.15.110:6811/33395","heartbeat_back_addr":"172.21.15.110:6815/33395","heartbeat_front_addr":"172.21.15.110:6813/33395","state":["exists","up"]},{"osd":4,"uuid":"86983430-5fa3-4d17-998f-03e2cd843237","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":33683},{"type":"v1","addr":"172.21.15.192:6817","nonce":33683}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":33683},{"type":"v1","addr":"172.21.15.192:6819","nonce":33683}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":33683},{"type":"v1","addr":"172.21.15.192:6823","nonce":33683}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6820","nonce":33683},{"type":"v1","addr":"172.21.15.192:6821","nonce":33683}]},"public_addr":"172.21.15.192:6817/33683","cluster_addr":"172.21.15.192:6819/33683","heartbeat_back_addr":"172.21.15.192:6823/33683","heartbeat_front_addr":"172.21.15.192:6821/33683","state":["exists","up"]},{"osd":5,"uuid":"6945257f-821d-4619-a63f-ebe857eba6f4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":33684},{"type":"v1","addr":"172.21.15.192:6801","nonce":33684}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":33684},{"type":"v1","addr":"172.21.15.192:6803","nonce":33684}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":33684},{"type":"v1","addr":"172.21.15.192:6807","nonce":33684}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6804","nonce":33684},{"type":"v1","addr":"172.21.15.192:6805","nonce":33684}]},"public_addr":"172.21.15.192:6801/33684","cluster_addr":"172.21.15.192:6803/33684","heartbeat_back_addr":"172.21.15.192:6807/33684","heartbeat_front_addr":"172.21.15.192:6805/33684","state":["exists","up"]},{"osd":6,"uuid":"7854d1ae-7060-4780-8f97-73cea5e7a12f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6824","nonce":33686},{"type":"v1","addr":"172.21.15.192:6825","nonce":33686}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":33686},{"type":"v1","addr":"172.21.15.192:6827","nonce":33686}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":33686},{"type":"v1","addr":"172.21.15.192:6831","nonce":33686}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6828","nonce":33686},{"type":"v1","addr":"172.21.15.192:6829","nonce":33686}]},"public_addr":"172.21.15.192:6825/33686","cluster_addr":"172.21.15.192:6827/33686","heartbeat_back_addr":"172.21.15.192:6831/33686","heartbeat_front_addr":"172.21.15.192:6829/33686","state":["exists","up"]},{"osd":7,"uuid":"f1ebdfd5-b859-482b-9f97-c18c4a26f445","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6808","nonce":33685},{"type":"v1","addr":"172.21.15.192:6809","nonce":33685}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":33685},{"type":"v1","addr":"172.21.15.192:6811","nonce":33685}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":33685},{"type":"v1","addr":"172.21.15.192:6815","nonce":33685}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6812","nonce":33685},{"type":"v1","addr":"172.21.15.192:6813","nonce":33685}]},"public_addr":"172.21.15.192:6809/33685","cluster_addr":"172.21.15.192:6811/33685","heartbeat_back_addr":"172.21.15.192:6815/33685","heartbeat_front_addr":"172.21.15.192:6813/33685","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T03:29:18.002048+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T03:29:17.478576+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T03:29:18.034944+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T03:29:17.628513+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T03:29:18.047338+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T03:29:17.772349+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T03:29:17.883958+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T03:29:18.151838+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-06T03:30:38.337 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:38.332+0000 7f9682a05700 1 -- 172.21.15.110:0/2158563431 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f9668060550 msgr2=0x7f9668062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:38.337 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:38.332+0000 7f9682a05700 1 --2- 172.21.15.110:0/2158563431 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f9668060550 0x7f9668062a00 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f967c16b4c0 tx=0x7f966c0091a0 comp rx=0 tx=0).stop 2023-10-06T03:30:38.337 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:38.332+0000 7f9682a05700 1 -- 172.21.15.110:0/2158563431 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f967c1559d0 msgr2=0x7f967c0c3a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:38.337 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:38.332+0000 7f9682a05700 1 --2- 172.21.15.110:0/2158563431 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f967c1559d0 0x7f967c0c3a60 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f96700011e0 tx=0x7f9670007e10 comp rx=0 tx=0).stop 2023-10-06T03:30:38.337 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:38.332+0000 7f9682a05700 1 -- 172.21.15.110:0/2158563431 shutdown_connections 2023-10-06T03:30:38.337 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:38.332+0000 7f9682a05700 1 --2- 172.21.15.110:0/2158563431 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f9668060550 0x7f9668062a00 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:38.337 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:38.332+0000 7f9682a05700 1 --2- 172.21.15.110:0/2158563431 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f967c159e80 0x7f967c0ad680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:38.338 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:38.332+0000 7f9682a05700 1 --2- 172.21.15.110:0/2158563431 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f967c155da0 0x7f967c16ae30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:38.338 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:38.332+0000 7f9682a05700 1 --2- 172.21.15.110:0/2158563431 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f967c1559d0 0x7f967c0c3a60 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:38.338 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:38.332+0000 7f9682a05700 1 -- 172.21.15.110:0/2158563431 >> 172.21.15.110:0/2158563431 conn(0x7f967c0cc470 msgr2=0x7f967c154d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:30:38.338 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:38.332+0000 7f9682a05700 1 -- 172.21.15.110:0/2158563431 shutdown_connections 2023-10-06T03:30:38.338 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:38.332+0000 7f9682a05700 1 -- 172.21.15.110:0/2158563431 wait complete. 2023-10-06T03:30:38.349 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2023-10-06T03:30:38.349 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-06T03:30:38.452 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:38.448+0000 7f173df4f700 1 Processor -- start 2023-10-06T03:30:38.453 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:38.448+0000 7f173df4f700 1 -- start start 2023-10-06T03:30:38.453 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:38.448+0000 7f173df4f700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1738156b80 0x7f17380c68d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:38.454 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:38.448+0000 7f173df4f700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f1738156f50 0x7f17380b0e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:38.454 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:38.448+0000 7f173df4f700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17380c7f30 con 0x7f17380c6e10 2023-10-06T03:30:38.454 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:38.448+0000 7f173df4f700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f17380c7320 con 0x7f1738156f50 2023-10-06T03:30:38.454 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:38.448+0000 7f173df4f700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17380c7600 con 0x7f1738156b80 2023-10-06T03:30:38.454 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:38.448+0000 7f1737fff700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f1738156f50 0x7f17380b0e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:38.454 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:38.448+0000 7f173cf4d700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1738156b80 0x7f17380c68d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:38.454 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:38.448+0000 7f1737fff700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f1738156f50 0x7f17380b0e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.110:48666/0 (socket says 172.21.15.110:48666) 2023-10-06T03:30:38.454 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:38.448+0000 7f1737fff700 1 -- 172.21.15.110:0/608974068 learned_addr learned my addr 172.21.15.110:0/608974068 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:30:38.455 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:38.448+0000 7f1737fff700 1 -- 172.21.15.110:0/608974068 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1738156b80 msgr2=0x7f17380c68d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:30:38.455 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:38.448+0000 7f1737fff700 1 --2- 172.21.15.110:0/608974068 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1738156b80 0x7f17380c68d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:38.455 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:38.448+0000 7f1737fff700 1 -- 172.21.15.110:0/608974068 >> v1:172.21.15.192:6789/0 conn(0x7f17380c6e10 legacy=0x7f17380ad560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:38.455 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:38.448+0000 7f1737fff700 1 -- 172.21.15.110:0/608974068 --> v2:172.21.15.192:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17380c78e0 con 0x7f1738156f50 2023-10-06T03:30:38.455 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:38.448+0000 7f1737fff700 1 --2- 172.21.15.110:0/608974068 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1738156f50 0x7f17380b0e40 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f1730009f20 tx=0x7f1730009840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=609aee2834a8fae6 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:30:38.456 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:38.448+0000 7f17377fe700 1 -- 172.21.15.110:0/608974068 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f173000e070 con 0x7f1738156f50 2023-10-06T03:30:38.456 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:38.448+0000 7f17377fe700 1 -- 172.21.15.110:0/608974068 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f173000b9d0 con 0x7f1738156f50 2023-10-06T03:30:38.456 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:38.448+0000 7f17377fe700 1 -- 172.21.15.110:0/608974068 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17300092e0 con 0x7f1738156f50 2023-10-06T03:30:38.456 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:38.452+0000 7f173df4f700 1 -- 172.21.15.110:0/608974068 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1738156f50 msgr2=0x7f17380b0e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:38.457 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:38.452+0000 7f173df4f700 1 --2- 172.21.15.110:0/608974068 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1738156f50 0x7f17380b0e40 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f1730009f20 tx=0x7f1730009840 comp rx=0 tx=0).stop 2023-10-06T03:30:38.457 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:38.452+0000 7f173df4f700 1 -- 172.21.15.110:0/608974068 shutdown_connections 2023-10-06T03:30:38.457 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:38.452+0000 7f173df4f700 1 --2- 172.21.15.110:0/608974068 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1738156f50 0x7f17380b0e40 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:38.457 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:38.452+0000 7f173df4f700 1 --2- 172.21.15.110:0/608974068 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1738156b80 0x7f17380c68d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:38.457 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:38.452+0000 7f173df4f700 1 -- 172.21.15.110:0/608974068 >> 172.21.15.110:0/608974068 conn(0x7f17380cc470 msgr2=0x7f17380c8410 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:30:38.457 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:38.452+0000 7f173df4f700 1 -- 172.21.15.110:0/608974068 shutdown_connections 2023-10-06T03:30:38.457 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:38.452+0000 7f173df4f700 1 -- 172.21.15.110:0/608974068 wait complete. 2023-10-06T03:30:38.458 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:38.452+0000 7f173df4f700 1 Processor -- start 2023-10-06T03:30:38.458 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:38.452+0000 7f173df4f700 1 -- start start 2023-10-06T03:30:38.461 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:38.456+0000 7f173df4f700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f17380c6e10 0x7f173814f180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:38.461 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:38.456+0000 7f173df4f700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1738156b80 0x7f1738167ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:38.461 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:38.456+0000 7f173df4f700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1738156f50 0x7f1738157b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:38.461 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:38.456+0000 7f173df4f700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f173814f6c0 con 0x7f1738156b80 2023-10-06T03:30:38.461 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:38.456+0000 7f173df4f700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17381576c0 con 0x7f17380c6e10 2023-10-06T03:30:38.461 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:38.456+0000 7f173df4f700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1738168850 con 0x7f1738156f50 2023-10-06T03:30:38.461 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:38.456+0000 7f1737fff700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1738156b80 0x7f1738167ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:38.461 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:38.456+0000 7f1737fff700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1738156b80 0x7f1738167ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:48156/0 (socket says 172.21.15.110:48156) 2023-10-06T03:30:38.461 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:38.456+0000 7f173cf4d700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f17380c6e10 0x7f173814f180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:38.461 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:38.456+0000 7f1737fff700 1 -- 172.21.15.110:0/4161901473 learned_addr learned my addr 172.21.15.110:0/4161901473 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:30:38.461 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:38.456+0000 7f173d74e700 1 --2- 172.21.15.110:0/4161901473 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1738156f50 0x7f1738157b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:38.462 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:38.456+0000 7f1737fff700 1 -- 172.21.15.110:0/4161901473 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1738156f50 msgr2=0x7f1738157b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:38.462 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:38.456+0000 7f1737fff700 1 --2- 172.21.15.110:0/4161901473 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1738156f50 0x7f1738157b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:38.462 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:38.456+0000 7f1737fff700 1 -- 172.21.15.110:0/4161901473 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f17380c6e10 msgr2=0x7f173814f180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:38.462 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:38.456+0000 7f1737fff700 1 --2- 172.21.15.110:0/4161901473 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f17380c6e10 0x7f173814f180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:38.462 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:38.456+0000 7f1737fff700 1 -- 172.21.15.110:0/4161901473 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1730009ae0 con 0x7f1738156b80 2023-10-06T03:30:38.462 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:38.456+0000 7f1737fff700 1 --2- 172.21.15.110:0/4161901473 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1738156b80 0x7f1738167ea0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f17380c6880 tx=0x7f1730018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:30:38.462 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:38.456+0000 7f1734ff9700 1 -- 172.21.15.110:0/4161901473 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17300088f0 con 0x7f1738156b80 2023-10-06T03:30:38.462 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:38.456+0000 7f1734ff9700 1 -- 172.21.15.110:0/4161901473 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1730008a50 con 0x7f1738156b80 2023-10-06T03:30:38.462 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:38.456+0000 7f173df4f700 1 -- 172.21.15.110:0/4161901473 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1738168b30 con 0x7f1738156b80 2023-10-06T03:30:38.462 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:38.456+0000 7f1734ff9700 1 -- 172.21.15.110:0/4161901473 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1730008dd0 con 0x7f1738156b80 2023-10-06T03:30:38.462 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:38.456+0000 7f173df4f700 1 -- 172.21.15.110:0/4161901473 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1738157350 con 0x7f1738156b80 2023-10-06T03:30:38.463 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:38.456+0000 7f173df4f700 1 -- 172.21.15.110:0/4161901473 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1738157850 con 0x7f1738156b80 2023-10-06T03:30:38.463 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:38.456+0000 7f1734ff9700 1 -- 172.21.15.110:0/4161901473 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7f1730003040 con 0x7f1738156b80 2023-10-06T03:30:38.463 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:38.460+0000 7f1734ff9700 1 --2- 172.21.15.110:0/4161901473 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f1718060580 0x7f1718062a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:38.464 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:38.460+0000 7f1734ff9700 1 -- 172.21.15.110:0/4161901473 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6378+0+0 (secure 0 0 0) 0x7f173007dfd0 con 0x7f1738156b80 2023-10-06T03:30:38.464 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:38.460+0000 7f173cf4d700 1 --2- 172.21.15.110:0/4161901473 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f1718060580 0x7f1718062a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:38.464 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:38.460+0000 7f173cf4d700 1 --2- 172.21.15.110:0/4161901473 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f1718060580 0x7f1718062a30 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f1720009db0 tx=0x7f17200091a0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:30:38.475 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:38.472+0000 7f1734ff9700 1 -- 172.21.15.110:0/4161901473 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f173004e320 con 0x7f1738156b80 2023-10-06T03:30:38.638 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:38.632+0000 7f173df4f700 1 -- 172.21.15.110:0/4161901473 --> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f1738070ff0 con 0x7f1718060580 2023-10-06T03:30:38.658 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:38.652+0000 7f1734ff9700 1 -- 172.21.15.110:0/4161901473 <== mgr.4110 v2:172.21.15.110:6832/33268 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+300704 (secure 0 0 0) 0x7f1738070ff0 con 0x7f1718060580 2023-10-06T03:30:38.662 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-10-06T03:30:38.664 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:38.656+0000 7f173df4f700 1 -- 172.21.15.110:0/4161901473 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f1718060580 msgr2=0x7f1718062a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:38.664 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:38.656+0000 7f173df4f700 1 --2- 172.21.15.110:0/4161901473 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f1718060580 0x7f1718062a30 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f1720009db0 tx=0x7f17200091a0 comp rx=0 tx=0).stop 2023-10-06T03:30:38.664 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:38.656+0000 7f173df4f700 1 -- 172.21.15.110:0/4161901473 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1738156b80 msgr2=0x7f1738167ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:38.664 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:38.656+0000 7f173df4f700 1 --2- 172.21.15.110:0/4161901473 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1738156b80 0x7f1738167ea0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f17380c6880 tx=0x7f1730018000 comp rx=0 tx=0).stop 2023-10-06T03:30:38.664 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:38.656+0000 7f173df4f700 1 -- 172.21.15.110:0/4161901473 shutdown_connections 2023-10-06T03:30:38.665 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:38.656+0000 7f173df4f700 1 --2- 172.21.15.110:0/4161901473 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f1718060580 0x7f1718062a30 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:38.665 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:38.656+0000 7f173df4f700 1 --2- 172.21.15.110:0/4161901473 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1738156f50 0x7f1738157b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:38.665 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:38.656+0000 7f173df4f700 1 --2- 172.21.15.110:0/4161901473 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1738156b80 0x7f1738167ea0 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:38.665 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:38.656+0000 7f173df4f700 1 --2- 172.21.15.110:0/4161901473 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f17380c6e10 0x7f173814f180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:38.665 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:38.656+0000 7f173df4f700 1 -- 172.21.15.110:0/4161901473 >> 172.21.15.110:0/4161901473 conn(0x7f17380cc470 msgr2=0x7f17380c49c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:30:38.665 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:38.656+0000 7f173df4f700 1 -- 172.21.15.110:0/4161901473 shutdown_connections 2023-10-06T03:30:38.665 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:38.656+0000 7f173df4f700 1 -- 172.21.15.110:0/4161901473 wait complete. 2023-10-06T03:30:38.665 INFO:teuthology.orchestra.run.smithi110.stderr:dumped all 2023-10-06T03:30:38.676 INFO:teuthology.orchestra.run.smithi110.stdout:{"pg_ready":true,"pg_map":{"version":73,"stamp":"2023-10-06T03:30:36.670397+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592514,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":177,"num_write_kb":2427,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":135,"ondisk_log_size":135,"up":96,"acting":96,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":28,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2399328,"kb_used_data":4640,"kb_used_omap":0,"kb_used_meta":2394624,"kb_avail":752575392,"statfs":{"total":773094113280,"available":770637201408,"internally_reserved":0,"allocated":4751360,"data_stored":2730122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2452094976},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":18,"apply_latency_ms":18,"commit_latency_ns":18000000,"apply_latency_ns":18000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2146,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.042843"},"pg_stats":[{"pgid":"7.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-06T03:30:33.627150+0000","last_change":"2023-10-06T03:30:33.627150+0000","last_active":"2023-10-06T03:30:33.627150+0000","last_peered":"2023-10-06T03:30:33.627150+0000","last_clean":"2023-10-06T03:30:33.627150+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-06T03:30:33.627150+0000","last_undegraded":"2023-10-06T03:30:33.627150+0000","last_fullsized":"2023-10-06T03:30:33.627150+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:30:33.627150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:30:33.627150+0000","last_clean_scrub_stamp":"2023-10-06T03:30:33.627150+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-06T03:30:33.627150+0000","last_change":"2023-10-06T03:30:33.627150+0000","last_active":"2023-10-06T03:30:33.627150+0000","last_peered":"2023-10-06T03:30:33.627150+0000","last_clean":"2023-10-06T03:30:33.627150+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-06T03:30:33.627150+0000","last_undegraded":"2023-10-06T03:30:33.627150+0000","last_fullsized":"2023-10-06T03:30:33.627150+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:30:33.627150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:30:33.627150+0000","last_clean_scrub_stamp":"2023-10-06T03:30:33.627150+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-06T03:30:33.627150+0000","last_change":"2023-10-06T03:30:33.627150+0000","last_active":"2023-10-06T03:30:33.627150+0000","last_peered":"2023-10-06T03:30:33.627150+0000","last_clean":"2023-10-06T03:30:33.627150+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-06T03:30:33.627150+0000","last_undegraded":"2023-10-06T03:30:33.627150+0000","last_fullsized":"2023-10-06T03:30:33.627150+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:30:33.627150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:30:33.627150+0000","last_clean_scrub_stamp":"2023-10-06T03:30:33.627150+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-06T03:30:33.627150+0000","last_change":"2023-10-06T03:30:33.627150+0000","last_active":"2023-10-06T03:30:33.627150+0000","last_peered":"2023-10-06T03:30:33.627150+0000","last_clean":"2023-10-06T03:30:33.627150+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-06T03:30:33.627150+0000","last_undegraded":"2023-10-06T03:30:33.627150+0000","last_fullsized":"2023-10-06T03:30:33.627150+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:30:33.627150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:30:33.627150+0000","last_clean_scrub_stamp":"2023-10-06T03:30:33.627150+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-06T03:30:33.627150+0000","last_change":"2023-10-06T03:30:33.627150+0000","last_active":"2023-10-06T03:30:33.627150+0000","last_peered":"2023-10-06T03:30:33.627150+0000","last_clean":"2023-10-06T03:30:33.627150+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-06T03:30:33.627150+0000","last_undegraded":"2023-10-06T03:30:33.627150+0000","last_fullsized":"2023-10-06T03:30:33.627150+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:30:33.627150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:30:33.627150+0000","last_clean_scrub_stamp":"2023-10-06T03:30:33.627150+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.3a","version":"0'0","reported_seq":9,"reported_epoch":41,"state":"active+clean","last_fresh":"2023-10-06T03:30:34.668332+0000","last_change":"2023-10-06T03:30:34.666278+0000","last_active":"2023-10-06T03:30:34.668332+0000","last_peered":"2023-10-06T03:30:34.668332+0000","last_clean":"2023-10-06T03:30:34.668332+0000","last_became_active":"2023-10-06T03:30:34.665206+0000","last_became_peered":"2023-10-06T03:30:34.665206+0000","last_unstale":"2023-10-06T03:30:34.668332+0000","last_undegraded":"2023-10-06T03:30:34.668332+0000","last_fullsized":"2023-10-06T03:30:34.668332+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:30:33.627150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:30:33.627150+0000","last_clean_scrub_stamp":"2023-10-06T03:30:33.627150+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T11:01:12.279639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-06T03:30:33.627150+0000","last_change":"2023-10-06T03:30:33.627150+0000","last_active":"2023-10-06T03:30:33.627150+0000","last_peered":"2023-10-06T03:30:33.627150+0000","last_clean":"2023-10-06T03:30:33.627150+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-06T03:30:33.627150+0000","last_undegraded":"2023-10-06T03:30:33.627150+0000","last_fullsized":"2023-10-06T03:30:33.627150+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:30:33.627150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:30:33.627150+0000","last_clean_scrub_stamp":"2023-10-06T03:30:33.627150+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-06T03:30:33.627150+0000","last_change":"2023-10-06T03:30:33.627150+0000","last_active":"2023-10-06T03:30:33.627150+0000","last_peered":"2023-10-06T03:30:33.627150+0000","last_clean":"2023-10-06T03:30:33.627150+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-06T03:30:33.627150+0000","last_undegraded":"2023-10-06T03:30:33.627150+0000","last_fullsized":"2023-10-06T03:30:33.627150+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:30:33.627150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:30:33.627150+0000","last_clean_scrub_stamp":"2023-10-06T03:30:33.627150+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-06T03:30:33.627150+0000","last_change":"2023-10-06T03:30:33.627150+0000","last_active":"2023-10-06T03:30:33.627150+0000","last_peered":"2023-10-06T03:30:33.627150+0000","last_clean":"2023-10-06T03:30:33.627150+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-06T03:30:33.627150+0000","last_undegraded":"2023-10-06T03:30:33.627150+0000","last_fullsized":"2023-10-06T03:30:33.627150+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:30:33.627150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:30:33.627150+0000","last_clean_scrub_stamp":"2023-10-06T03:30:33.627150+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-06T03:30:33.627150+0000","last_change":"2023-10-06T03:30:33.627150+0000","last_active":"2023-10-06T03:30:33.627150+0000","last_peered":"2023-10-06T03:30:33.627150+0000","last_clean":"2023-10-06T03:30:33.627150+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-06T03:30:33.627150+0000","last_undegraded":"2023-10-06T03:30:33.627150+0000","last_fullsized":"2023-10-06T03:30:33.627150+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:30:33.627150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:30:33.627150+0000","last_clean_scrub_stamp":"2023-10-06T03:30:33.627150+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-06T03:30:33.627150+0000","last_change":"2023-10-06T03:30:33.627150+0000","last_active":"2023-10-06T03:30:33.627150+0000","last_peered":"2023-10-06T03:30:33.627150+0000","last_clean":"2023-10-06T03:30:33.627150+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-06T03:30:33.627150+0000","last_undegraded":"2023-10-06T03:30:33.627150+0000","last_fullsized":"2023-10-06T03:30:33.627150+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:30:33.627150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:30:33.627150+0000","last_clean_scrub_stamp":"2023-10-06T03:30:33.627150+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-06T03:30:33.627150+0000","last_change":"2023-10-06T03:30:33.627150+0000","last_active":"2023-10-06T03:30:33.627150+0000","last_peered":"2023-10-06T03:30:33.627150+0000","last_clean":"2023-10-06T03:30:33.627150+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-06T03:30:33.627150+0000","last_undegraded":"2023-10-06T03:30:33.627150+0000","last_fullsized":"2023-10-06T03:30:33.627150+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:30:33.627150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:30:33.627150+0000","last_clean_scrub_stamp":"2023-10-06T03:30:33.627150+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.33","version":"0'0","reported_seq":8,"reported_epoch":41,"state":"active+clean","last_fresh":"2023-10-06T03:30:34.669087+0000","last_change":"2023-10-06T03:30:34.669087+0000","last_active":"2023-10-06T03:30:34.669087+0000","last_peered":"2023-10-06T03:30:34.669087+0000","last_clean":"2023-10-06T03:30:34.669087+0000","last_became_active":"2023-10-06T03:30:34.668804+0000","last_became_peered":"2023-10-06T03:30:34.668804+0000","last_unstale":"2023-10-06T03:30:34.669087+0000","last_undegraded":"2023-10-06T03:30:34.669087+0000","last_fullsized":"2023-10-06T03:30:34.669087+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:30:33.627150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:30:33.627150+0000","last_clean_scrub_stamp":"2023-10-06T03:30:33.627150+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T10:56:57.565128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-06T03:30:33.627150+0000","last_change":"2023-10-06T03:30:33.627150+0000","last_active":"2023-10-06T03:30:33.627150+0000","last_peered":"2023-10-06T03:30:33.627150+0000","last_clean":"2023-10-06T03:30:33.627150+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-06T03:30:33.627150+0000","last_undegraded":"2023-10-06T03:30:33.627150+0000","last_fullsized":"2023-10-06T03:30:33.627150+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:30:33.627150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:30:33.627150+0000","last_clean_scrub_stamp":"2023-10-06T03:30:33.627150+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.31","version":"0'0","reported_seq":9,"reported_epoch":41,"state":"active+clean","last_fresh":"2023-10-06T03:30:34.668446+0000","last_change":"2023-10-06T03:30:34.657238+0000","last_active":"2023-10-06T03:30:34.668446+0000","last_peered":"2023-10-06T03:30:34.668446+0000","last_clean":"2023-10-06T03:30:34.668446+0000","last_became_active":"2023-10-06T03:30:34.656927+0000","last_became_peered":"2023-10-06T03:30:34.656927+0000","last_unstale":"2023-10-06T03:30:34.668446+0000","last_undegraded":"2023-10-06T03:30:34.668446+0000","last_fullsized":"2023-10-06T03:30:34.668446+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:30:33.627150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:30:33.627150+0000","last_clean_scrub_stamp":"2023-10-06T03:30:33.627150+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T09:06:23.239915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-06T03:30:33.627150+0000","last_change":"2023-10-06T03:30:33.627150+0000","last_active":"2023-10-06T03:30:33.627150+0000","last_peered":"2023-10-06T03:30:33.627150+0000","last_clean":"2023-10-06T03:30:33.627150+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-06T03:30:33.627150+0000","last_undegraded":"2023-10-06T03:30:33.627150+0000","last_fullsized":"2023-10-06T03:30:33.627150+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:30:33.627150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:30:33.627150+0000","last_clean_scrub_stamp":"2023-10-06T03:30:33.627150+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-06T03:30:33.627150+0000","last_change":"2023-10-06T03:30:33.627150+0000","last_active":"2023-10-06T03:30:33.627150+0000","last_peered":"2023-10-06T03:30:33.627150+0000","last_clean":"2023-10-06T03:30:33.627150+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-06T03:30:33.627150+0000","last_undegraded":"2023-10-06T03:30:33.627150+0000","last_fullsized":"2023-10-06T03:30:33.627150+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:30:33.627150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:30:33.627150+0000","last_clean_scrub_stamp":"2023-10-06T03:30:33.627150+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-06T03:30:33.627150+0000","last_change":"2023-10-06T03:30:33.627150+0000","last_active":"2023-10-06T03:30:33.627150+0000","last_peered":"2023-10-06T03:30:33.627150+0000","last_clean":"2023-10-06T03:30:33.627150+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-06T03:30:33.627150+0000","last_undegraded":"2023-10-06T03:30:33.627150+0000","last_fullsized":"2023-10-06T03:30:33.627150+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:30:33.627150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:30:33.627150+0000","last_clean_scrub_stamp":"2023-10-06T03:30:33.627150+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-06T03:30:33.627150+0000","last_change":"2023-10-06T03:30:33.627150+0000","last_active":"2023-10-06T03:30:33.627150+0000","last_peered":"2023-10-06T03:30:33.627150+0000","last_clean":"2023-10-06T03:30:33.627150+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-06T03:30:33.627150+0000","last_undegraded":"2023-10-06T03:30:33.627150+0000","last_fullsized":"2023-10-06T03:30:33.627150+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:30:33.627150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:30:33.627150+0000","last_clean_scrub_stamp":"2023-10-06T03:30:33.627150+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.2c","version":"0'0","reported_seq":8,"reported_epoch":41,"state":"active+clean","last_fresh":"2023-10-06T03:30:34.685955+0000","last_change":"2023-10-06T03:30:34.685955+0000","last_active":"2023-10-06T03:30:34.685955+0000","last_peered":"2023-10-06T03:30:34.685955+0000","last_clean":"2023-10-06T03:30:34.685955+0000","last_became_active":"2023-10-06T03:30:34.685624+0000","last_became_peered":"2023-10-06T03:30:34.685624+0000","last_unstale":"2023-10-06T03:30:34.685955+0000","last_undegraded":"2023-10-06T03:30:34.685955+0000","last_fullsized":"2023-10-06T03:30:34.685955+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:30:33.627150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:30:33.627150+0000","last_clean_scrub_stamp":"2023-10-06T03:30:33.627150+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T10:53:07.184282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.2b","version":"0'0","reported_seq":9,"reported_epoch":41,"state":"active+clean","last_fresh":"2023-10-06T03:30:34.673513+0000","last_change":"2023-10-06T03:30:34.668467+0000","last_active":"2023-10-06T03:30:34.673513+0000","last_peered":"2023-10-06T03:30:34.673513+0000","last_clean":"2023-10-06T03:30:34.673513+0000","last_became_active":"2023-10-06T03:30:34.668171+0000","last_became_peered":"2023-10-06T03:30:34.668171+0000","last_unstale":"2023-10-06T03:30:34.673513+0000","last_undegraded":"2023-10-06T03:30:34.673513+0000","last_fullsized":"2023-10-06T03:30:34.673513+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:30:33.627150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:30:33.627150+0000","last_clean_scrub_stamp":"2023-10-06T03:30:33.627150+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T09:19:27.526549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-06T03:30:33.627150+0000","last_change":"2023-10-06T03:30:33.627150+0000","last_active":"2023-10-06T03:30:33.627150+0000","last_peered":"2023-10-06T03:30:33.627150+0000","last_clean":"2023-10-06T03:30:33.627150+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-06T03:30:33.627150+0000","last_undegraded":"2023-10-06T03:30:33.627150+0000","last_fullsized":"2023-10-06T03:30:33.627150+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:30:33.627150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:30:33.627150+0000","last_clean_scrub_stamp":"2023-10-06T03:30:33.627150+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-06T03:30:33.627150+0000","last_change":"2023-10-06T03:30:33.627150+0000","last_active":"2023-10-06T03:30:33.627150+0000","last_peered":"2023-10-06T03:30:33.627150+0000","last_clean":"2023-10-06T03:30:33.627150+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-06T03:30:33.627150+0000","last_undegraded":"2023-10-06T03:30:33.627150+0000","last_fullsized":"2023-10-06T03:30:33.627150+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:30:33.627150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:30:33.627150+0000","last_clean_scrub_stamp":"2023-10-06T03:30:33.627150+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-06T03:30:33.627150+0000","last_change":"2023-10-06T03:30:33.627150+0000","last_active":"2023-10-06T03:30:33.627150+0000","last_peered":"2023-10-06T03:30:33.627150+0000","last_clean":"2023-10-06T03:30:33.627150+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-06T03:30:33.627150+0000","last_undegraded":"2023-10-06T03:30:33.627150+0000","last_fullsized":"2023-10-06T03:30:33.627150+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:30:33.627150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:30:33.627150+0000","last_clean_scrub_stamp":"2023-10-06T03:30:33.627150+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-06T03:30:33.627150+0000","last_change":"2023-10-06T03:30:33.627150+0000","last_active":"2023-10-06T03:30:33.627150+0000","last_peered":"2023-10-06T03:30:33.627150+0000","last_clean":"2023-10-06T03:30:33.627150+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-06T03:30:33.627150+0000","last_undegraded":"2023-10-06T03:30:33.627150+0000","last_fullsized":"2023-10-06T03:30:33.627150+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:30:33.627150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:30:33.627150+0000","last_clean_scrub_stamp":"2023-10-06T03:30:33.627150+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.26","version":"0'0","reported_seq":8,"reported_epoch":41,"state":"active+clean","last_fresh":"2023-10-06T03:30:34.669291+0000","last_change":"2023-10-06T03:30:34.669291+0000","last_active":"2023-10-06T03:30:34.669291+0000","last_peered":"2023-10-06T03:30:34.669291+0000","last_clean":"2023-10-06T03:30:34.669291+0000","last_became_active":"2023-10-06T03:30:34.668770+0000","last_became_peered":"2023-10-06T03:30:34.668770+0000","last_unstale":"2023-10-06T03:30:34.669291+0000","last_undegraded":"2023-10-06T03:30:34.669291+0000","last_fullsized":"2023-10-06T03:30:34.669291+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:30:33.627150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:30:33.627150+0000","last_clean_scrub_stamp":"2023-10-06T03:30:33.627150+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T08:26:46.586078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-06T03:30:33.627150+0000","last_change":"2023-10-06T03:30:33.627150+0000","last_active":"2023-10-06T03:30:33.627150+0000","last_peered":"2023-10-06T03:30:33.627150+0000","last_clean":"2023-10-06T03:30:33.627150+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-06T03:30:33.627150+0000","last_undegraded":"2023-10-06T03:30:33.627150+0000","last_fullsized":"2023-10-06T03:30:33.627150+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:30:33.627150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:30:33.627150+0000","last_clean_scrub_stamp":"2023-10-06T03:30:33.627150+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-06T03:30:33.627150+0000","last_change":"2023-10-06T03:30:33.627150+0000","last_active":"2023-10-06T03:30:33.627150+0000","last_peered":"2023-10-06T03:30:33.627150+0000","last_clean":"2023-10-06T03:30:33.627150+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-06T03:30:33.627150+0000","last_undegraded":"2023-10-06T03:30:33.627150+0000","last_fullsized":"2023-10-06T03:30:33.627150+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:30:33.627150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:30:33.627150+0000","last_clean_scrub_stamp":"2023-10-06T03:30:33.627150+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-06T03:30:33.627150+0000","last_change":"2023-10-06T03:30:33.627150+0000","last_active":"2023-10-06T03:30:33.627150+0000","last_peered":"2023-10-06T03:30:33.627150+0000","last_clean":"2023-10-06T03:30:33.627150+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-06T03:30:33.627150+0000","last_undegraded":"2023-10-06T03:30:33.627150+0000","last_fullsized":"2023-10-06T03:30:33.627150+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:30:33.627150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:30:33.627150+0000","last_clean_scrub_stamp":"2023-10-06T03:30:33.627150+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-06T03:30:33.627150+0000","last_change":"2023-10-06T03:30:33.627150+0000","last_active":"2023-10-06T03:30:33.627150+0000","last_peered":"2023-10-06T03:30:33.627150+0000","last_clean":"2023-10-06T03:30:33.627150+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-06T03:30:33.627150+0000","last_undegraded":"2023-10-06T03:30:33.627150+0000","last_fullsized":"2023-10-06T03:30:33.627150+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:30:33.627150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:30:33.627150+0000","last_clean_scrub_stamp":"2023-10-06T03:30:33.627150+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.21","version":"0'0","reported_seq":9,"reported_epoch":41,"state":"active+clean","last_fresh":"2023-10-06T03:30:34.673558+0000","last_change":"2023-10-06T03:30:34.669467+0000","last_active":"2023-10-06T03:30:34.673558+0000","last_peered":"2023-10-06T03:30:34.673558+0000","last_clean":"2023-10-06T03:30:34.673558+0000","last_became_active":"2023-10-06T03:30:34.668221+0000","last_became_peered":"2023-10-06T03:30:34.668221+0000","last_unstale":"2023-10-06T03:30:34.673558+0000","last_undegraded":"2023-10-06T03:30:34.673558+0000","last_fullsized":"2023-10-06T03:30:34.673558+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:30:33.627150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:30:33.627150+0000","last_clean_scrub_stamp":"2023-10-06T03:30:33.627150+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T08:10:34.526977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.20","version":"0'0","reported_seq":8,"reported_epoch":41,"state":"active+clean","last_fresh":"2023-10-06T03:30:34.671765+0000","last_change":"2023-10-06T03:30:34.671765+0000","last_active":"2023-10-06T03:30:34.671765+0000","last_peered":"2023-10-06T03:30:34.671765+0000","last_clean":"2023-10-06T03:30:34.671765+0000","last_became_active":"2023-10-06T03:30:34.671477+0000","last_became_peered":"2023-10-06T03:30:34.671477+0000","last_unstale":"2023-10-06T03:30:34.671765+0000","last_undegraded":"2023-10-06T03:30:34.671765+0000","last_fullsized":"2023-10-06T03:30:34.671765+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:30:33.627150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:30:33.627150+0000","last_clean_scrub_stamp":"2023-10-06T03:30:33.627150+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T11:13:30.805084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.1e","version":"0'0","reported_seq":8,"reported_epoch":41,"state":"active+clean","last_fresh":"2023-10-06T03:30:34.666944+0000","last_change":"2023-10-06T03:30:34.666944+0000","last_active":"2023-10-06T03:30:34.666944+0000","last_peered":"2023-10-06T03:30:34.666944+0000","last_clean":"2023-10-06T03:30:34.666944+0000","last_became_active":"2023-10-06T03:30:34.666627+0000","last_became_peered":"2023-10-06T03:30:34.666627+0000","last_unstale":"2023-10-06T03:30:34.666944+0000","last_undegraded":"2023-10-06T03:30:34.666944+0000","last_fullsized":"2023-10-06T03:30:34.666944+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:30:33.627150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:30:33.627150+0000","last_clean_scrub_stamp":"2023-10-06T03:30:33.627150+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T11:46:04.452767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"41'4","reported_seq":14,"reported_epoch":41,"state":"active+clean","last_fresh":"2023-10-06T03:30:34.688144+0000","last_change":"2023-10-06T03:30:33.667669+0000","last_active":"2023-10-06T03:30:34.688144+0000","last_peered":"2023-10-06T03:30:34.688144+0000","last_clean":"2023-10-06T03:30:34.688144+0000","last_became_active":"2023-10-06T03:30:33.667335+0000","last_became_peered":"2023-10-06T03:30:33.667335+0000","last_unstale":"2023-10-06T03:30:34.688144+0000","last_undegraded":"2023-10-06T03:30:34.688144+0000","last_fullsized":"2023-10-06T03:30:34.688144+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:30:32.615612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:30:32.615612+0000","last_clean_scrub_stamp":"2023-10-06T03:30:32.615612+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T07:00:29.333955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":554,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-06T03:30:33.627150+0000","last_change":"2023-10-06T03:30:33.627150+0000","last_active":"2023-10-06T03:30:33.627150+0000","last_peered":"2023-10-06T03:30:33.627150+0000","last_clean":"2023-10-06T03:30:33.627150+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-06T03:30:33.627150+0000","last_undegraded":"2023-10-06T03:30:33.627150+0000","last_fullsized":"2023-10-06T03:30:33.627150+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:30:33.627150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:30:33.627150+0000","last_clean_scrub_stamp":"2023-10-06T03:30:33.627150+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-10-06T03:30:32.615612+0000","last_change":"2023-10-06T03:30:32.642473+0000","last_active":"2023-10-06T03:30:32.615612+0000","last_peered":"2023-10-06T03:30:32.615612+0000","last_clean":"2023-10-06T03:30:32.615612+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-06T03:30:32.615612+0000","last_undegraded":"2023-10-06T03:30:32.615612+0000","last_fullsized":"2023-10-06T03:30:32.615612+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:30:32.615612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:30:32.615612+0000","last_clean_scrub_stamp":"2023-10-06T03:30:32.615612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T13:38:35.796680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[6],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-06T03:30:33.627150+0000","last_change":"2023-10-06T03:30:33.627150+0000","last_active":"2023-10-06T03:30:33.627150+0000","last_peered":"2023-10-06T03:30:33.627150+0000","last_clean":"2023-10-06T03:30:33.627150+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-06T03:30:33.627150+0000","last_undegraded":"2023-10-06T03:30:33.627150+0000","last_fullsized":"2023-10-06T03:30:33.627150+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:30:33.627150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:30:33.627150+0000","last_clean_scrub_stamp":"2023-10-06T03:30:33.627150+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.1d","version":"41'2","reported_seq":11,"reported_epoch":41,"state":"active+clean","last_fresh":"2023-10-06T03:30:34.684098+0000","last_change":"2023-10-06T03:30:33.670694+0000","last_active":"2023-10-06T03:30:34.684098+0000","last_peered":"2023-10-06T03:30:34.684098+0000","last_clean":"2023-10-06T03:30:34.684098+0000","last_became_active":"2023-10-06T03:30:33.670328+0000","last_became_peered":"2023-10-06T03:30:33.670328+0000","last_unstale":"2023-10-06T03:30:34.684098+0000","last_undegraded":"2023-10-06T03:30:34.684098+0000","last_fullsized":"2023-10-06T03:30:34.684098+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:30:32.615612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:30:32.615612+0000","last_clean_scrub_stamp":"2023-10-06T03:30:32.615612+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T10:27:44.696525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1d","version":"0'0","reported_seq":8,"reported_epoch":41,"state":"active+clean","last_fresh":"2023-10-06T03:30:34.675677+0000","last_change":"2023-10-06T03:30:34.675677+0000","last_active":"2023-10-06T03:30:34.675677+0000","last_peered":"2023-10-06T03:30:34.675677+0000","last_clean":"2023-10-06T03:30:34.675677+0000","last_became_active":"2023-10-06T03:30:34.675303+0000","last_became_peered":"2023-10-06T03:30:34.675303+0000","last_unstale":"2023-10-06T03:30:34.675677+0000","last_undegraded":"2023-10-06T03:30:34.675677+0000","last_fullsized":"2023-10-06T03:30:34.675677+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:30:33.627150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:30:33.627150+0000","last_clean_scrub_stamp":"2023-10-06T03:30:33.627150+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T11:10:52.361888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-06T03:30:32.615612+0000","last_change":"2023-10-06T03:30:32.615612+0000","last_active":"2023-10-06T03:30:32.615612+0000","last_peered":"2023-10-06T03:30:32.615612+0000","last_clean":"2023-10-06T03:30:32.615612+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-06T03:30:32.615612+0000","last_undegraded":"2023-10-06T03:30:32.615612+0000","last_fullsized":"2023-10-06T03:30:32.615612+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:30:32.615612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:30:32.615612+0000","last_clean_scrub_stamp":"2023-10-06T03:30:32.615612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-06T03:30:33.627150+0000","last_change":"2023-10-06T03:30:33.627150+0000","last_active":"2023-10-06T03:30:33.627150+0000","last_peered":"2023-10-06T03:30:33.627150+0000","last_clean":"2023-10-06T03:30:33.627150+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-06T03:30:33.627150+0000","last_undegraded":"2023-10-06T03:30:33.627150+0000","last_fullsized":"2023-10-06T03:30:33.627150+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:30:33.627150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:30:33.627150+0000","last_clean_scrub_stamp":"2023-10-06T03:30:33.627150+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.1b","version":"41'1","reported_seq":11,"reported_epoch":41,"state":"active+clean","last_fresh":"2023-10-06T03:30:34.687069+0000","last_change":"2023-10-06T03:30:33.667614+0000","last_active":"2023-10-06T03:30:34.687069+0000","last_peered":"2023-10-06T03:30:34.687069+0000","last_clean":"2023-10-06T03:30:34.687069+0000","last_became_active":"2023-10-06T03:30:33.665468+0000","last_became_peered":"2023-10-06T03:30:33.665468+0000","last_unstale":"2023-10-06T03:30:34.687069+0000","last_undegraded":"2023-10-06T03:30:34.687069+0000","last_fullsized":"2023-10-06T03:30:34.687069+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:30:32.615612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:30:32.615612+0000","last_clean_scrub_stamp":"2023-10-06T03:30:32.615612+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T11:17:50.406093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-06T03:30:33.627150+0000","last_change":"2023-10-06T03:30:33.627150+0000","last_active":"2023-10-06T03:30:33.627150+0000","last_peered":"2023-10-06T03:30:33.627150+0000","last_clean":"2023-10-06T03:30:33.627150+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-06T03:30:33.627150+0000","last_undegraded":"2023-10-06T03:30:33.627150+0000","last_fullsized":"2023-10-06T03:30:33.627150+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:30:33.627150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:30:33.627150+0000","last_clean_scrub_stamp":"2023-10-06T03:30:33.627150+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-10-06T03:30:32.615612+0000","last_change":"2023-10-06T03:30:32.641487+0000","last_active":"2023-10-06T03:30:32.615612+0000","last_peered":"2023-10-06T03:30:32.615612+0000","last_clean":"2023-10-06T03:30:32.615612+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-06T03:30:32.615612+0000","last_undegraded":"2023-10-06T03:30:32.615612+0000","last_fullsized":"2023-10-06T03:30:32.615612+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:30:32.615612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:30:32.615612+0000","last_clean_scrub_stamp":"2023-10-06T03:30:32.615612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T04:56:39.607886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[5],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-06T03:30:33.627150+0000","last_change":"2023-10-06T03:30:33.627150+0000","last_active":"2023-10-06T03:30:33.627150+0000","last_peered":"2023-10-06T03:30:33.627150+0000","last_clean":"2023-10-06T03:30:33.627150+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-06T03:30:33.627150+0000","last_undegraded":"2023-10-06T03:30:33.627150+0000","last_fullsized":"2023-10-06T03:30:33.627150+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:30:33.627150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:30:33.627150+0000","last_clean_scrub_stamp":"2023-10-06T03:30:33.627150+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-10-06T03:30:32.615612+0000","last_change":"2023-10-06T03:30:32.638357+0000","last_active":"2023-10-06T03:30:32.615612+0000","last_peered":"2023-10-06T03:30:32.615612+0000","last_clean":"2023-10-06T03:30:32.615612+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-06T03:30:32.615612+0000","last_undegraded":"2023-10-06T03:30:32.615612+0000","last_fullsized":"2023-10-06T03:30:32.615612+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:30:32.615612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:30:32.615612+0000","last_clean_scrub_stamp":"2023-10-06T03:30:32.615612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T10:35:08.797798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-06T03:30:33.627150+0000","last_change":"2023-10-06T03:30:33.627150+0000","last_active":"2023-10-06T03:30:33.627150+0000","last_peered":"2023-10-06T03:30:33.627150+0000","last_clean":"2023-10-06T03:30:33.627150+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-06T03:30:33.627150+0000","last_undegraded":"2023-10-06T03:30:33.627150+0000","last_fullsized":"2023-10-06T03:30:33.627150+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:30:33.627150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:30:33.627150+0000","last_clean_scrub_stamp":"2023-10-06T03:30:33.627150+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-06T03:30:32.615612+0000","last_change":"2023-10-06T03:30:32.615612+0000","last_active":"2023-10-06T03:30:32.615612+0000","last_peered":"2023-10-06T03:30:32.615612+0000","last_clean":"2023-10-06T03:30:32.615612+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-06T03:30:32.615612+0000","last_undegraded":"2023-10-06T03:30:32.615612+0000","last_fullsized":"2023-10-06T03:30:32.615612+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:30:32.615612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:30:32.615612+0000","last_clean_scrub_stamp":"2023-10-06T03:30:32.615612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-06T03:30:33.627150+0000","last_change":"2023-10-06T03:30:33.627150+0000","last_active":"2023-10-06T03:30:33.627150+0000","last_peered":"2023-10-06T03:30:33.627150+0000","last_clean":"2023-10-06T03:30:33.627150+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-06T03:30:33.627150+0000","last_undegraded":"2023-10-06T03:30:33.627150+0000","last_fullsized":"2023-10-06T03:30:33.627150+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:30:33.627150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:30:33.627150+0000","last_clean_scrub_stamp":"2023-10-06T03:30:33.627150+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-10-06T03:30:32.615612+0000","last_change":"2023-10-06T03:30:32.641004+0000","last_active":"2023-10-06T03:30:32.615612+0000","last_peered":"2023-10-06T03:30:32.615612+0000","last_clean":"2023-10-06T03:30:32.615612+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-06T03:30:32.615612+0000","last_undegraded":"2023-10-06T03:30:32.615612+0000","last_fullsized":"2023-10-06T03:30:32.615612+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:30:32.615612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:30:32.615612+0000","last_clean_scrub_stamp":"2023-10-06T03:30:32.615612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T05:43:04.324789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-06T03:30:33.627150+0000","last_change":"2023-10-06T03:30:33.627150+0000","last_active":"2023-10-06T03:30:33.627150+0000","last_peered":"2023-10-06T03:30:33.627150+0000","last_clean":"2023-10-06T03:30:33.627150+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-06T03:30:33.627150+0000","last_undegraded":"2023-10-06T03:30:33.627150+0000","last_fullsized":"2023-10-06T03:30:33.627150+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:30:33.627150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:30:33.627150+0000","last_clean_scrub_stamp":"2023-10-06T03:30:33.627150+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-06T03:30:32.615612+0000","last_change":"2023-10-06T03:30:32.615612+0000","last_active":"2023-10-06T03:30:32.615612+0000","last_peered":"2023-10-06T03:30:32.615612+0000","last_clean":"2023-10-06T03:30:32.615612+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-06T03:30:32.615612+0000","last_undegraded":"2023-10-06T03:30:32.615612+0000","last_fullsized":"2023-10-06T03:30:32.615612+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:30:32.615612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:30:32.615612+0000","last_clean_scrub_stamp":"2023-10-06T03:30:32.615612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.14","version":"0'0","reported_seq":9,"reported_epoch":41,"state":"active+clean","last_fresh":"2023-10-06T03:30:34.667526+0000","last_change":"2023-10-06T03:30:34.664096+0000","last_active":"2023-10-06T03:30:34.667526+0000","last_peered":"2023-10-06T03:30:34.667526+0000","last_clean":"2023-10-06T03:30:34.667526+0000","last_became_active":"2023-10-06T03:30:34.663768+0000","last_became_peered":"2023-10-06T03:30:34.663768+0000","last_unstale":"2023-10-06T03:30:34.667526+0000","last_undegraded":"2023-10-06T03:30:34.667526+0000","last_fullsized":"2023-10-06T03:30:34.667526+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:30:33.627150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:30:33.627150+0000","last_clean_scrub_stamp":"2023-10-06T03:30:33.627150+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T04:59:46.261266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-06T03:30:32.615612+0000","last_change":"2023-10-06T03:30:32.615612+0000","last_active":"2023-10-06T03:30:32.615612+0000","last_peered":"2023-10-06T03:30:32.615612+0000","last_clean":"2023-10-06T03:30:32.615612+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-06T03:30:32.615612+0000","last_undegraded":"2023-10-06T03:30:32.615612+0000","last_fullsized":"2023-10-06T03:30:32.615612+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:30:32.615612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:30:32.615612+0000","last_clean_scrub_stamp":"2023-10-06T03:30:32.615612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-06T03:30:33.627150+0000","last_change":"2023-10-06T03:30:33.627150+0000","last_active":"2023-10-06T03:30:33.627150+0000","last_peered":"2023-10-06T03:30:33.627150+0000","last_clean":"2023-10-06T03:30:33.627150+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-06T03:30:33.627150+0000","last_undegraded":"2023-10-06T03:30:33.627150+0000","last_fullsized":"2023-10-06T03:30:33.627150+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:30:33.627150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:30:33.627150+0000","last_clean_scrub_stamp":"2023-10-06T03:30:33.627150+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.14","version":"41'6","reported_seq":15,"reported_epoch":41,"state":"active+clean","last_fresh":"2023-10-06T03:30:34.687563+0000","last_change":"2023-10-06T03:30:33.668573+0000","last_active":"2023-10-06T03:30:34.687563+0000","last_peered":"2023-10-06T03:30:34.687563+0000","last_clean":"2023-10-06T03:30:34.687563+0000","last_became_active":"2023-10-06T03:30:33.668238+0000","last_became_peered":"2023-10-06T03:30:33.668238+0000","last_unstale":"2023-10-06T03:30:34.687563+0000","last_undegraded":"2023-10-06T03:30:34.687563+0000","last_fullsized":"2023-10-06T03:30:34.687563+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:30:32.615612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:30:32.615612+0000","last_clean_scrub_stamp":"2023-10-06T03:30:32.615612+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T14:37:18.115456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1536,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-06T03:30:33.627150+0000","last_change":"2023-10-06T03:30:33.627150+0000","last_active":"2023-10-06T03:30:33.627150+0000","last_peered":"2023-10-06T03:30:33.627150+0000","last_clean":"2023-10-06T03:30:33.627150+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-06T03:30:33.627150+0000","last_undegraded":"2023-10-06T03:30:33.627150+0000","last_fullsized":"2023-10-06T03:30:33.627150+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:30:33.627150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:30:33.627150+0000","last_clean_scrub_stamp":"2023-10-06T03:30:33.627150+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.13","version":"41'2","reported_seq":12,"reported_epoch":41,"state":"active+clean","last_fresh":"2023-10-06T03:30:34.692597+0000","last_change":"2023-10-06T03:30:33.664807+0000","last_active":"2023-10-06T03:30:34.692597+0000","last_peered":"2023-10-06T03:30:34.692597+0000","last_clean":"2023-10-06T03:30:34.692597+0000","last_became_active":"2023-10-06T03:30:33.664547+0000","last_became_peered":"2023-10-06T03:30:33.664547+0000","last_unstale":"2023-10-06T03:30:34.692597+0000","last_undegraded":"2023-10-06T03:30:34.692597+0000","last_fullsized":"2023-10-06T03:30:34.692597+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:30:32.615612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:30:32.615612+0000","last_clean_scrub_stamp":"2023-10-06T03:30:32.615612+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T12:12:51.259303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-06T03:30:33.627150+0000","last_change":"2023-10-06T03:30:33.627150+0000","last_active":"2023-10-06T03:30:33.627150+0000","last_peered":"2023-10-06T03:30:33.627150+0000","last_clean":"2023-10-06T03:30:33.627150+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-06T03:30:33.627150+0000","last_undegraded":"2023-10-06T03:30:33.627150+0000","last_fullsized":"2023-10-06T03:30:33.627150+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:30:33.627150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:30:33.627150+0000","last_clean_scrub_stamp":"2023-10-06T03:30:33.627150+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-06T03:30:32.615612+0000","last_change":"2023-10-06T03:30:32.615612+0000","last_active":"2023-10-06T03:30:32.615612+0000","last_peered":"2023-10-06T03:30:32.615612+0000","last_clean":"2023-10-06T03:30:32.615612+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-06T03:30:32.615612+0000","last_undegraded":"2023-10-06T03:30:32.615612+0000","last_fullsized":"2023-10-06T03:30:32.615612+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:30:32.615612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:30:32.615612+0000","last_clean_scrub_stamp":"2023-10-06T03:30:32.615612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.10","version":"0'0","reported_seq":9,"reported_epoch":41,"state":"active+clean","last_fresh":"2023-10-06T03:30:34.667620+0000","last_change":"2023-10-06T03:30:34.664448+0000","last_active":"2023-10-06T03:30:34.667620+0000","last_peered":"2023-10-06T03:30:34.667620+0000","last_clean":"2023-10-06T03:30:34.667620+0000","last_became_active":"2023-10-06T03:30:34.664132+0000","last_became_peered":"2023-10-06T03:30:34.664132+0000","last_unstale":"2023-10-06T03:30:34.667620+0000","last_undegraded":"2023-10-06T03:30:34.667620+0000","last_fullsized":"2023-10-06T03:30:34.667620+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:30:33.627150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:30:33.627150+0000","last_clean_scrub_stamp":"2023-10-06T03:30:33.627150+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T08:18:03.563387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":10,"reported_epoch":41,"state":"active+clean","last_fresh":"2023-10-06T03:30:34.651348+0000","last_change":"2023-10-06T03:30:33.662513+0000","last_active":"2023-10-06T03:30:34.651348+0000","last_peered":"2023-10-06T03:30:34.651348+0000","last_clean":"2023-10-06T03:30:34.651348+0000","last_became_active":"2023-10-06T03:30:33.662244+0000","last_became_peered":"2023-10-06T03:30:33.662244+0000","last_unstale":"2023-10-06T03:30:34.651348+0000","last_undegraded":"2023-10-06T03:30:34.651348+0000","last_fullsized":"2023-10-06T03:30:34.651348+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:30:32.615612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:30:32.615612+0000","last_clean_scrub_stamp":"2023-10-06T03:30:32.615612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T15:05:13.223269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-06T03:30:33.627150+0000","last_change":"2023-10-06T03:30:33.627150+0000","last_active":"2023-10-06T03:30:33.627150+0000","last_peered":"2023-10-06T03:30:33.627150+0000","last_clean":"2023-10-06T03:30:33.627150+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-06T03:30:33.627150+0000","last_undegraded":"2023-10-06T03:30:33.627150+0000","last_fullsized":"2023-10-06T03:30:33.627150+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:30:33.627150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:30:33.627150+0000","last_clean_scrub_stamp":"2023-10-06T03:30:33.627150+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-06T03:30:32.615612+0000","last_change":"2023-10-06T03:30:32.615612+0000","last_active":"2023-10-06T03:30:32.615612+0000","last_peered":"2023-10-06T03:30:32.615612+0000","last_clean":"2023-10-06T03:30:32.615612+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-06T03:30:32.615612+0000","last_undegraded":"2023-10-06T03:30:32.615612+0000","last_fullsized":"2023-10-06T03:30:32.615612+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:30:32.615612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:30:32.615612+0000","last_clean_scrub_stamp":"2023-10-06T03:30:32.615612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.e","version":"0'0","reported_seq":8,"reported_epoch":41,"state":"active+clean","last_fresh":"2023-10-06T03:30:34.666880+0000","last_change":"2023-10-06T03:30:34.666880+0000","last_active":"2023-10-06T03:30:34.666880+0000","last_peered":"2023-10-06T03:30:34.666880+0000","last_clean":"2023-10-06T03:30:34.666880+0000","last_became_active":"2023-10-06T03:30:34.666582+0000","last_became_peered":"2023-10-06T03:30:34.666582+0000","last_unstale":"2023-10-06T03:30:34.666880+0000","last_undegraded":"2023-10-06T03:30:34.666880+0000","last_fullsized":"2023-10-06T03:30:34.666880+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:30:33.627150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:30:33.627150+0000","last_clean_scrub_stamp":"2023-10-06T03:30:33.627150+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T11:23:03.827062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"41'1","reported_seq":10,"reported_epoch":41,"state":"active+clean","last_fresh":"2023-10-06T03:30:34.657064+0000","last_change":"2023-10-06T03:30:33.671975+0000","last_active":"2023-10-06T03:30:34.657064+0000","last_peered":"2023-10-06T03:30:34.657064+0000","last_clean":"2023-10-06T03:30:34.657064+0000","last_became_active":"2023-10-06T03:30:33.671652+0000","last_became_peered":"2023-10-06T03:30:33.671652+0000","last_unstale":"2023-10-06T03:30:34.657064+0000","last_undegraded":"2023-10-06T03:30:34.657064+0000","last_fullsized":"2023-10-06T03:30:34.657064+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:30:32.615612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:30:32.615612+0000","last_clean_scrub_stamp":"2023-10-06T03:30:32.615612+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T13:10:54.937694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-06T03:30:33.627150+0000","last_change":"2023-10-06T03:30:33.627150+0000","last_active":"2023-10-06T03:30:33.627150+0000","last_peered":"2023-10-06T03:30:33.627150+0000","last_clean":"2023-10-06T03:30:33.627150+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-06T03:30:33.627150+0000","last_undegraded":"2023-10-06T03:30:33.627150+0000","last_fullsized":"2023-10-06T03:30:33.627150+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:30:33.627150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:30:33.627150+0000","last_clean_scrub_stamp":"2023-10-06T03:30:33.627150+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-10-06T03:30:32.615612+0000","last_change":"2023-10-06T03:30:32.641558+0000","last_active":"2023-10-06T03:30:32.615612+0000","last_peered":"2023-10-06T03:30:32.615612+0000","last_clean":"2023-10-06T03:30:32.615612+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-06T03:30:32.615612+0000","last_undegraded":"2023-10-06T03:30:32.615612+0000","last_fullsized":"2023-10-06T03:30:32.615612+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:30:32.615612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:30:32.615612+0000","last_clean_scrub_stamp":"2023-10-06T03:30:32.615612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T14:57:48.864706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-06T03:30:33.627150+0000","last_change":"2023-10-06T03:30:33.627150+0000","last_active":"2023-10-06T03:30:33.627150+0000","last_peered":"2023-10-06T03:30:33.627150+0000","last_clean":"2023-10-06T03:30:33.627150+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-06T03:30:33.627150+0000","last_undegraded":"2023-10-06T03:30:33.627150+0000","last_fullsized":"2023-10-06T03:30:33.627150+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:30:33.627150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:30:33.627150+0000","last_clean_scrub_stamp":"2023-10-06T03:30:33.627150+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-10-06T03:30:32.615612+0000","last_change":"2023-10-06T03:30:32.639745+0000","last_active":"2023-10-06T03:30:32.615612+0000","last_peered":"2023-10-06T03:30:32.615612+0000","last_clean":"2023-10-06T03:30:32.615612+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-06T03:30:32.615612+0000","last_undegraded":"2023-10-06T03:30:32.615612+0000","last_fullsized":"2023-10-06T03:30:32.615612+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:30:32.615612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:30:32.615612+0000","last_clean_scrub_stamp":"2023-10-06T03:30:32.615612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T11:23:02.815524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-06T03:30:33.627150+0000","last_change":"2023-10-06T03:30:33.627150+0000","last_active":"2023-10-06T03:30:33.627150+0000","last_peered":"2023-10-06T03:30:33.627150+0000","last_clean":"2023-10-06T03:30:33.627150+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-06T03:30:33.627150+0000","last_undegraded":"2023-10-06T03:30:33.627150+0000","last_fullsized":"2023-10-06T03:30:33.627150+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:30:33.627150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:30:33.627150+0000","last_clean_scrub_stamp":"2023-10-06T03:30:33.627150+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":10,"reported_epoch":41,"state":"active+clean","last_fresh":"2023-10-06T03:30:34.652110+0000","last_change":"2023-10-06T03:30:33.660529+0000","last_active":"2023-10-06T03:30:34.652110+0000","last_peered":"2023-10-06T03:30:34.652110+0000","last_clean":"2023-10-06T03:30:34.652110+0000","last_became_active":"2023-10-06T03:30:33.660205+0000","last_became_peered":"2023-10-06T03:30:33.660205+0000","last_unstale":"2023-10-06T03:30:34.652110+0000","last_undegraded":"2023-10-06T03:30:34.652110+0000","last_fullsized":"2023-10-06T03:30:34.652110+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:30:32.615612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:30:32.615612+0000","last_clean_scrub_stamp":"2023-10-06T03:30:32.615612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T08:43:57.890425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":9,"reported_epoch":41,"state":"active+clean","last_fresh":"2023-10-06T03:30:34.668289+0000","last_change":"2023-10-06T03:30:34.665508+0000","last_active":"2023-10-06T03:30:34.668289+0000","last_peered":"2023-10-06T03:30:34.668289+0000","last_clean":"2023-10-06T03:30:34.668289+0000","last_became_active":"2023-10-06T03:30:34.664787+0000","last_became_peered":"2023-10-06T03:30:34.664787+0000","last_unstale":"2023-10-06T03:30:34.668289+0000","last_undegraded":"2023-10-06T03:30:34.668289+0000","last_fullsized":"2023-10-06T03:30:34.668289+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:30:33.627150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:30:33.627150+0000","last_clean_scrub_stamp":"2023-10-06T03:30:33.627150+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T15:11:06.944301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.b","version":"41'2","reported_seq":12,"reported_epoch":41,"state":"active+clean","last_fresh":"2023-10-06T03:30:34.687133+0000","last_change":"2023-10-06T03:30:33.667463+0000","last_active":"2023-10-06T03:30:34.687133+0000","last_peered":"2023-10-06T03:30:34.687133+0000","last_clean":"2023-10-06T03:30:34.687133+0000","last_became_active":"2023-10-06T03:30:33.665107+0000","last_became_peered":"2023-10-06T03:30:33.665107+0000","last_unstale":"2023-10-06T03:30:34.687133+0000","last_undegraded":"2023-10-06T03:30:34.687133+0000","last_fullsized":"2023-10-06T03:30:34.687133+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:30:32.615612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:30:32.615612+0000","last_clean_scrub_stamp":"2023-10-06T03:30:32.615612+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T07:30:17.435201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"0'0","reported_seq":8,"reported_epoch":41,"state":"active+clean","last_fresh":"2023-10-06T03:30:34.666719+0000","last_change":"2023-10-06T03:30:34.666719+0000","last_active":"2023-10-06T03:30:34.666719+0000","last_peered":"2023-10-06T03:30:34.666719+0000","last_clean":"2023-10-06T03:30:34.666719+0000","last_became_active":"2023-10-06T03:30:34.665873+0000","last_became_peered":"2023-10-06T03:30:34.665873+0000","last_unstale":"2023-10-06T03:30:34.666719+0000","last_undegraded":"2023-10-06T03:30:34.666719+0000","last_fullsized":"2023-10-06T03:30:34.666719+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:30:33.627150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:30:33.627150+0000","last_clean_scrub_stamp":"2023-10-06T03:30:33.627150+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T03:45:43.730583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-10-06T03:30:32.615612+0000","last_change":"2023-10-06T03:30:32.638733+0000","last_active":"2023-10-06T03:30:32.615612+0000","last_peered":"2023-10-06T03:30:32.615612+0000","last_clean":"2023-10-06T03:30:32.615612+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-06T03:30:32.615612+0000","last_undegraded":"2023-10-06T03:30:32.615612+0000","last_fullsized":"2023-10-06T03:30:32.615612+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:30:32.615612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:30:32.615612+0000","last_clean_scrub_stamp":"2023-10-06T03:30:32.615612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T07:39:22.429891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[6],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":8,"reported_epoch":41,"state":"active+clean","last_fresh":"2023-10-06T03:30:34.668145+0000","last_change":"2023-10-06T03:30:34.668145+0000","last_active":"2023-10-06T03:30:34.668145+0000","last_peered":"2023-10-06T03:30:34.668145+0000","last_clean":"2023-10-06T03:30:34.668145+0000","last_became_active":"2023-10-06T03:30:34.667870+0000","last_became_peered":"2023-10-06T03:30:34.667870+0000","last_unstale":"2023-10-06T03:30:34.668145+0000","last_undegraded":"2023-10-06T03:30:34.668145+0000","last_fullsized":"2023-10-06T03:30:34.668145+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:30:33.627150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:30:33.627150+0000","last_clean_scrub_stamp":"2023-10-06T03:30:33.627150+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T05:05:42.527147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-06T03:30:32.615612+0000","last_change":"2023-10-06T03:30:32.615612+0000","last_active":"2023-10-06T03:30:32.615612+0000","last_peered":"2023-10-06T03:30:32.615612+0000","last_clean":"2023-10-06T03:30:32.615612+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-06T03:30:32.615612+0000","last_undegraded":"2023-10-06T03:30:32.615612+0000","last_fullsized":"2023-10-06T03:30:32.615612+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:30:32.615612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:30:32.615612+0000","last_clean_scrub_stamp":"2023-10-06T03:30:32.615612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-06T03:30:33.627150+0000","last_change":"2023-10-06T03:30:33.627150+0000","last_active":"2023-10-06T03:30:33.627150+0000","last_peered":"2023-10-06T03:30:33.627150+0000","last_clean":"2023-10-06T03:30:33.627150+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-06T03:30:33.627150+0000","last_undegraded":"2023-10-06T03:30:33.627150+0000","last_fullsized":"2023-10-06T03:30:33.627150+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:30:33.627150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:30:33.627150+0000","last_clean_scrub_stamp":"2023-10-06T03:30:33.627150+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-06T03:30:32.615612+0000","last_change":"2023-10-06T03:30:32.615612+0000","last_active":"2023-10-06T03:30:32.615612+0000","last_peered":"2023-10-06T03:30:32.615612+0000","last_clean":"2023-10-06T03:30:32.615612+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-06T03:30:32.615612+0000","last_undegraded":"2023-10-06T03:30:32.615612+0000","last_fullsized":"2023-10-06T03:30:32.615612+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:30:32.615612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:30:32.615612+0000","last_clean_scrub_stamp":"2023-10-06T03:30:32.615612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":9,"reported_epoch":41,"state":"active+clean","last_fresh":"2023-10-06T03:30:34.667228+0000","last_change":"2023-10-06T03:30:34.662684+0000","last_active":"2023-10-06T03:30:34.667228+0000","last_peered":"2023-10-06T03:30:34.667228+0000","last_clean":"2023-10-06T03:30:34.667228+0000","last_became_active":"2023-10-06T03:30:34.662365+0000","last_became_peered":"2023-10-06T03:30:34.662365+0000","last_unstale":"2023-10-06T03:30:34.667228+0000","last_undegraded":"2023-10-06T03:30:34.667228+0000","last_fullsized":"2023-10-06T03:30:34.667228+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:30:33.627150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:30:33.627150+0000","last_clean_scrub_stamp":"2023-10-06T03:30:33.627150+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T04:33:34.388983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-06T03:30:32.615612+0000","last_change":"2023-10-06T03:30:32.615612+0000","last_active":"2023-10-06T03:30:32.615612+0000","last_peered":"2023-10-06T03:30:32.615612+0000","last_clean":"2023-10-06T03:30:32.615612+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-06T03:30:32.615612+0000","last_undegraded":"2023-10-06T03:30:32.615612+0000","last_fullsized":"2023-10-06T03:30:32.615612+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:30:32.615612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:30:32.615612+0000","last_clean_scrub_stamp":"2023-10-06T03:30:32.615612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":24,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-06T03:30:29.360432+0000","last_change":"2023-10-06T03:29:29.387402+0000","last_active":"2023-10-06T03:30:29.360432+0000","last_peered":"2023-10-06T03:30:29.360432+0000","last_clean":"2023-10-06T03:30:29.360432+0000","last_became_active":"2023-10-06T03:29:29.386812+0000","last_became_peered":"2023-10-06T03:29:29.386812+0000","last_unstale":"2023-10-06T03:30:29.360432+0000","last_undegraded":"2023-10-06T03:30:29.360432+0000","last_fullsized":"2023-10-06T03:30:29.360432+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:28.357506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:28.357506+0000","last_clean_scrub_stamp":"2023-10-06T03:29:28.357506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T12:09:43.895810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":9,"reported_epoch":41,"state":"active+clean","last_fresh":"2023-10-06T03:30:34.667592+0000","last_change":"2023-10-06T03:30:34.664854+0000","last_active":"2023-10-06T03:30:34.667592+0000","last_peered":"2023-10-06T03:30:34.667592+0000","last_clean":"2023-10-06T03:30:34.667592+0000","last_became_active":"2023-10-06T03:30:34.664466+0000","last_became_peered":"2023-10-06T03:30:34.664466+0000","last_unstale":"2023-10-06T03:30:34.667592+0000","last_undegraded":"2023-10-06T03:30:34.667592+0000","last_fullsized":"2023-10-06T03:30:34.667592+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:30:33.627150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:30:33.627150+0000","last_clean_scrub_stamp":"2023-10-06T03:30:33.627150+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T09:21:33.343368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-10-06T03:30:32.615612+0000","last_change":"2023-10-06T03:30:32.640766+0000","last_active":"2023-10-06T03:30:32.615612+0000","last_peered":"2023-10-06T03:30:32.615612+0000","last_clean":"2023-10-06T03:30:32.615612+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-06T03:30:32.615612+0000","last_undegraded":"2023-10-06T03:30:32.615612+0000","last_fullsized":"2023-10-06T03:30:32.615612+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:30:32.615612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:30:32.615612+0000","last_clean_scrub_stamp":"2023-10-06T03:30:32.615612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T08:26:45.574540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":27,"reported_epoch":40,"state":"active+clean","last_fresh":"2023-10-06T03:30:33.658461+0000","last_change":"2023-10-06T03:29:29.386238+0000","last_active":"2023-10-06T03:30:33.658461+0000","last_peered":"2023-10-06T03:30:33.658461+0000","last_clean":"2023-10-06T03:30:33.658461+0000","last_became_active":"2023-10-06T03:29:29.385865+0000","last_became_peered":"2023-10-06T03:29:29.385865+0000","last_unstale":"2023-10-06T03:30:33.658461+0000","last_undegraded":"2023-10-06T03:30:33.658461+0000","last_fullsized":"2023-10-06T03:30:33.658461+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:28.357506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:28.357506+0000","last_clean_scrub_stamp":"2023-10-06T03:29:28.357506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T12:43:34.714730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-06T03:30:33.627150+0000","last_change":"2023-10-06T03:30:33.627150+0000","last_active":"2023-10-06T03:30:33.627150+0000","last_peered":"2023-10-06T03:30:33.627150+0000","last_clean":"2023-10-06T03:30:33.627150+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-06T03:30:33.627150+0000","last_undegraded":"2023-10-06T03:30:33.627150+0000","last_fullsized":"2023-10-06T03:30:33.627150+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:30:33.627150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:30:33.627150+0000","last_clean_scrub_stamp":"2023-10-06T03:30:33.627150+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":9,"reported_epoch":40,"state":"active+clean","last_fresh":"2023-10-06T03:30:33.669068+0000","last_change":"2023-10-06T03:30:33.664832+0000","last_active":"2023-10-06T03:30:33.669068+0000","last_peered":"2023-10-06T03:30:33.669068+0000","last_clean":"2023-10-06T03:30:33.669068+0000","last_became_active":"2023-10-06T03:30:33.664537+0000","last_became_peered":"2023-10-06T03:30:33.664537+0000","last_unstale":"2023-10-06T03:30:33.669068+0000","last_undegraded":"2023-10-06T03:30:33.669068+0000","last_fullsized":"2023-10-06T03:30:33.669068+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:30:32.615612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:30:32.615612+0000","last_clean_scrub_stamp":"2023-10-06T03:30:32.615612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T13:44:37.657361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":24,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-10-06T03:30:27.917792+0000","last_change":"2023-10-06T03:29:29.384279+0000","last_active":"2023-10-06T03:30:27.917792+0000","last_peered":"2023-10-06T03:30:27.917792+0000","last_clean":"2023-10-06T03:30:27.917792+0000","last_became_active":"2023-10-06T03:29:29.383533+0000","last_became_peered":"2023-10-06T03:29:29.383533+0000","last_unstale":"2023-10-06T03:30:27.917792+0000","last_undegraded":"2023-10-06T03:30:27.917792+0000","last_fullsized":"2023-10-06T03:30:27.917792+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:28.357506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:28.357506+0000","last_clean_scrub_stamp":"2023-10-06T03:29:28.357506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T08:34:19.641525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-06T03:30:33.627150+0000","last_change":"2023-10-06T03:30:33.627150+0000","last_active":"2023-10-06T03:30:33.627150+0000","last_peered":"2023-10-06T03:30:33.627150+0000","last_clean":"2023-10-06T03:30:33.627150+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-06T03:30:33.627150+0000","last_undegraded":"2023-10-06T03:30:33.627150+0000","last_fullsized":"2023-10-06T03:30:33.627150+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:30:33.627150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:30:33.627150+0000","last_clean_scrub_stamp":"2023-10-06T03:30:33.627150+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-06T03:30:32.615612+0000","last_change":"2023-10-06T03:30:32.615612+0000","last_active":"2023-10-06T03:30:32.615612+0000","last_peered":"2023-10-06T03:30:32.615612+0000","last_clean":"2023-10-06T03:30:32.615612+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-06T03:30:32.615612+0000","last_undegraded":"2023-10-06T03:30:32.615612+0000","last_fullsized":"2023-10-06T03:30:32.615612+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:30:32.615612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:30:32.615612+0000","last_clean_scrub_stamp":"2023-10-06T03:30:32.615612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":27,"reported_epoch":40,"state":"active+clean","last_fresh":"2023-10-06T03:30:33.657268+0000","last_change":"2023-10-06T03:29:29.382183+0000","last_active":"2023-10-06T03:30:33.657268+0000","last_peered":"2023-10-06T03:30:33.657268+0000","last_clean":"2023-10-06T03:30:33.657268+0000","last_became_active":"2023-10-06T03:29:29.381918+0000","last_became_peered":"2023-10-06T03:29:29.381918+0000","last_unstale":"2023-10-06T03:30:33.657268+0000","last_undegraded":"2023-10-06T03:30:33.657268+0000","last_fullsized":"2023-10-06T03:30:33.657268+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:28.357506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:28.357506+0000","last_clean_scrub_stamp":"2023-10-06T03:29:28.357506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T11:56:26.013716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-06T03:30:33.627150+0000","last_change":"2023-10-06T03:30:33.627150+0000","last_active":"2023-10-06T03:30:33.627150+0000","last_peered":"2023-10-06T03:30:33.627150+0000","last_clean":"2023-10-06T03:30:33.627150+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-06T03:30:33.627150+0000","last_undegraded":"2023-10-06T03:30:33.627150+0000","last_fullsized":"2023-10-06T03:30:33.627150+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:30:33.627150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:30:33.627150+0000","last_clean_scrub_stamp":"2023-10-06T03:30:33.627150+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":10,"reported_epoch":41,"state":"active+clean","last_fresh":"2023-10-06T03:30:34.650414+0000","last_change":"2023-10-06T03:30:33.660915+0000","last_active":"2023-10-06T03:30:34.650414+0000","last_peered":"2023-10-06T03:30:34.650414+0000","last_clean":"2023-10-06T03:30:34.650414+0000","last_became_active":"2023-10-06T03:30:33.660639+0000","last_became_peered":"2023-10-06T03:30:33.660639+0000","last_unstale":"2023-10-06T03:30:34.650414+0000","last_undegraded":"2023-10-06T03:30:34.650414+0000","last_fullsized":"2023-10-06T03:30:34.650414+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:30:32.615612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:30:32.615612+0000","last_clean_scrub_stamp":"2023-10-06T03:30:32.615612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T09:13:46.237757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":25,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-06T03:30:27.918207+0000","last_change":"2023-10-06T03:29:29.384415+0000","last_active":"2023-10-06T03:30:27.918207+0000","last_peered":"2023-10-06T03:30:27.918207+0000","last_clean":"2023-10-06T03:30:27.918207+0000","last_became_active":"2023-10-06T03:29:29.383894+0000","last_became_peered":"2023-10-06T03:29:29.383894+0000","last_unstale":"2023-10-06T03:30:27.918207+0000","last_undegraded":"2023-10-06T03:30:27.918207+0000","last_fullsized":"2023-10-06T03:30:27.918207+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:28.357506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:28.357506+0000","last_clean_scrub_stamp":"2023-10-06T03:29:28.357506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T13:09:00.657107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-06T03:30:33.627150+0000","last_change":"2023-10-06T03:30:33.627150+0000","last_active":"2023-10-06T03:30:33.627150+0000","last_peered":"2023-10-06T03:30:33.627150+0000","last_clean":"2023-10-06T03:30:33.627150+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-06T03:30:33.627150+0000","last_undegraded":"2023-10-06T03:30:33.627150+0000","last_fullsized":"2023-10-06T03:30:33.627150+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:30:33.627150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:30:33.627150+0000","last_clean_scrub_stamp":"2023-10-06T03:30:33.627150+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-06T03:30:32.615612+0000","last_change":"2023-10-06T03:30:32.615612+0000","last_active":"2023-10-06T03:30:32.615612+0000","last_peered":"2023-10-06T03:30:32.615612+0000","last_clean":"2023-10-06T03:30:32.615612+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-06T03:30:32.615612+0000","last_undegraded":"2023-10-06T03:30:32.615612+0000","last_fullsized":"2023-10-06T03:30:32.615612+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:30:32.615612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:30:32.615612+0000","last_clean_scrub_stamp":"2023-10-06T03:30:32.615612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":28,"reported_epoch":40,"state":"active+clean","last_fresh":"2023-10-06T03:30:33.652979+0000","last_change":"2023-10-06T03:29:29.606195+0000","last_active":"2023-10-06T03:30:33.652979+0000","last_peered":"2023-10-06T03:30:33.652979+0000","last_clean":"2023-10-06T03:30:33.652979+0000","last_became_active":"2023-10-06T03:29:29.605489+0000","last_became_peered":"2023-10-06T03:29:29.605489+0000","last_unstale":"2023-10-06T03:30:33.652979+0000","last_undegraded":"2023-10-06T03:30:33.652979+0000","last_fullsized":"2023-10-06T03:30:33.652979+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:28.357506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:28.357506+0000","last_clean_scrub_stamp":"2023-10-06T03:29:28.357506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T10:21:35.864895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-06T03:30:33.627150+0000","last_change":"2023-10-06T03:30:33.627150+0000","last_active":"2023-10-06T03:30:33.627150+0000","last_peered":"2023-10-06T03:30:33.627150+0000","last_clean":"2023-10-06T03:30:33.627150+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-06T03:30:33.627150+0000","last_undegraded":"2023-10-06T03:30:33.627150+0000","last_fullsized":"2023-10-06T03:30:33.627150+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:30:33.627150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:30:33.627150+0000","last_clean_scrub_stamp":"2023-10-06T03:30:33.627150+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":9,"reported_epoch":40,"state":"active+clean","last_fresh":"2023-10-06T03:30:33.671676+0000","last_change":"2023-10-06T03:30:33.670575+0000","last_active":"2023-10-06T03:30:33.671676+0000","last_peered":"2023-10-06T03:30:33.671676+0000","last_clean":"2023-10-06T03:30:33.671676+0000","last_became_active":"2023-10-06T03:30:33.670190+0000","last_became_peered":"2023-10-06T03:30:33.670190+0000","last_unstale":"2023-10-06T03:30:33.671676+0000","last_undegraded":"2023-10-06T03:30:33.671676+0000","last_fullsized":"2023-10-06T03:30:33.671676+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:30:32.615612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:30:32.615612+0000","last_clean_scrub_stamp":"2023-10-06T03:30:32.615612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T11:15:18.455519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":24,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-10-06T03:30:27.917791+0000","last_change":"2023-10-06T03:29:29.384353+0000","last_active":"2023-10-06T03:30:27.917791+0000","last_peered":"2023-10-06T03:30:27.917791+0000","last_clean":"2023-10-06T03:30:27.917791+0000","last_became_active":"2023-10-06T03:29:29.383884+0000","last_became_peered":"2023-10-06T03:29:29.383884+0000","last_unstale":"2023-10-06T03:30:27.917791+0000","last_undegraded":"2023-10-06T03:30:27.917791+0000","last_fullsized":"2023-10-06T03:30:27.917791+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:28.357506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:28.357506+0000","last_clean_scrub_stamp":"2023-10-06T03:29:28.357506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T14:43:09.107140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-06T03:30:33.627150+0000","last_change":"2023-10-06T03:30:33.627150+0000","last_active":"2023-10-06T03:30:33.627150+0000","last_peered":"2023-10-06T03:30:33.627150+0000","last_clean":"2023-10-06T03:30:33.627150+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-06T03:30:33.627150+0000","last_undegraded":"2023-10-06T03:30:33.627150+0000","last_fullsized":"2023-10-06T03:30:33.627150+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:30:33.627150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:30:33.627150+0000","last_clean_scrub_stamp":"2023-10-06T03:30:33.627150+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2023-10-06T03:30:32.615612+0000","last_change":"2023-10-06T03:30:32.639782+0000","last_active":"2023-10-06T03:30:32.615612+0000","last_peered":"2023-10-06T03:30:32.615612+0000","last_clean":"2023-10-06T03:30:32.615612+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-06T03:30:32.615612+0000","last_undegraded":"2023-10-06T03:30:32.615612+0000","last_fullsized":"2023-10-06T03:30:32.615612+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:30:32.615612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:30:32.615612+0000","last_clean_scrub_stamp":"2023-10-06T03:30:32.615612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T10:53:06.172743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":25,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-06T03:30:27.918204+0000","last_change":"2023-10-06T03:29:29.386120+0000","last_active":"2023-10-06T03:30:27.918204+0000","last_peered":"2023-10-06T03:30:27.918204+0000","last_clean":"2023-10-06T03:30:27.918204+0000","last_became_active":"2023-10-06T03:29:29.385692+0000","last_became_peered":"2023-10-06T03:29:29.385692+0000","last_unstale":"2023-10-06T03:30:27.918204+0000","last_undegraded":"2023-10-06T03:30:27.918204+0000","last_fullsized":"2023-10-06T03:30:27.918204+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:28.357506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:28.357506+0000","last_clean_scrub_stamp":"2023-10-06T03:29:28.357506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T10:21:35.864895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":177,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-10-06T03:30:27.917836+0000","last_change":"2023-10-06T03:29:21.978311+0000","last_active":"2023-10-06T03:30:27.917836+0000","last_peered":"2023-10-06T03:30:27.917836+0000","last_clean":"2023-10-06T03:30:27.917836+0000","last_became_active":"2023-10-06T03:29:21.978017+0000","last_became_peered":"2023-10-06T03:29:21.978017+0000","last_unstale":"2023-10-06T03:30:27.917836+0000","last_undegraded":"2023-10-06T03:30:27.917836+0000","last_fullsized":"2023-10-06T03:30:27.917836+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:20.957955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:20.957955+0000","last_clean_scrub_stamp":"2023-10-06T03:29:20.957955+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T07:33:10.429084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":36,"acting":36,"num_store_stats":3},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":2146,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":2238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":18,"ondisk_log_size":18,"up":42,"acting":42,"num_store_stats":5},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":6,"up_from":16,"seq":68719476755,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297704,"kb_used_data":416,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074136,"statfs":{"total":96636764160,"available":96331915264,"internally_reserved":0,"allocated":425984,"data_stored":184768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Oct 6 03:30:26 2023","interfaces":[{"interface":"back","average":{"1min":0.876,"5min":0.876,"15min":0.876},"min":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"max":{"1min":1.149,"5min":1.149,"15min":1.149},"last":0.94499999999999995},{"interface":"front","average":{"1min":0.90400000000000003,"5min":0.90400000000000003,"15min":0.90400000000000003},"min":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"max":{"1min":1.254,"5min":1.254,"15min":1.254},"last":1.141}]},{"osd":1,"last update":"Fri Oct 6 03:30:26 2023","interfaces":[{"interface":"back","average":{"1min":0.81299999999999994,"5min":0.81299999999999994,"15min":0.81299999999999994},"min":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"max":{"1min":1.4730000000000001,"5min":1.4730000000000001,"15min":1.4730000000000001},"last":0.71899999999999997},{"interface":"front","average":{"1min":0.97099999999999997,"5min":0.97099999999999997,"15min":0.97099999999999997},"min":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"max":{"1min":1.3440000000000001,"5min":1.3440000000000001,"15min":1.3440000000000001},"last":1.1160000000000001}]},{"osd":2,"last update":"Fri Oct 6 03:30:26 2023","interfaces":[{"interface":"back","average":{"1min":0.94099999999999995,"5min":0.94099999999999995,"15min":0.94099999999999995},"min":{"1min":0.629,"5min":0.629,"15min":0.629},"max":{"1min":1.236,"5min":1.236,"15min":1.236},"last":1.0740000000000001},{"interface":"front","average":{"1min":0.95599999999999996,"5min":0.95599999999999996,"15min":0.95599999999999996},"min":{"1min":0.623,"5min":0.623,"15min":0.623},"max":{"1min":1.3660000000000001,"5min":1.3660000000000001,"15min":1.3660000000000001},"last":1.0469999999999999}]},{"osd":3,"last update":"Fri Oct 6 03:30:26 2023","interfaces":[{"interface":"back","average":{"1min":0.99299999999999999,"5min":0.99299999999999999,"15min":0.99299999999999999},"min":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":1.3660000000000001,"5min":1.3660000000000001,"15min":1.3660000000000001},"last":0.84999999999999998},{"interface":"front","average":{"1min":0.95599999999999996,"5min":0.95599999999999996,"15min":0.95599999999999996},"min":{"1min":0.76300000000000001,"5min":0.76300000000000001,"15min":0.76300000000000001},"max":{"1min":1.2609999999999999,"5min":1.2609999999999999,"15min":1.2609999999999999},"last":0.89900000000000002}]},{"osd":4,"last update":"Fri Oct 6 03:30:26 2023","interfaces":[{"interface":"back","average":{"1min":1.1040000000000001,"5min":1.1040000000000001,"15min":1.1040000000000001},"min":{"1min":0.875,"5min":0.875,"15min":0.875},"max":{"1min":1.2869999999999999,"5min":1.2869999999999999,"15min":1.2869999999999999},"last":1.0149999999999999},{"interface":"front","average":{"1min":1.1140000000000001,"5min":1.1140000000000001,"15min":1.1140000000000001},"min":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.73399999999999999},"max":{"1min":1.4450000000000001,"5min":1.4450000000000001,"15min":1.4450000000000001},"last":0.97699999999999998}]},{"osd":5,"last update":"Fri Oct 6 03:30:26 2023","interfaces":[{"interface":"back","average":{"1min":1.105,"5min":1.105,"15min":1.105},"min":{"1min":0.86799999999999999,"5min":0.86799999999999999,"15min":0.86799999999999999},"max":{"1min":1.613,"5min":1.613,"15min":1.613},"last":0.82199999999999995},{"interface":"front","average":{"1min":1.1699999999999999,"5min":1.1699999999999999,"15min":1.1699999999999999},"min":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"max":{"1min":1.351,"5min":1.351,"15min":1.351},"last":1.1579999999999999}]},{"osd":7,"last update":"Fri Oct 6 03:30:26 2023","interfaces":[{"interface":"back","average":{"1min":1.2370000000000001,"5min":1.2370000000000001,"15min":1.2370000000000001},"min":{"1min":1.012,"5min":1.012,"15min":1.012},"max":{"1min":1.5409999999999999,"5min":1.5409999999999999,"15min":1.5409999999999999},"last":1.0900000000000001},{"interface":"front","average":{"1min":1.212,"5min":1.212,"15min":1.212},"min":{"1min":0.96099999999999997,"5min":0.96099999999999997,"15min":0.96099999999999997},"max":{"1min":1.704,"5min":1.704,"15min":1.704},"last":0.995}]}]},{"osd":5,"up_from":16,"seq":68719476755,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297716,"kb_used_data":428,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074124,"statfs":{"total":96636764160,"available":96331902976,"internally_reserved":0,"allocated":438272,"data_stored":191246,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Oct 6 03:30:23 2023","interfaces":[{"interface":"back","average":{"1min":0.86399999999999999,"5min":0.86399999999999999,"15min":0.86399999999999999},"min":{"1min":0.505,"5min":0.505,"15min":0.505},"max":{"1min":1.4079999999999999,"5min":1.4079999999999999,"15min":1.4079999999999999},"last":1.034},{"interface":"front","average":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":0.89800000000000002},"min":{"1min":0.64500000000000002,"5min":0.64500000000000002,"15min":0.64500000000000002},"max":{"1min":1.3779999999999999,"5min":1.3779999999999999,"15min":1.3779999999999999},"last":0.67600000000000005}]},{"osd":1,"last update":"Fri Oct 6 03:30:23 2023","interfaces":[{"interface":"back","average":{"1min":0.97799999999999998,"5min":0.97799999999999998,"15min":0.97799999999999998},"min":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":1.2829999999999999,"5min":1.2829999999999999,"15min":1.2829999999999999},"last":0.78100000000000003},{"interface":"front","average":{"1min":0.85999999999999999,"5min":0.85999999999999999,"15min":0.85999999999999999},"min":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"max":{"1min":1.581,"5min":1.581,"15min":1.581},"last":1.0680000000000001}]},{"osd":2,"last update":"Fri Oct 6 03:30:23 2023","interfaces":[{"interface":"back","average":{"1min":1,"5min":1,"15min":1},"min":{"1min":0.67300000000000004,"5min":0.67300000000000004,"15min":0.67300000000000004},"max":{"1min":1.258,"5min":1.258,"15min":1.258},"last":0.86099999999999999},{"interface":"front","average":{"1min":0.90600000000000003,"5min":0.90600000000000003,"15min":0.90600000000000003},"min":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.67100000000000004},"max":{"1min":1.367,"5min":1.367,"15min":1.367},"last":0.81699999999999995}]},{"osd":3,"last update":"Fri Oct 6 03:30:23 2023","interfaces":[{"interface":"back","average":{"1min":1.034,"5min":1.034,"15min":1.034},"min":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.77200000000000002},"max":{"1min":1.4079999999999999,"5min":1.4079999999999999,"15min":1.4079999999999999},"last":0.91500000000000004},{"interface":"front","average":{"1min":1.034,"5min":1.034,"15min":1.034},"min":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"max":{"1min":1.244,"5min":1.244,"15min":1.244},"last":1.258}]},{"osd":4,"last update":"Fri Oct 6 03:30:23 2023","interfaces":[{"interface":"back","average":{"1min":1.0920000000000001,"5min":1.0920000000000001,"15min":1.0920000000000001},"min":{"1min":0.752,"5min":0.752,"15min":0.752},"max":{"1min":2.2050000000000001,"5min":2.2050000000000001,"15min":2.2050000000000001},"last":0.94699999999999995},{"interface":"front","average":{"1min":1.1870000000000001,"5min":1.1870000000000001,"15min":1.1870000000000001},"min":{"1min":0.90900000000000003,"5min":0.90900000000000003,"15min":0.90900000000000003},"max":{"1min":1.518,"5min":1.518,"15min":1.518},"last":0.998}]},{"osd":6,"last update":"Fri Oct 6 03:30:23 2023","interfaces":[{"interface":"back","average":{"1min":1.1040000000000001,"5min":1.1040000000000001,"15min":1.1040000000000001},"min":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"max":{"1min":1.3169999999999999,"5min":1.3169999999999999,"15min":1.3169999999999999},"last":1.1970000000000001},{"interface":"front","average":{"1min":1.2370000000000001,"5min":1.2370000000000001,"15min":1.2370000000000001},"min":{"1min":0.94599999999999995,"5min":0.94599999999999995,"15min":0.94599999999999995},"max":{"1min":2.3809999999999998,"5min":2.3809999999999998,"15min":2.3809999999999998},"last":1.1559999999999999}]},{"osd":7,"last update":"Fri Oct 6 03:30:23 2023","interfaces":[{"interface":"back","average":{"1min":1.2370000000000001,"5min":1.2370000000000001,"15min":1.2370000000000001},"min":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.79500000000000004},"max":{"1min":1.637,"5min":1.637,"15min":1.637},"last":1.121},{"interface":"front","average":{"1min":1.2929999999999999,"5min":1.2929999999999999,"15min":1.2929999999999999},"min":{"1min":1.0429999999999999,"5min":1.0429999999999999,"15min":1.0429999999999999},"max":{"1min":2.3330000000000002,"5min":2.3330000000000002,"15min":2.3330000000000002},"last":1.3129999999999999}]}]},{"osd":4,"up_from":16,"seq":68719476754,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297704,"kb_used_data":416,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074136,"statfs":{"total":96636764160,"available":96331915264,"internally_reserved":0,"allocated":425984,"data_stored":184768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Oct 6 03:30:25 2023","interfaces":[{"interface":"back","average":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"min":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"max":{"1min":1.129,"5min":1.129,"15min":1.129},"last":0.68500000000000005},{"interface":"front","average":{"1min":0.96299999999999997,"5min":0.96299999999999997,"15min":0.96299999999999997},"min":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"max":{"1min":1.629,"5min":1.629,"15min":1.629},"last":0.48999999999999999}]},{"osd":1,"last update":"Fri Oct 6 03:30:25 2023","interfaces":[{"interface":"back","average":{"1min":0.89300000000000002,"5min":0.89300000000000002,"15min":0.89300000000000002},"min":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"max":{"1min":1.23,"5min":1.23,"15min":1.23},"last":0.86799999999999999},{"interface":"front","average":{"1min":0.86799999999999999,"5min":0.86799999999999999,"15min":0.86799999999999999},"min":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"max":{"1min":1.3680000000000001,"5min":1.3680000000000001,"15min":1.3680000000000001},"last":0.44900000000000001}]},{"osd":2,"last update":"Fri Oct 6 03:30:25 2023","interfaces":[{"interface":"back","average":{"1min":0.98999999999999999,"5min":0.98999999999999999,"15min":0.98999999999999999},"min":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":1.5549999999999999,"5min":1.5549999999999999,"15min":1.5549999999999999},"last":0.83499999999999996},{"interface":"front","average":{"1min":0.92100000000000004,"5min":0.92100000000000004,"15min":0.92100000000000004},"min":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.60799999999999998},"max":{"1min":1.306,"5min":1.306,"15min":1.306},"last":1.107}]},{"osd":3,"last update":"Fri Oct 6 03:30:25 2023","interfaces":[{"interface":"back","average":{"1min":1.1040000000000001,"5min":1.1040000000000001,"15min":1.1040000000000001},"min":{"1min":0.84499999999999997,"5min":0.84499999999999997,"15min":0.84499999999999997},"max":{"1min":1.5580000000000001,"5min":1.5580000000000001,"15min":1.5580000000000001},"last":0.73799999999999999},{"interface":"front","average":{"1min":1.04,"5min":1.04,"15min":1.04},"min":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.67500000000000004},"max":{"1min":1.6699999999999999,"5min":1.6699999999999999,"15min":1.6699999999999999},"last":1.052}]},{"osd":5,"last update":"Fri Oct 6 03:30:25 2023","interfaces":[{"interface":"back","average":{"1min":1.044,"5min":1.044,"15min":1.044},"min":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.73399999999999999},"max":{"1min":1.5169999999999999,"5min":1.5169999999999999,"15min":1.5169999999999999},"last":0.80000000000000004},{"interface":"front","average":{"1min":1.266,"5min":1.266,"15min":1.266},"min":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":0.82799999999999996},"max":{"1min":2.2029999999999998,"5min":2.2029999999999998,"15min":2.2029999999999998},"last":0.92800000000000005}]},{"osd":6,"last update":"Fri Oct 6 03:30:25 2023","interfaces":[{"interface":"back","average":{"1min":1.181,"5min":1.181,"15min":1.181},"min":{"1min":0.79900000000000004,"5min":0.79900000000000004,"15min":0.79900000000000004},"max":{"1min":2.0209999999999999,"5min":2.0209999999999999,"15min":2.0209999999999999},"last":1.155},{"interface":"front","average":{"1min":1.179,"5min":1.179,"15min":1.179},"min":{"1min":0.71399999999999997,"5min":0.71399999999999997,"15min":0.71399999999999997},"max":{"1min":2.1440000000000001,"5min":2.1440000000000001,"15min":2.1440000000000001},"last":0.96599999999999997}]},{"osd":7,"last update":"Fri Oct 6 03:30:25 2023","interfaces":[{"interface":"back","average":{"1min":1.337,"5min":1.337,"15min":1.337},"min":{"1min":0.94399999999999995,"5min":0.94399999999999995,"15min":0.94399999999999995},"max":{"1min":2.0939999999999999,"5min":2.0939999999999999,"15min":2.0939999999999999},"last":1.018},{"interface":"front","average":{"1min":1.3089999999999999,"5min":1.3089999999999999,"15min":1.3089999999999999},"min":{"1min":0.95899999999999996,"5min":0.95899999999999996,"15min":0.95899999999999996},"max":{"1min":1.968,"5min":1.968,"15min":1.968},"last":1.2070000000000001}]}]},{"osd":7,"up_from":15,"seq":64424509458,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298284,"kb_used_data":996,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94073556,"statfs":{"total":96636764160,"available":96331321344,"internally_reserved":0,"allocated":1019904,"data_stored":775136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Oct 6 03:30:26 2023","interfaces":[{"interface":"back","average":{"1min":0.878,"5min":0.878,"15min":0.878},"min":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"max":{"1min":1.228,"5min":1.228,"15min":1.228},"last":0.81200000000000006},{"interface":"front","average":{"1min":0.878,"5min":0.878,"15min":0.878},"min":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":1.181,"5min":1.181,"15min":1.181},"last":0.64800000000000002}]},{"osd":1,"last update":"Fri Oct 6 03:30:26 2023","interfaces":[{"interface":"back","average":{"1min":0.93000000000000005,"5min":0.93000000000000005,"15min":0.93000000000000005},"min":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"max":{"1min":1.2889999999999999,"5min":1.2889999999999999,"15min":1.2889999999999999},"last":0.59599999999999997},{"interface":"front","average":{"1min":0.94299999999999995,"5min":0.94299999999999995,"15min":0.94299999999999995},"min":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"max":{"1min":1.341,"5min":1.341,"15min":1.341},"last":0.88700000000000001}]},{"osd":2,"last update":"Fri Oct 6 03:30:26 2023","interfaces":[{"interface":"back","average":{"1min":0.95299999999999996,"5min":0.95299999999999996,"15min":0.95299999999999996},"min":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.72599999999999998},"max":{"1min":1.2050000000000001,"5min":1.2050000000000001,"15min":1.2050000000000001},"last":0.73199999999999998},{"interface":"front","average":{"1min":1.006,"5min":1.006,"15min":1.006},"min":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"max":{"1min":1.409,"5min":1.409,"15min":1.409},"last":0.75900000000000001}]},{"osd":3,"last update":"Fri Oct 6 03:30:26 2023","interfaces":[{"interface":"back","average":{"1min":0.86299999999999999,"5min":0.86299999999999999,"15min":0.86299999999999999},"min":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"max":{"1min":1.1339999999999999,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":1.073},{"interface":"front","average":{"1min":0.96899999999999997,"5min":0.96899999999999997,"15min":0.96899999999999997},"min":{"1min":0.63400000000000001,"5min":0.63400000000000001,"15min":0.63400000000000001},"max":{"1min":1.256,"5min":1.256,"15min":1.256},"last":0.78700000000000003}]},{"osd":4,"last update":"Fri Oct 6 03:30:26 2023","interfaces":[{"interface":"back","average":{"1min":0.91900000000000004,"5min":0.91900000000000004,"15min":0.91900000000000004},"min":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"max":{"1min":1.2230000000000001,"5min":1.2230000000000001,"15min":1.2230000000000001},"last":0.96299999999999997},{"interface":"front","average":{"1min":1.0529999999999999,"5min":1.0529999999999999,"15min":1.0529999999999999},"min":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"max":{"1min":1.4750000000000001,"5min":1.4750000000000001,"15min":1.4750000000000001},"last":0.69999999999999996}]},{"osd":5,"last update":"Fri Oct 6 03:30:26 2023","interfaces":[{"interface":"back","average":{"1min":1.0229999999999999,"5min":1.0229999999999999,"15min":1.0229999999999999},"min":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":0.77900000000000003},"max":{"1min":1.431,"5min":1.431,"15min":1.431},"last":1.014},{"interface":"front","average":{"1min":1.105,"5min":1.105,"15min":1.105},"min":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"max":{"1min":1.407,"5min":1.407,"15min":1.407},"last":0.84699999999999998}]},{"osd":6,"last update":"Fri Oct 6 03:30:26 2023","interfaces":[{"interface":"back","average":{"1min":1.1399999999999999,"5min":1.1399999999999999,"15min":1.1399999999999999},"min":{"1min":0.86799999999999999,"5min":0.86799999999999999,"15min":0.86799999999999999},"max":{"1min":1.526,"5min":1.526,"15min":1.526},"last":0.94099999999999995},{"interface":"front","average":{"1min":1.198,"5min":1.198,"15min":1.198},"min":{"1min":0.98399999999999999,"5min":0.98399999999999999,"15min":0.98399999999999999},"max":{"1min":1.6719999999999999,"5min":1.6719999999999999,"15min":1.6719999999999999},"last":1.125}]}]},{"osd":0,"up_from":15,"seq":64424509459,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302380,"kb_used_data":996,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94069460,"statfs":{"total":96636764160,"available":96327127040,"internally_reserved":0,"allocated":1019904,"data_stored":775136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Oct 6 03:30:22 2023","interfaces":[{"interface":"back","average":{"1min":0.629,"5min":0.629,"15min":0.629},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":1.0029999999999999,"5min":1.0029999999999999,"15min":1.0029999999999999},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.74199999999999999},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":1.179,"5min":1.179,"15min":1.179},"last":0.46600000000000003}]},{"osd":2,"last update":"Fri Oct 6 03:30:22 2023","interfaces":[{"interface":"back","average":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":1.1879999999999999,"5min":1.1879999999999999,"15min":1.1879999999999999},"last":0.63},{"interface":"front","average":{"1min":0.72499999999999998,"5min":0.72499999999999998,"15min":0.72499999999999998},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":1.3879999999999999,"5min":1.3879999999999999,"15min":1.3879999999999999},"last":0.60299999999999998}]},{"osd":3,"last update":"Fri Oct 6 03:30:22 2023","interfaces":[{"interface":"back","average":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":1.1930000000000001,"5min":1.1930000000000001,"15min":1.1930000000000001},"last":0.84299999999999997},{"interface":"front","average":{"1min":0.86399999999999999,"5min":0.86399999999999999,"15min":0.86399999999999999},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":1.1970000000000001,"5min":1.1970000000000001,"15min":1.1970000000000001},"last":0.747}]},{"osd":4,"last update":"Fri Oct 6 03:30:22 2023","interfaces":[{"interface":"back","average":{"1min":1.0189999999999999,"5min":1.0189999999999999,"15min":1.0189999999999999},"min":{"1min":0.72899999999999998,"5min":0.72899999999999998,"15min":0.72899999999999998},"max":{"1min":1.55,"5min":1.55,"15min":1.55},"last":0.872},{"interface":"front","average":{"1min":0.99099999999999999,"5min":0.99099999999999999,"15min":0.99099999999999999},"min":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"max":{"1min":1.5269999999999999,"5min":1.5269999999999999,"15min":1.5269999999999999},"last":0.89300000000000002}]},{"osd":5,"last update":"Fri Oct 6 03:30:22 2023","interfaces":[{"interface":"back","average":{"1min":0.98499999999999999,"5min":0.98499999999999999,"15min":0.98499999999999999},"min":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"max":{"1min":1.5409999999999999,"5min":1.5409999999999999,"15min":1.5409999999999999},"last":0.81100000000000005},{"interface":"front","average":{"1min":1.087,"5min":1.087,"15min":1.087},"min":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"max":{"1min":1.353,"5min":1.353,"15min":1.353},"last":0.99399999999999999}]},{"osd":6,"last update":"Fri Oct 6 03:30:22 2023","interfaces":[{"interface":"back","average":{"1min":1.145,"5min":1.145,"15min":1.145},"min":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"max":{"1min":1.502,"5min":1.502,"15min":1.502},"last":1.125},{"interface":"front","average":{"1min":1.1299999999999999,"5min":1.1299999999999999,"15min":1.1299999999999999},"min":{"1min":0.64900000000000002,"5min":0.64900000000000002,"15min":0.64900000000000002},"max":{"1min":1.6439999999999999,"5min":1.6439999999999999,"15min":1.6439999999999999},"last":1.014}]},{"osd":7,"last update":"Fri Oct 6 03:30:22 2023","interfaces":[{"interface":"back","average":{"1min":1.099,"5min":1.099,"15min":1.099},"min":{"1min":0.874,"5min":0.874,"15min":0.874},"max":{"1min":1.369,"5min":1.369,"15min":1.369},"last":0.95099999999999996},{"interface":"front","average":{"1min":1.2030000000000001,"5min":1.2030000000000001,"15min":1.2030000000000001},"min":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"max":{"1min":1.573,"5min":1.573,"15min":1.573},"last":1.0820000000000001}]}]},{"osd":3,"up_from":14,"seq":60129542163,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":301848,"kb_used_data":464,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94069992,"statfs":{"total":96636764160,"available":96327671808,"internally_reserved":0,"allocated":475136,"data_stored":206220,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Oct 6 03:30:26 2023","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.81899999999999995,"5min":0.81899999999999995,"15min":0.81899999999999995},"last":0.70299999999999996},{"interface":"front","average":{"1min":0.754,"5min":0.754,"15min":0.754},"min":{"1min":0.505,"5min":0.505,"15min":0.505},"max":{"1min":1.0920000000000001,"5min":1.0920000000000001,"15min":1.0920000000000001},"last":0.85399999999999998}]},{"osd":1,"last update":"Fri Oct 6 03:30:20 2023","interfaces":[{"interface":"back","average":{"1min":0.83199999999999996,"5min":0.83199999999999996,"15min":0.83199999999999996},"min":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":1.2210000000000001,"5min":1.2210000000000001,"15min":1.2210000000000001},"last":0.621},{"interface":"front","average":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":1.321,"5min":1.321,"15min":1.321},"last":0.88700000000000001}]},{"osd":2,"last update":"Fri Oct 6 03:30:20 2023","interfaces":[{"interface":"back","average":{"1min":0.86799999999999999,"5min":0.86799999999999999,"15min":0.86799999999999999},"min":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.61199999999999999},"max":{"1min":1.228,"5min":1.228,"15min":1.228},"last":0.73499999999999999},{"interface":"front","average":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"min":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":1.3680000000000001,"5min":1.3680000000000001,"15min":1.3680000000000001},"last":0.753}]},{"osd":4,"last update":"Fri Oct 6 03:30:26 2023","interfaces":[{"interface":"back","average":{"1min":1.0129999999999999,"5min":1.0129999999999999,"15min":1.0129999999999999},"min":{"1min":0.76800000000000002,"5min":0.76800000000000002,"15min":0.76800000000000002},"max":{"1min":1.6359999999999999,"5min":1.6359999999999999,"15min":1.6359999999999999},"last":0.80700000000000005},{"interface":"front","average":{"1min":0.96699999999999997,"5min":0.96699999999999997,"15min":0.96699999999999997},"min":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"max":{"1min":1.409,"5min":1.409,"15min":1.409},"last":0.97499999999999998}]},{"osd":5,"last update":"Fri Oct 6 03:30:26 2023","interfaces":[{"interface":"back","average":{"1min":0.98999999999999999,"5min":0.98999999999999999,"15min":0.98999999999999999},"min":{"1min":0.746,"5min":0.746,"15min":0.746},"max":{"1min":1.387,"5min":1.387,"15min":1.387},"last":1.1599999999999999},{"interface":"front","average":{"1min":1.1359999999999999,"5min":1.1359999999999999,"15min":1.1359999999999999},"min":{"1min":0.89000000000000001,"5min":0.89000000000000001,"15min":0.89000000000000001},"max":{"1min":1.615,"5min":1.615,"15min":1.615},"last":0.94499999999999995}]},{"osd":6,"last update":"Fri Oct 6 03:30:26 2023","interfaces":[{"interface":"back","average":{"1min":1.079,"5min":1.079,"15min":1.079},"min":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"max":{"1min":1.643,"5min":1.643,"15min":1.643},"last":1.056},{"interface":"front","average":{"1min":1.105,"5min":1.105,"15min":1.105},"min":{"1min":0.74399999999999999,"5min":0.74399999999999999,"15min":0.74399999999999999},"max":{"1min":1.6819999999999999,"5min":1.6819999999999999,"15min":1.6819999999999999},"last":1.232}]},{"osd":7,"last update":"Fri Oct 6 03:30:26 2023","interfaces":[{"interface":"back","average":{"1min":1.1850000000000001,"5min":1.1850000000000001,"15min":1.1850000000000001},"min":{"1min":0.92000000000000004,"5min":0.92000000000000004,"15min":0.92000000000000004},"max":{"1min":1.597,"5min":1.597,"15min":1.597},"last":1.0149999999999999},{"interface":"front","average":{"1min":1.111,"5min":1.111,"15min":1.111},"min":{"1min":0.86599999999999999,"5min":0.86599999999999999,"15min":0.86599999999999999},"max":{"1min":1.446,"5min":1.446,"15min":1.446},"last":1.123}]}]},{"osd":2,"up_from":14,"seq":60129542162,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":301852,"kb_used_data":468,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94069988,"statfs":{"total":96636764160,"available":96327667712,"internally_reserved":0,"allocated":479232,"data_stored":207168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Oct 6 03:30:21 2023","interfaces":[{"interface":"back","average":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":1.1240000000000001,"5min":1.1240000000000001,"15min":1.1240000000000001},"last":0.88700000000000001},{"interface":"front","average":{"1min":0.70399999999999996,"5min":0.70399999999999996,"15min":0.70399999999999996},"min":{"1min":0.495,"5min":0.495,"15min":0.495},"max":{"1min":1.1559999999999999,"5min":1.1559999999999999,"15min":1.1559999999999999},"last":0.82099999999999995}]},{"osd":1,"last update":"Fri Oct 6 03:30:21 2023","interfaces":[{"interface":"back","average":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"min":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":1.1060000000000001,"5min":1.1060000000000001,"15min":1.1060000000000001},"last":0.67400000000000004},{"interface":"front","average":{"1min":0.73499999999999999,"5min":0.73499999999999999,"15min":0.73499999999999999},"min":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":1.2390000000000001,"5min":1.2390000000000001,"15min":1.2390000000000001},"last":0.65300000000000002}]},{"osd":3,"last update":"Fri Oct 6 03:30:21 2023","interfaces":[{"interface":"back","average":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":0.78500000000000003},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":1.165,"5min":1.165,"15min":1.165},"last":0.79200000000000004},{"interface":"front","average":{"1min":0.873,"5min":0.873,"15min":0.873},"min":{"1min":0.626,"5min":0.626,"15min":0.626},"max":{"1min":1.2949999999999999,"5min":1.2949999999999999,"15min":1.2949999999999999},"last":0.435}]},{"osd":4,"last update":"Fri Oct 6 03:30:23 2023","interfaces":[{"interface":"back","average":{"1min":1.095,"5min":1.095,"15min":1.095},"min":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"max":{"1min":2.3450000000000002,"5min":2.3450000000000002,"15min":2.3450000000000002},"last":1.0069999999999999},{"interface":"front","average":{"1min":0.95599999999999996,"5min":0.95599999999999996,"15min":0.95599999999999996},"min":{"1min":0.74399999999999999,"5min":0.74399999999999999,"15min":0.74399999999999999},"max":{"1min":1.1950000000000001,"5min":1.1950000000000001,"15min":1.1950000000000001},"last":0.85699999999999998}]},{"osd":5,"last update":"Fri Oct 6 03:30:23 2023","interfaces":[{"interface":"back","average":{"1min":1.139,"5min":1.139,"15min":1.139},"min":{"1min":0.79900000000000004,"5min":0.79900000000000004,"15min":0.79900000000000004},"max":{"1min":2.5169999999999999,"5min":2.5169999999999999,"15min":2.5169999999999999},"last":0.96099999999999997},{"interface":"front","average":{"1min":1.107,"5min":1.107,"15min":1.107},"min":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":0.81799999999999995},"max":{"1min":1.7250000000000001,"5min":1.7250000000000001,"15min":1.7250000000000001},"last":1.032}]},{"osd":6,"last update":"Fri Oct 6 03:30:23 2023","interfaces":[{"interface":"back","average":{"1min":1.014,"5min":1.014,"15min":1.014},"min":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"max":{"1min":1.218,"5min":1.218,"15min":1.218},"last":0.76600000000000001},{"interface":"front","average":{"1min":1.1899999999999999,"5min":1.1899999999999999,"15min":1.1899999999999999},"min":{"1min":0.85899999999999999,"5min":0.85899999999999999,"15min":0.85899999999999999},"max":{"1min":2.2450000000000001,"5min":2.2450000000000001,"15min":2.2450000000000001},"last":1.1539999999999999}]},{"osd":7,"last update":"Fri Oct 6 03:30:21 2023","interfaces":[{"interface":"back","average":{"1min":1.1639999999999999,"5min":1.1639999999999999,"15min":1.1639999999999999},"min":{"1min":0.81599999999999995,"5min":0.81599999999999995,"15min":0.81599999999999995},"max":{"1min":1.8140000000000001,"5min":1.8140000000000001,"15min":1.8140000000000001},"last":0.98499999999999999},{"interface":"front","average":{"1min":1.0860000000000001,"5min":1.0860000000000001,"15min":1.0860000000000001},"min":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.79500000000000004},"max":{"1min":1.2749999999999999,"5min":1.2749999999999999,"15min":1.2749999999999999},"last":0.92500000000000004}]}]},{"osd":1,"up_from":14,"seq":60129542162,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":301840,"kb_used_data":456,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94070000,"statfs":{"total":96636764160,"available":96327680000,"internally_reserved":0,"allocated":466944,"data_stored":205680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Oct 6 03:30:26 2023","interfaces":[{"interface":"back","average":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.60799999999999998},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.78500000000000003},{"interface":"front","average":{"1min":0.71599999999999997,"5min":0.71599999999999997,"15min":0.71599999999999997},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":1.5329999999999999,"5min":1.5329999999999999,"15min":1.5329999999999999},"last":0.74199999999999999}]},{"osd":2,"last update":"Fri Oct 6 03:30:20 2023","interfaces":[{"interface":"back","average":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.93899999999999995,"5min":0.93899999999999995,"15min":0.93899999999999995},"last":0.625},{"interface":"front","average":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"min":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"max":{"1min":1.278,"5min":1.278,"15min":1.278},"last":0.51000000000000001}]},{"osd":3,"last update":"Fri Oct 6 03:30:20 2023","interfaces":[{"interface":"back","average":{"1min":0.72899999999999998,"5min":0.72899999999999998,"15min":0.72899999999999998},"min":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"max":{"1min":0.90500000000000003,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.54900000000000004},{"interface":"front","average":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.3700000000000001,"5min":1.3700000000000001,"15min":1.3700000000000001},"last":0.83099999999999996}]},{"osd":4,"last update":"Fri Oct 6 03:30:26 2023","interfaces":[{"interface":"back","average":{"1min":0.94299999999999995,"5min":0.94299999999999995,"15min":0.94299999999999995},"min":{"1min":0.66400000000000003,"5min":0.66400000000000003,"15min":0.66400000000000003},"max":{"1min":1.7549999999999999,"5min":1.7549999999999999,"15min":1.7549999999999999},"last":0.91300000000000003},{"interface":"front","average":{"1min":0.97499999999999998,"5min":0.97499999999999998,"15min":0.97499999999999998},"min":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":1.415,"5min":1.415,"15min":1.415},"last":1.103}]},{"osd":5,"last update":"Fri Oct 6 03:30:26 2023","interfaces":[{"interface":"back","average":{"1min":0.93999999999999995,"5min":0.93999999999999995,"15min":0.93999999999999995},"min":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"max":{"1min":1.4910000000000001,"5min":1.4910000000000001,"15min":1.4910000000000001},"last":0.96999999999999997},{"interface":"front","average":{"1min":1.0049999999999999,"5min":1.0049999999999999,"15min":1.0049999999999999},"min":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":0.70499999999999996},"max":{"1min":1.4370000000000001,"5min":1.4370000000000001,"15min":1.4370000000000001},"last":1.137}]},{"osd":6,"last update":"Fri Oct 6 03:30:26 2023","interfaces":[{"interface":"back","average":{"1min":1.022,"5min":1.022,"15min":1.022},"min":{"1min":0.69199999999999995,"5min":0.69199999999999995,"15min":0.69199999999999995},"max":{"1min":1.603,"5min":1.603,"15min":1.603},"last":1.159},{"interface":"front","average":{"1min":1.1080000000000001,"5min":1.1080000000000001,"15min":1.1080000000000001},"min":{"1min":0.69199999999999995,"5min":0.69199999999999995,"15min":0.69199999999999995},"max":{"1min":1.5069999999999999,"5min":1.5069999999999999,"15min":1.5069999999999999},"last":0.998}]},{"osd":7,"last update":"Fri Oct 6 03:30:26 2023","interfaces":[{"interface":"back","average":{"1min":1.04,"5min":1.04,"15min":1.04},"min":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.60799999999999998},"max":{"1min":1.6160000000000001,"5min":1.6160000000000001,"15min":1.6160000000000001},"last":1.0249999999999999},{"interface":"front","average":{"1min":1.071,"5min":1.071,"15min":1.071},"min":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.67500000000000004},"max":{"1min":1.5700000000000001,"5min":1.5700000000000001,"15min":1.5700000000000001},"last":1.0489999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1558,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":610,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-06T03:30:38.683 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-06T03:30:38.786 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:38.780+0000 7f73b359e700 1 Processor -- start 2023-10-06T03:30:38.787 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:38.780+0000 7f73b359e700 1 -- start start 2023-10-06T03:30:38.789 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:38.784+0000 7f73b359e700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f73b4153290 0x7f73b4159e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:38.789 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:38.784+0000 7f73b359e700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f73b4069660 0x7f73b406d700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:38.789 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:38.784+0000 7f73b359e700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73b406d270 con 0x7f73b40cc1a0 2023-10-06T03:30:38.789 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:38.784+0000 7f73b359e700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f73b40cb070 con 0x7f73b4069660 2023-10-06T03:30:38.790 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:38.784+0000 7f73b359e700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f73b415a370 con 0x7f73b4153290 2023-10-06T03:30:38.790 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:38.784+0000 7f73abfff700 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f73b40cc1a0 0x7f73b4068f50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:54646/0 (socket says 172.21.15.110:54646) 2023-10-06T03:30:38.790 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:38.784+0000 7f73abfff700 1 -- 172.21.15.110:0/848629308 learned_addr learned my addr 172.21.15.110:0/848629308 (peer_addr_for_me v1:172.21.15.110:0/0) 2023-10-06T03:30:38.790 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:38.784+0000 7f73b2d9d700 1 --2- 172.21.15.110:0/848629308 >> v2:172.21.15.110:3300/0 conn(0x7f73b4069660 0x7f73b406d700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:38.790 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:38.784+0000 7f73b259c700 1 --2- 172.21.15.110:0/848629308 >> v2:172.21.15.192:3300/0 conn(0x7f73b4153290 0x7f73b4159e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:38.790 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:38.784+0000 7f73b1d9b700 1 -- 172.21.15.110:0/848629308 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4116618598 0 0) 0x7f73b406d270 con 0x7f73b40cc1a0 2023-10-06T03:30:38.790 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:38.784+0000 7f73b1d9b700 1 -- 172.21.15.110:0/848629308 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f73a0003570 con 0x7f73b40cc1a0 2023-10-06T03:30:38.790 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:38.784+0000 7f73b2d9d700 1 -- 172.21.15.110:0/848629308 >> v2:172.21.15.192:3300/0 conn(0x7f73b4153290 msgr2=0x7f73b4159e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:30:38.790 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:38.784+0000 7f73b2d9d700 1 --2- 172.21.15.110:0/848629308 >> v2:172.21.15.192:3300/0 conn(0x7f73b4153290 0x7f73b4159e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:38.790 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:38.784+0000 7f73b2d9d700 1 -- 172.21.15.110:0/848629308 >> v1:172.21.15.110:6789/0 conn(0x7f73b40cc1a0 legacy=0x7f73b4068f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:38.790 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:38.784+0000 7f73b2d9d700 1 -- 172.21.15.110:0/848629308 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73b406cc60 con 0x7f73b4069660 2023-10-06T03:30:38.791 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:38.784+0000 7f73b259c700 1 --2- 172.21.15.110:0/848629308 >> v2:172.21.15.192:3300/0 conn(0x7f73b4153290 0x7f73b4159e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-06T03:30:38.791 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:38.784+0000 7f73b2d9d700 1 --2- 172.21.15.110:0/848629308 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f73b4069660 0x7f73b406d700 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f73b40685f0 tx=0x7f73ac009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=97d8d0b7577ca2d8 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:30:38.791 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:38.784+0000 7f73b1d9b700 1 -- 172.21.15.110:0/848629308 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73ac00e070 con 0x7f73b4069660 2023-10-06T03:30:38.791 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:38.784+0000 7f73b1d9b700 1 -- 172.21.15.110:0/848629308 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73ac00bb60 con 0x7f73b4069660 2023-10-06T03:30:38.791 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:38.784+0000 7f73b1d9b700 1 -- 172.21.15.110:0/848629308 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73ac012430 con 0x7f73b4069660 2023-10-06T03:30:38.792 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:38.784+0000 7f73b359e700 1 -- 172.21.15.110:0/848629308 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f73b4069660 msgr2=0x7f73b406d700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:38.792 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:38.784+0000 7f73b359e700 1 --2- 172.21.15.110:0/848629308 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f73b4069660 0x7f73b406d700 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f73b40685f0 tx=0x7f73ac009780 comp rx=0 tx=0).stop 2023-10-06T03:30:38.792 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:38.784+0000 7f73b359e700 1 -- 172.21.15.110:0/848629308 shutdown_connections 2023-10-06T03:30:38.792 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:38.784+0000 7f73b359e700 1 --2- 172.21.15.110:0/848629308 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f73b4069660 0x7f73b406d700 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:38.792 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:38.784+0000 7f73b359e700 1 --2- 172.21.15.110:0/848629308 >> v2:172.21.15.192:3300/0 conn(0x7f73b4153290 0x7f73b4159e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:38.792 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:38.784+0000 7f73b359e700 1 -- 172.21.15.110:0/848629308 >> 172.21.15.110:0/848629308 conn(0x7f73b4067700 msgr2=0x7f73b4067ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:30:38.792 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:38.784+0000 7f73b359e700 1 -- 172.21.15.110:0/848629308 shutdown_connections 2023-10-06T03:30:38.792 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:38.784+0000 7f73b359e700 1 -- 172.21.15.110:0/848629308 wait complete. 2023-10-06T03:30:38.793 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:38.788+0000 7f73b359e700 1 Processor -- start 2023-10-06T03:30:38.793 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:38.788+0000 7f73b359e700 1 -- start start 2023-10-06T03:30:38.795 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:38.788+0000 7f73b359e700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f73b4069660 0x7f73b40c5bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:38.795 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:38.788+0000 7f73b359e700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f73b40cc1a0 0x7f73b40c32e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:38.795 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:38.788+0000 7f73b359e700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f73b4153290 0x7f73b40c7300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:38.795 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:38.788+0000 7f73b359e700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73b416bde0 con 0x7f73b40cc1a0 2023-10-06T03:30:38.795 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:38.788+0000 7f73b359e700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73b40c57a0 con 0x7f73b4153290 2023-10-06T03:30:38.795 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:38.788+0000 7f73b359e700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73b40b2e10 con 0x7f73b4069660 2023-10-06T03:30:38.795 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:38.788+0000 7f73abfff700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f73b40cc1a0 0x7f73b40c32e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:38.795 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:38.788+0000 7f73abfff700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f73b40cc1a0 0x7f73b40c32e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:48178/0 (socket says 172.21.15.110:48178) 2023-10-06T03:30:38.795 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:38.788+0000 7f73abfff700 1 -- 172.21.15.110:0/2446917754 learned_addr learned my addr 172.21.15.110:0/2446917754 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:30:38.796 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:38.788+0000 7f73b259c700 1 --2- 172.21.15.110:0/2446917754 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f73b4069660 0x7f73b40c5bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:38.796 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:38.788+0000 7f73b2d9d700 1 --2- 172.21.15.110:0/2446917754 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f73b4153290 0x7f73b40c7300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:38.796 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:38.788+0000 7f73abfff700 1 -- 172.21.15.110:0/2446917754 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f73b4069660 msgr2=0x7f73b40c5bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:38.796 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:38.788+0000 7f73abfff700 1 --2- 172.21.15.110:0/2446917754 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f73b4069660 0x7f73b40c5bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:38.796 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:38.788+0000 7f73abfff700 1 -- 172.21.15.110:0/2446917754 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f73b4153290 msgr2=0x7f73b40c7300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:38.796 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:38.788+0000 7f73abfff700 1 --2- 172.21.15.110:0/2446917754 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f73b4153290 0x7f73b40c7300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:38.796 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:38.788+0000 7f73abfff700 1 -- 172.21.15.110:0/2446917754 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73ac009a20 con 0x7f73b40cc1a0 2023-10-06T03:30:38.796 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:38.788+0000 7f73abfff700 1 --2- 172.21.15.110:0/2446917754 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f73b40cc1a0 0x7f73b40c32e0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f739c0011e0 tx=0x7f739c007df0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:30:38.797 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:38.792+0000 7f73aa7fc700 1 -- 172.21.15.110:0/2446917754 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f739c005b30 con 0x7f73b40cc1a0 2023-10-06T03:30:38.797 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:38.792+0000 7f73aa7fc700 1 -- 172.21.15.110:0/2446917754 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f739c005c90 con 0x7f73b40cc1a0 2023-10-06T03:30:38.797 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:38.792+0000 7f73b359e700 1 -- 172.21.15.110:0/2446917754 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73b40b30f0 con 0x7f73b40cc1a0 2023-10-06T03:30:38.797 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:38.792+0000 7f73aa7fc700 1 -- 172.21.15.110:0/2446917754 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f739c007300 con 0x7f73b40cc1a0 2023-10-06T03:30:38.798 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:38.792+0000 7f73b359e700 1 -- 172.21.15.110:0/2446917754 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73b40b36b0 con 0x7f73b40cc1a0 2023-10-06T03:30:38.798 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:38.792+0000 7f73b359e700 1 -- 172.21.15.110:0/2446917754 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f73b416bf20 con 0x7f73b40cc1a0 2023-10-06T03:30:38.798 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:38.792+0000 7f73aa7fc700 1 -- 172.21.15.110:0/2446917754 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7f739c007670 con 0x7f73b40cc1a0 2023-10-06T03:30:38.798 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:38.792+0000 7f73aa7fc700 1 --2- 172.21.15.110:0/2446917754 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f7394060550 0x7f7394062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:38.799 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:38.792+0000 7f73aa7fc700 1 -- 172.21.15.110:0/2446917754 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6378+0+0 (secure 0 0 0) 0x7f739c07be50 con 0x7f73b40cc1a0 2023-10-06T03:30:38.799 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:38.792+0000 7f73b259c700 1 --2- 172.21.15.110:0/2446917754 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f7394060550 0x7f7394062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:38.799 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:38.792+0000 7f73b259c700 1 --2- 172.21.15.110:0/2446917754 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f7394060550 0x7f7394062a00 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f73a4006f00 tx=0x7f73a4008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:30:38.804 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:38.800+0000 7f73aa7fc700 1 -- 172.21.15.110:0/2446917754 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f739c04c1a0 con 0x7f73b40cc1a0 2023-10-06T03:30:38.961 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:38.956+0000 7f73b359e700 1 -- 172.21.15.110:0/2446917754 --> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f73b40b2bd0 con 0x7f7394060550 2023-10-06T03:30:38.979 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:38.972+0000 7f73aa7fc700 1 -- 172.21.15.110:0/2446917754 <== mgr.4110 v2:172.21.15.110:6832/33268 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+309455 (secure 0 0 0) 0x7f73b40b2bd0 con 0x7f7394060550 2023-10-06T03:30:38.983 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-10-06T03:30:38.985 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:38.980+0000 7f73b359e700 1 -- 172.21.15.110:0/2446917754 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f7394060550 msgr2=0x7f7394062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:38.985 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:38.980+0000 7f73b359e700 1 --2- 172.21.15.110:0/2446917754 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f7394060550 0x7f7394062a00 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f73a4006f00 tx=0x7f73a4008040 comp rx=0 tx=0).stop 2023-10-06T03:30:38.985 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:38.980+0000 7f73b359e700 1 -- 172.21.15.110:0/2446917754 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f73b40cc1a0 msgr2=0x7f73b40c32e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:38.985 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:38.980+0000 7f73b359e700 1 --2- 172.21.15.110:0/2446917754 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f73b40cc1a0 0x7f73b40c32e0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f739c0011e0 tx=0x7f739c007df0 comp rx=0 tx=0).stop 2023-10-06T03:30:38.985 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:38.980+0000 7f73b359e700 1 -- 172.21.15.110:0/2446917754 shutdown_connections 2023-10-06T03:30:38.985 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:38.980+0000 7f73b359e700 1 --2- 172.21.15.110:0/2446917754 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f7394060550 0x7f7394062a00 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:38.986 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:38.980+0000 7f73b359e700 1 --2- 172.21.15.110:0/2446917754 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f73b4153290 0x7f73b40c7300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:38.986 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:38.980+0000 7f73b359e700 1 --2- 172.21.15.110:0/2446917754 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f73b40cc1a0 0x7f73b40c32e0 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:38.986 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:38.980+0000 7f73b359e700 1 --2- 172.21.15.110:0/2446917754 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f73b4069660 0x7f73b40c5bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:38.986 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:38.980+0000 7f73b359e700 1 -- 172.21.15.110:0/2446917754 >> 172.21.15.110:0/2446917754 conn(0x7f73b4067700 msgr2=0x7f73b4068f50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:30:38.986 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:38.980+0000 7f73b359e700 1 -- 172.21.15.110:0/2446917754 shutdown_connections 2023-10-06T03:30:38.986 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:38.980+0000 7f73b359e700 1 -- 172.21.15.110:0/2446917754 wait complete. 2023-10-06T03:30:38.986 INFO:teuthology.orchestra.run.smithi110.stderr:dumped all 2023-10-06T03:30:38.997 INFO:teuthology.orchestra.run.smithi110.stdout:{"pg_ready":true,"pg_map":{"version":74,"stamp":"2023-10-06T03:30:38.671012+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592674,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2429,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":210,"acting":210,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":83,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2399588,"kb_used_data":4900,"kb_used_omap":0,"kb_used_meta":2394624,"kb_avail":752575132,"statfs":{"total":773094113280,"available":770636935168,"internally_reserved":0,"allocated":5017600,"data_stored":2865033,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2452094976},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":33,"apply_latency_ms":33,"commit_latency_ns":33000000,"apply_latency_ns":33000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2306,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.002421"},"pg_stats":[{"pgid":"7.3f","version":"0'0","reported_seq":9,"reported_epoch":41,"state":"active+clean","last_fresh":"2023-10-06T03:30:34.680322+0000","last_change":"2023-10-06T03:30:34.677485+0000","last_active":"2023-10-06T03:30:34.680322+0000","last_peered":"2023-10-06T03:30:34.680322+0000","last_clean":"2023-10-06T03:30:34.680322+0000","last_became_active":"2023-10-06T03:30:34.677054+0000","last_became_peered":"2023-10-06T03:30:34.677054+0000","last_unstale":"2023-10-06T03:30:34.680322+0000","last_undegraded":"2023-10-06T03:30:34.680322+0000","last_fullsized":"2023-10-06T03:30:34.680322+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:30:33.627150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:30:33.627150+0000","last_clean_scrub_stamp":"2023-10-06T03:30:33.627150+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T07:57:54.285214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.3e","version":"0'0","reported_seq":9,"reported_epoch":41,"state":"active+clean","last_fresh":"2023-10-06T03:30:34.695248+0000","last_change":"2023-10-06T03:30:34.692207+0000","last_active":"2023-10-06T03:30:34.695248+0000","last_peered":"2023-10-06T03:30:34.695248+0000","last_clean":"2023-10-06T03:30:34.695248+0000","last_became_active":"2023-10-06T03:30:34.691919+0000","last_became_peered":"2023-10-06T03:30:34.691919+0000","last_unstale":"2023-10-06T03:30:34.695248+0000","last_undegraded":"2023-10-06T03:30:34.695248+0000","last_fullsized":"2023-10-06T03:30:34.695248+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:30:33.627150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:30:33.627150+0000","last_clean_scrub_stamp":"2023-10-06T03:30:33.627150+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T07:31:05.945597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3d","version":"0'0","reported_seq":9,"reported_epoch":41,"state":"active+clean","last_fresh":"2023-10-06T03:30:34.674268+0000","last_change":"2023-10-06T03:30:34.665857+0000","last_active":"2023-10-06T03:30:34.674268+0000","last_peered":"2023-10-06T03:30:34.674268+0000","last_clean":"2023-10-06T03:30:34.674268+0000","last_became_active":"2023-10-06T03:30:34.665551+0000","last_became_peered":"2023-10-06T03:30:34.665551+0000","last_unstale":"2023-10-06T03:30:34.674268+0000","last_undegraded":"2023-10-06T03:30:34.674268+0000","last_fullsized":"2023-10-06T03:30:34.674268+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:30:33.627150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:30:33.627150+0000","last_clean_scrub_stamp":"2023-10-06T03:30:33.627150+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T09:55:05.791053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.3c","version":"0'0","reported_seq":9,"reported_epoch":41,"state":"active+clean","last_fresh":"2023-10-06T03:30:34.690669+0000","last_change":"2023-10-06T03:30:34.678826+0000","last_active":"2023-10-06T03:30:34.690669+0000","last_peered":"2023-10-06T03:30:34.690669+0000","last_clean":"2023-10-06T03:30:34.690669+0000","last_became_active":"2023-10-06T03:30:34.678199+0000","last_became_peered":"2023-10-06T03:30:34.678199+0000","last_unstale":"2023-10-06T03:30:34.690669+0000","last_undegraded":"2023-10-06T03:30:34.690669+0000","last_fullsized":"2023-10-06T03:30:34.690669+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:30:33.627150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:30:33.627150+0000","last_clean_scrub_stamp":"2023-10-06T03:30:33.627150+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T11:47:51.243155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3b","version":"0'0","reported_seq":9,"reported_epoch":41,"state":"active+clean","last_fresh":"2023-10-06T03:30:34.695285+0000","last_change":"2023-10-06T03:30:34.692221+0000","last_active":"2023-10-06T03:30:34.695285+0000","last_peered":"2023-10-06T03:30:34.695285+0000","last_clean":"2023-10-06T03:30:34.695285+0000","last_became_active":"2023-10-06T03:30:34.691871+0000","last_became_peered":"2023-10-06T03:30:34.691871+0000","last_unstale":"2023-10-06T03:30:34.695285+0000","last_undegraded":"2023-10-06T03:30:34.695285+0000","last_fullsized":"2023-10-06T03:30:34.695285+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:30:33.627150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:30:33.627150+0000","last_clean_scrub_stamp":"2023-10-06T03:30:33.627150+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T13:17:11.416397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3a","version":"0'0","reported_seq":9,"reported_epoch":41,"state":"active+clean","last_fresh":"2023-10-06T03:30:34.668332+0000","last_change":"2023-10-06T03:30:34.666278+0000","last_active":"2023-10-06T03:30:34.668332+0000","last_peered":"2023-10-06T03:30:34.668332+0000","last_clean":"2023-10-06T03:30:34.668332+0000","last_became_active":"2023-10-06T03:30:34.665206+0000","last_became_peered":"2023-10-06T03:30:34.665206+0000","last_unstale":"2023-10-06T03:30:34.668332+0000","last_undegraded":"2023-10-06T03:30:34.668332+0000","last_fullsized":"2023-10-06T03:30:34.668332+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:30:33.627150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:30:33.627150+0000","last_clean_scrub_stamp":"2023-10-06T03:30:33.627150+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T11:01:12.279639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.39","version":"0'0","reported_seq":9,"reported_epoch":41,"state":"active+clean","last_fresh":"2023-10-06T03:30:34.679102+0000","last_change":"2023-10-06T03:30:34.675337+0000","last_active":"2023-10-06T03:30:34.679102+0000","last_peered":"2023-10-06T03:30:34.679102+0000","last_clean":"2023-10-06T03:30:34.679102+0000","last_became_active":"2023-10-06T03:30:34.675044+0000","last_became_peered":"2023-10-06T03:30:34.675044+0000","last_unstale":"2023-10-06T03:30:34.679102+0000","last_undegraded":"2023-10-06T03:30:34.679102+0000","last_fullsized":"2023-10-06T03:30:34.679102+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:30:33.627150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:30:33.627150+0000","last_clean_scrub_stamp":"2023-10-06T03:30:33.627150+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T13:17:11.416397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.38","version":"0'0","reported_seq":9,"reported_epoch":41,"state":"active+clean","last_fresh":"2023-10-06T03:30:34.691705+0000","last_change":"2023-10-06T03:30:34.679455+0000","last_active":"2023-10-06T03:30:34.691705+0000","last_peered":"2023-10-06T03:30:34.691705+0000","last_clean":"2023-10-06T03:30:34.691705+0000","last_became_active":"2023-10-06T03:30:34.678241+0000","last_became_peered":"2023-10-06T03:30:34.678241+0000","last_unstale":"2023-10-06T03:30:34.691705+0000","last_undegraded":"2023-10-06T03:30:34.691705+0000","last_fullsized":"2023-10-06T03:30:34.691705+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:30:33.627150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:30:33.627150+0000","last_clean_scrub_stamp":"2023-10-06T03:30:33.627150+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T04:24:20.424556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.37","version":"0'0","reported_seq":9,"reported_epoch":41,"state":"active+clean","last_fresh":"2023-10-06T03:30:34.690726+0000","last_change":"2023-10-06T03:30:34.678974+0000","last_active":"2023-10-06T03:30:34.690726+0000","last_peered":"2023-10-06T03:30:34.690726+0000","last_clean":"2023-10-06T03:30:34.690726+0000","last_became_active":"2023-10-06T03:30:34.678479+0000","last_became_peered":"2023-10-06T03:30:34.678479+0000","last_unstale":"2023-10-06T03:30:34.690726+0000","last_undegraded":"2023-10-06T03:30:34.690726+0000","last_fullsized":"2023-10-06T03:30:34.690726+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:30:33.627150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:30:33.627150+0000","last_clean_scrub_stamp":"2023-10-06T03:30:33.627150+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T14:02:35.591162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.36","version":"0'0","reported_seq":9,"reported_epoch":41,"state":"active+clean","last_fresh":"2023-10-06T03:30:34.690852+0000","last_change":"2023-10-06T03:30:34.679369+0000","last_active":"2023-10-06T03:30:34.690852+0000","last_peered":"2023-10-06T03:30:34.690852+0000","last_clean":"2023-10-06T03:30:34.690852+0000","last_became_active":"2023-10-06T03:30:34.678957+0000","last_became_peered":"2023-10-06T03:30:34.678957+0000","last_unstale":"2023-10-06T03:30:34.690852+0000","last_undegraded":"2023-10-06T03:30:34.690852+0000","last_fullsized":"2023-10-06T03:30:34.690852+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:30:33.627150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:30:33.627150+0000","last_clean_scrub_stamp":"2023-10-06T03:30:33.627150+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T14:58:13.432302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.35","version":"0'0","reported_seq":9,"reported_epoch":41,"state":"active+clean","last_fresh":"2023-10-06T03:30:34.690740+0000","last_change":"2023-10-06T03:30:34.678590+0000","last_active":"2023-10-06T03:30:34.690740+0000","last_peered":"2023-10-06T03:30:34.690740+0000","last_clean":"2023-10-06T03:30:34.690740+0000","last_became_active":"2023-10-06T03:30:34.675004+0000","last_became_peered":"2023-10-06T03:30:34.675004+0000","last_unstale":"2023-10-06T03:30:34.690740+0000","last_undegraded":"2023-10-06T03:30:34.690740+0000","last_fullsized":"2023-10-06T03:30:34.690740+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:30:33.627150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:30:33.627150+0000","last_clean_scrub_stamp":"2023-10-06T03:30:33.627150+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T06:03:52.528859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.34","version":"0'0","reported_seq":9,"reported_epoch":41,"state":"active+clean","last_fresh":"2023-10-06T03:30:34.676797+0000","last_change":"2023-10-06T03:30:34.675938+0000","last_active":"2023-10-06T03:30:34.676797+0000","last_peered":"2023-10-06T03:30:34.676797+0000","last_clean":"2023-10-06T03:30:34.676797+0000","last_became_active":"2023-10-06T03:30:34.675580+0000","last_became_peered":"2023-10-06T03:30:34.675580+0000","last_unstale":"2023-10-06T03:30:34.676797+0000","last_undegraded":"2023-10-06T03:30:34.676797+0000","last_fullsized":"2023-10-06T03:30:34.676797+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:30:33.627150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:30:33.627150+0000","last_clean_scrub_stamp":"2023-10-06T03:30:33.627150+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T04:36:13.727648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.33","version":"0'0","reported_seq":8,"reported_epoch":41,"state":"active+clean","last_fresh":"2023-10-06T03:30:34.669087+0000","last_change":"2023-10-06T03:30:34.669087+0000","last_active":"2023-10-06T03:30:34.669087+0000","last_peered":"2023-10-06T03:30:34.669087+0000","last_clean":"2023-10-06T03:30:34.669087+0000","last_became_active":"2023-10-06T03:30:34.668804+0000","last_became_peered":"2023-10-06T03:30:34.668804+0000","last_unstale":"2023-10-06T03:30:34.669087+0000","last_undegraded":"2023-10-06T03:30:34.669087+0000","last_fullsized":"2023-10-06T03:30:34.669087+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:30:33.627150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:30:33.627150+0000","last_clean_scrub_stamp":"2023-10-06T03:30:33.627150+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T10:56:57.565128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.32","version":"0'0","reported_seq":9,"reported_epoch":41,"state":"active+clean","last_fresh":"2023-10-06T03:30:34.692001+0000","last_change":"2023-10-06T03:30:34.684814+0000","last_active":"2023-10-06T03:30:34.692001+0000","last_peered":"2023-10-06T03:30:34.692001+0000","last_clean":"2023-10-06T03:30:34.692001+0000","last_became_active":"2023-10-06T03:30:34.678325+0000","last_became_peered":"2023-10-06T03:30:34.678325+0000","last_unstale":"2023-10-06T03:30:34.692001+0000","last_undegraded":"2023-10-06T03:30:34.692001+0000","last_fullsized":"2023-10-06T03:30:34.692001+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:30:33.627150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:30:33.627150+0000","last_clean_scrub_stamp":"2023-10-06T03:30:33.627150+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T12:09:15.937213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.31","version":"0'0","reported_seq":9,"reported_epoch":41,"state":"active+clean","last_fresh":"2023-10-06T03:30:34.668446+0000","last_change":"2023-10-06T03:30:34.657238+0000","last_active":"2023-10-06T03:30:34.668446+0000","last_peered":"2023-10-06T03:30:34.668446+0000","last_clean":"2023-10-06T03:30:34.668446+0000","last_became_active":"2023-10-06T03:30:34.656927+0000","last_became_peered":"2023-10-06T03:30:34.656927+0000","last_unstale":"2023-10-06T03:30:34.668446+0000","last_undegraded":"2023-10-06T03:30:34.668446+0000","last_fullsized":"2023-10-06T03:30:34.668446+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:30:33.627150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:30:33.627150+0000","last_clean_scrub_stamp":"2023-10-06T03:30:33.627150+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T09:06:23.239915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.30","version":"0'0","reported_seq":9,"reported_epoch":41,"state":"active+clean","last_fresh":"2023-10-06T03:30:34.695855+0000","last_change":"2023-10-06T03:30:34.693337+0000","last_active":"2023-10-06T03:30:34.695855+0000","last_peered":"2023-10-06T03:30:34.695855+0000","last_clean":"2023-10-06T03:30:34.695855+0000","last_became_active":"2023-10-06T03:30:34.693007+0000","last_became_peered":"2023-10-06T03:30:34.693007+0000","last_unstale":"2023-10-06T03:30:34.695855+0000","last_undegraded":"2023-10-06T03:30:34.695855+0000","last_fullsized":"2023-10-06T03:30:34.695855+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:30:33.627150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:30:33.627150+0000","last_clean_scrub_stamp":"2023-10-06T03:30:33.627150+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T09:15:01.820547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.2f","version":"0'0","reported_seq":9,"reported_epoch":41,"state":"active+clean","last_fresh":"2023-10-06T03:30:34.675136+0000","last_change":"2023-10-06T03:30:34.670796+0000","last_active":"2023-10-06T03:30:34.675136+0000","last_peered":"2023-10-06T03:30:34.675136+0000","last_clean":"2023-10-06T03:30:34.675136+0000","last_became_active":"2023-10-06T03:30:34.670490+0000","last_became_peered":"2023-10-06T03:30:34.670490+0000","last_unstale":"2023-10-06T03:30:34.675136+0000","last_undegraded":"2023-10-06T03:30:34.675136+0000","last_fullsized":"2023-10-06T03:30:34.675136+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:30:33.627150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:30:33.627150+0000","last_clean_scrub_stamp":"2023-10-06T03:30:33.627150+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T06:03:52.528859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2e","version":"0'0","reported_seq":9,"reported_epoch":41,"state":"active+clean","last_fresh":"2023-10-06T03:30:34.687472+0000","last_change":"2023-10-06T03:30:34.685889+0000","last_active":"2023-10-06T03:30:34.687472+0000","last_peered":"2023-10-06T03:30:34.687472+0000","last_clean":"2023-10-06T03:30:34.687472+0000","last_became_active":"2023-10-06T03:30:34.685564+0000","last_became_peered":"2023-10-06T03:30:34.685564+0000","last_unstale":"2023-10-06T03:30:34.687472+0000","last_undegraded":"2023-10-06T03:30:34.687472+0000","last_fullsized":"2023-10-06T03:30:34.687472+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:30:33.627150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:30:33.627150+0000","last_clean_scrub_stamp":"2023-10-06T03:30:33.627150+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T10:12:42.343138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2d","version":"0'0","reported_seq":9,"reported_epoch":41,"state":"active+clean","last_fresh":"2023-10-06T03:30:34.695415+0000","last_change":"2023-10-06T03:30:34.692098+0000","last_active":"2023-10-06T03:30:34.695415+0000","last_peered":"2023-10-06T03:30:34.695415+0000","last_clean":"2023-10-06T03:30:34.695415+0000","last_became_active":"2023-10-06T03:30:34.691749+0000","last_became_peered":"2023-10-06T03:30:34.691749+0000","last_unstale":"2023-10-06T03:30:34.695415+0000","last_undegraded":"2023-10-06T03:30:34.695415+0000","last_fullsized":"2023-10-06T03:30:34.695415+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:30:33.627150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:30:33.627150+0000","last_clean_scrub_stamp":"2023-10-06T03:30:33.627150+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T15:21:55.713021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.2c","version":"0'0","reported_seq":8,"reported_epoch":41,"state":"active+clean","last_fresh":"2023-10-06T03:30:34.685955+0000","last_change":"2023-10-06T03:30:34.685955+0000","last_active":"2023-10-06T03:30:34.685955+0000","last_peered":"2023-10-06T03:30:34.685955+0000","last_clean":"2023-10-06T03:30:34.685955+0000","last_became_active":"2023-10-06T03:30:34.685624+0000","last_became_peered":"2023-10-06T03:30:34.685624+0000","last_unstale":"2023-10-06T03:30:34.685955+0000","last_undegraded":"2023-10-06T03:30:34.685955+0000","last_fullsized":"2023-10-06T03:30:34.685955+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:30:33.627150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:30:33.627150+0000","last_clean_scrub_stamp":"2023-10-06T03:30:33.627150+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T10:53:07.184282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.2b","version":"0'0","reported_seq":9,"reported_epoch":41,"state":"active+clean","last_fresh":"2023-10-06T03:30:34.673513+0000","last_change":"2023-10-06T03:30:34.668467+0000","last_active":"2023-10-06T03:30:34.673513+0000","last_peered":"2023-10-06T03:30:34.673513+0000","last_clean":"2023-10-06T03:30:34.673513+0000","last_became_active":"2023-10-06T03:30:34.668171+0000","last_became_peered":"2023-10-06T03:30:34.668171+0000","last_unstale":"2023-10-06T03:30:34.673513+0000","last_undegraded":"2023-10-06T03:30:34.673513+0000","last_fullsized":"2023-10-06T03:30:34.673513+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:30:33.627150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:30:33.627150+0000","last_clean_scrub_stamp":"2023-10-06T03:30:33.627150+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T09:19:27.526549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.2a","version":"0'0","reported_seq":9,"reported_epoch":41,"state":"active+clean","last_fresh":"2023-10-06T03:30:34.690810+0000","last_change":"2023-10-06T03:30:34.679206+0000","last_active":"2023-10-06T03:30:34.690810+0000","last_peered":"2023-10-06T03:30:34.690810+0000","last_clean":"2023-10-06T03:30:34.690810+0000","last_became_active":"2023-10-06T03:30:34.675195+0000","last_became_peered":"2023-10-06T03:30:34.675195+0000","last_unstale":"2023-10-06T03:30:34.690810+0000","last_undegraded":"2023-10-06T03:30:34.690810+0000","last_fullsized":"2023-10-06T03:30:34.690810+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:30:33.627150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:30:33.627150+0000","last_clean_scrub_stamp":"2023-10-06T03:30:33.627150+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T09:55:05.791053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.29","version":"0'0","reported_seq":9,"reported_epoch":41,"state":"active+clean","last_fresh":"2023-10-06T03:30:34.678545+0000","last_change":"2023-10-06T03:30:34.675324+0000","last_active":"2023-10-06T03:30:34.678545+0000","last_peered":"2023-10-06T03:30:34.678545+0000","last_clean":"2023-10-06T03:30:34.678545+0000","last_became_active":"2023-10-06T03:30:34.675010+0000","last_became_peered":"2023-10-06T03:30:34.675010+0000","last_unstale":"2023-10-06T03:30:34.678545+0000","last_undegraded":"2023-10-06T03:30:34.678545+0000","last_fullsized":"2023-10-06T03:30:34.678545+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:30:33.627150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:30:33.627150+0000","last_clean_scrub_stamp":"2023-10-06T03:30:33.627150+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T07:39:23.441430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.28","version":"0'0","reported_seq":9,"reported_epoch":41,"state":"active+clean","last_fresh":"2023-10-06T03:30:34.675185+0000","last_change":"2023-10-06T03:30:34.668299+0000","last_active":"2023-10-06T03:30:34.675185+0000","last_peered":"2023-10-06T03:30:34.675185+0000","last_clean":"2023-10-06T03:30:34.675185+0000","last_became_active":"2023-10-06T03:30:34.668028+0000","last_became_peered":"2023-10-06T03:30:34.668028+0000","last_unstale":"2023-10-06T03:30:34.675185+0000","last_undegraded":"2023-10-06T03:30:34.675185+0000","last_fullsized":"2023-10-06T03:30:34.675185+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:30:33.627150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:30:33.627150+0000","last_clean_scrub_stamp":"2023-10-06T03:30:33.627150+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T10:53:07.184282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.27","version":"0'0","reported_seq":9,"reported_epoch":41,"state":"active+clean","last_fresh":"2023-10-06T03:30:34.686194+0000","last_change":"2023-10-06T03:30:34.681790+0000","last_active":"2023-10-06T03:30:34.686194+0000","last_peered":"2023-10-06T03:30:34.686194+0000","last_clean":"2023-10-06T03:30:34.686194+0000","last_became_active":"2023-10-06T03:30:34.681477+0000","last_became_peered":"2023-10-06T03:30:34.681477+0000","last_unstale":"2023-10-06T03:30:34.686194+0000","last_undegraded":"2023-10-06T03:30:34.686194+0000","last_fullsized":"2023-10-06T03:30:34.686194+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:30:33.627150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:30:33.627150+0000","last_clean_scrub_stamp":"2023-10-06T03:30:33.627150+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T12:27:36.823538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.26","version":"0'0","reported_seq":8,"reported_epoch":41,"state":"active+clean","last_fresh":"2023-10-06T03:30:34.669291+0000","last_change":"2023-10-06T03:30:34.669291+0000","last_active":"2023-10-06T03:30:34.669291+0000","last_peered":"2023-10-06T03:30:34.669291+0000","last_clean":"2023-10-06T03:30:34.669291+0000","last_became_active":"2023-10-06T03:30:34.668770+0000","last_became_peered":"2023-10-06T03:30:34.668770+0000","last_unstale":"2023-10-06T03:30:34.669291+0000","last_undegraded":"2023-10-06T03:30:34.669291+0000","last_fullsized":"2023-10-06T03:30:34.669291+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:30:33.627150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:30:33.627150+0000","last_clean_scrub_stamp":"2023-10-06T03:30:33.627150+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T08:26:46.586078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.25","version":"0'0","reported_seq":9,"reported_epoch":41,"state":"active+clean","last_fresh":"2023-10-06T03:30:34.674291+0000","last_change":"2023-10-06T03:30:34.667433+0000","last_active":"2023-10-06T03:30:34.674291+0000","last_peered":"2023-10-06T03:30:34.674291+0000","last_clean":"2023-10-06T03:30:34.674291+0000","last_became_active":"2023-10-06T03:30:34.667114+0000","last_became_peered":"2023-10-06T03:30:34.667114+0000","last_unstale":"2023-10-06T03:30:34.674291+0000","last_undegraded":"2023-10-06T03:30:34.674291+0000","last_fullsized":"2023-10-06T03:30:34.674291+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:30:33.627150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:30:33.627150+0000","last_clean_scrub_stamp":"2023-10-06T03:30:33.627150+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T05:04:39.755028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.24","version":"0'0","reported_seq":9,"reported_epoch":41,"state":"active+clean","last_fresh":"2023-10-06T03:30:34.675600+0000","last_change":"2023-10-06T03:30:34.671792+0000","last_active":"2023-10-06T03:30:34.675600+0000","last_peered":"2023-10-06T03:30:34.675600+0000","last_clean":"2023-10-06T03:30:34.675600+0000","last_became_active":"2023-10-06T03:30:34.671445+0000","last_became_peered":"2023-10-06T03:30:34.671445+0000","last_unstale":"2023-10-06T03:30:34.675600+0000","last_undegraded":"2023-10-06T03:30:34.675600+0000","last_fullsized":"2023-10-06T03:30:34.675600+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:30:33.627150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:30:33.627150+0000","last_clean_scrub_stamp":"2023-10-06T03:30:33.627150+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T04:31:28.001115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.23","version":"0'0","reported_seq":9,"reported_epoch":41,"state":"active+clean","last_fresh":"2023-10-06T03:30:34.678528+0000","last_change":"2023-10-06T03:30:34.675283+0000","last_active":"2023-10-06T03:30:34.678528+0000","last_peered":"2023-10-06T03:30:34.678528+0000","last_clean":"2023-10-06T03:30:34.678528+0000","last_became_active":"2023-10-06T03:30:34.674959+0000","last_became_peered":"2023-10-06T03:30:34.674959+0000","last_unstale":"2023-10-06T03:30:34.678528+0000","last_undegraded":"2023-10-06T03:30:34.678528+0000","last_fullsized":"2023-10-06T03:30:34.678528+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:30:33.627150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:30:33.627150+0000","last_clean_scrub_stamp":"2023-10-06T03:30:33.627150+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T14:37:19.126995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.22","version":"0'0","reported_seq":9,"reported_epoch":41,"state":"active+clean","last_fresh":"2023-10-06T03:30:34.675055+0000","last_change":"2023-10-06T03:30:34.670953+0000","last_active":"2023-10-06T03:30:34.675055+0000","last_peered":"2023-10-06T03:30:34.675055+0000","last_clean":"2023-10-06T03:30:34.675055+0000","last_became_active":"2023-10-06T03:30:34.670643+0000","last_became_peered":"2023-10-06T03:30:34.670643+0000","last_unstale":"2023-10-06T03:30:34.675055+0000","last_undegraded":"2023-10-06T03:30:34.675055+0000","last_fullsized":"2023-10-06T03:30:34.675055+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:30:33.627150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:30:33.627150+0000","last_clean_scrub_stamp":"2023-10-06T03:30:33.627150+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T08:18:07.397765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.21","version":"0'0","reported_seq":9,"reported_epoch":41,"state":"active+clean","last_fresh":"2023-10-06T03:30:34.673558+0000","last_change":"2023-10-06T03:30:34.669467+0000","last_active":"2023-10-06T03:30:34.673558+0000","last_peered":"2023-10-06T03:30:34.673558+0000","last_clean":"2023-10-06T03:30:34.673558+0000","last_became_active":"2023-10-06T03:30:34.668221+0000","last_became_peered":"2023-10-06T03:30:34.668221+0000","last_unstale":"2023-10-06T03:30:34.673558+0000","last_undegraded":"2023-10-06T03:30:34.673558+0000","last_fullsized":"2023-10-06T03:30:34.673558+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:30:33.627150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:30:33.627150+0000","last_clean_scrub_stamp":"2023-10-06T03:30:33.627150+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T08:10:34.526977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.20","version":"0'0","reported_seq":8,"reported_epoch":41,"state":"active+clean","last_fresh":"2023-10-06T03:30:34.671765+0000","last_change":"2023-10-06T03:30:34.671765+0000","last_active":"2023-10-06T03:30:34.671765+0000","last_peered":"2023-10-06T03:30:34.671765+0000","last_clean":"2023-10-06T03:30:34.671765+0000","last_became_active":"2023-10-06T03:30:34.671477+0000","last_became_peered":"2023-10-06T03:30:34.671477+0000","last_unstale":"2023-10-06T03:30:34.671765+0000","last_undegraded":"2023-10-06T03:30:34.671765+0000","last_fullsized":"2023-10-06T03:30:34.671765+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:30:33.627150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:30:33.627150+0000","last_clean_scrub_stamp":"2023-10-06T03:30:33.627150+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T11:13:30.805084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.1e","version":"0'0","reported_seq":8,"reported_epoch":41,"state":"active+clean","last_fresh":"2023-10-06T03:30:34.666944+0000","last_change":"2023-10-06T03:30:34.666944+0000","last_active":"2023-10-06T03:30:34.666944+0000","last_peered":"2023-10-06T03:30:34.666944+0000","last_clean":"2023-10-06T03:30:34.666944+0000","last_became_active":"2023-10-06T03:30:34.666627+0000","last_became_peered":"2023-10-06T03:30:34.666627+0000","last_unstale":"2023-10-06T03:30:34.666944+0000","last_undegraded":"2023-10-06T03:30:34.666944+0000","last_fullsized":"2023-10-06T03:30:34.666944+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:30:33.627150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:30:33.627150+0000","last_clean_scrub_stamp":"2023-10-06T03:30:33.627150+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T11:46:04.452767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"41'4","reported_seq":14,"reported_epoch":41,"state":"active+clean","last_fresh":"2023-10-06T03:30:34.688144+0000","last_change":"2023-10-06T03:30:33.667669+0000","last_active":"2023-10-06T03:30:34.688144+0000","last_peered":"2023-10-06T03:30:34.688144+0000","last_clean":"2023-10-06T03:30:34.688144+0000","last_became_active":"2023-10-06T03:30:33.667335+0000","last_became_peered":"2023-10-06T03:30:33.667335+0000","last_unstale":"2023-10-06T03:30:34.688144+0000","last_undegraded":"2023-10-06T03:30:34.688144+0000","last_fullsized":"2023-10-06T03:30:34.688144+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:30:32.615612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:30:32.615612+0000","last_clean_scrub_stamp":"2023-10-06T03:30:32.615612+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T07:00:29.333955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":554,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1f","version":"0'0","reported_seq":9,"reported_epoch":41,"state":"active+clean","last_fresh":"2023-10-06T03:30:34.679479+0000","last_change":"2023-10-06T03:30:34.676314+0000","last_active":"2023-10-06T03:30:34.679479+0000","last_peered":"2023-10-06T03:30:34.679479+0000","last_clean":"2023-10-06T03:30:34.679479+0000","last_became_active":"2023-10-06T03:30:34.676004+0000","last_became_peered":"2023-10-06T03:30:34.676004+0000","last_unstale":"2023-10-06T03:30:34.679479+0000","last_undegraded":"2023-10-06T03:30:34.679479+0000","last_fullsized":"2023-10-06T03:30:34.679479+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:30:33.627150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:30:33.627150+0000","last_clean_scrub_stamp":"2023-10-06T03:30:33.627150+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T10:45:45.777985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1e","version":"41'1","reported_seq":11,"reported_epoch":41,"state":"active+clean","last_fresh":"2023-10-06T03:30:34.686508+0000","last_change":"2023-10-06T03:30:33.663845+0000","last_active":"2023-10-06T03:30:34.686508+0000","last_peered":"2023-10-06T03:30:34.686508+0000","last_clean":"2023-10-06T03:30:34.686508+0000","last_became_active":"2023-10-06T03:30:33.663571+0000","last_became_peered":"2023-10-06T03:30:33.663571+0000","last_unstale":"2023-10-06T03:30:34.686508+0000","last_undegraded":"2023-10-06T03:30:34.686508+0000","last_fullsized":"2023-10-06T03:30:34.686508+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:30:32.615612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:30:32.615612+0000","last_clean_scrub_stamp":"2023-10-06T03:30:32.615612+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T13:17:10.404859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1c","version":"0'0","reported_seq":9,"reported_epoch":41,"state":"active+clean","last_fresh":"2023-10-06T03:30:34.672367+0000","last_change":"2023-10-06T03:30:34.670214+0000","last_active":"2023-10-06T03:30:34.672367+0000","last_peered":"2023-10-06T03:30:34.672367+0000","last_clean":"2023-10-06T03:30:34.672367+0000","last_became_active":"2023-10-06T03:30:34.669883+0000","last_became_peered":"2023-10-06T03:30:34.669883+0000","last_unstale":"2023-10-06T03:30:34.672367+0000","last_undegraded":"2023-10-06T03:30:34.672367+0000","last_fullsized":"2023-10-06T03:30:34.672367+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:30:33.627150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:30:33.627150+0000","last_clean_scrub_stamp":"2023-10-06T03:30:33.627150+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T09:42:24.260904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1d","version":"41'2","reported_seq":11,"reported_epoch":41,"state":"active+clean","last_fresh":"2023-10-06T03:30:34.684098+0000","last_change":"2023-10-06T03:30:33.670694+0000","last_active":"2023-10-06T03:30:34.684098+0000","last_peered":"2023-10-06T03:30:34.684098+0000","last_clean":"2023-10-06T03:30:34.684098+0000","last_became_active":"2023-10-06T03:30:33.670328+0000","last_became_peered":"2023-10-06T03:30:33.670328+0000","last_unstale":"2023-10-06T03:30:34.684098+0000","last_undegraded":"2023-10-06T03:30:34.684098+0000","last_fullsized":"2023-10-06T03:30:34.684098+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:30:32.615612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:30:32.615612+0000","last_clean_scrub_stamp":"2023-10-06T03:30:32.615612+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T10:27:44.696525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1d","version":"0'0","reported_seq":8,"reported_epoch":41,"state":"active+clean","last_fresh":"2023-10-06T03:30:34.675677+0000","last_change":"2023-10-06T03:30:34.675677+0000","last_active":"2023-10-06T03:30:34.675677+0000","last_peered":"2023-10-06T03:30:34.675677+0000","last_clean":"2023-10-06T03:30:34.675677+0000","last_became_active":"2023-10-06T03:30:34.675303+0000","last_became_peered":"2023-10-06T03:30:34.675303+0000","last_unstale":"2023-10-06T03:30:34.675677+0000","last_undegraded":"2023-10-06T03:30:34.675677+0000","last_fullsized":"2023-10-06T03:30:34.675677+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:30:33.627150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:30:33.627150+0000","last_clean_scrub_stamp":"2023-10-06T03:30:33.627150+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T11:10:52.361888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1c","version":"41'2","reported_seq":12,"reported_epoch":41,"state":"active+clean","last_fresh":"2023-10-06T03:30:34.686339+0000","last_change":"2023-10-06T03:30:33.672742+0000","last_active":"2023-10-06T03:30:34.686339+0000","last_peered":"2023-10-06T03:30:34.686339+0000","last_clean":"2023-10-06T03:30:34.686339+0000","last_became_active":"2023-10-06T03:30:33.672478+0000","last_became_peered":"2023-10-06T03:30:33.672478+0000","last_unstale":"2023-10-06T03:30:34.686339+0000","last_undegraded":"2023-10-06T03:30:34.686339+0000","last_fullsized":"2023-10-06T03:30:34.686339+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:30:32.615612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:30:32.615612+0000","last_clean_scrub_stamp":"2023-10-06T03:30:32.615612+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T11:13:29.793546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1a","version":"0'0","reported_seq":9,"reported_epoch":41,"state":"active+clean","last_fresh":"2023-10-06T03:30:34.679252+0000","last_change":"2023-10-06T03:30:34.676274+0000","last_active":"2023-10-06T03:30:34.679252+0000","last_peered":"2023-10-06T03:30:34.679252+0000","last_clean":"2023-10-06T03:30:34.679252+0000","last_became_active":"2023-10-06T03:30:34.675941+0000","last_became_peered":"2023-10-06T03:30:34.675941+0000","last_unstale":"2023-10-06T03:30:34.679252+0000","last_undegraded":"2023-10-06T03:30:34.679252+0000","last_fullsized":"2023-10-06T03:30:34.679252+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:30:33.627150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:30:33.627150+0000","last_clean_scrub_stamp":"2023-10-06T03:30:33.627150+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T14:02:35.591162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"41'1","reported_seq":11,"reported_epoch":41,"state":"active+clean","last_fresh":"2023-10-06T03:30:34.687069+0000","last_change":"2023-10-06T03:30:33.667614+0000","last_active":"2023-10-06T03:30:34.687069+0000","last_peered":"2023-10-06T03:30:34.687069+0000","last_clean":"2023-10-06T03:30:34.687069+0000","last_became_active":"2023-10-06T03:30:33.665468+0000","last_became_peered":"2023-10-06T03:30:33.665468+0000","last_unstale":"2023-10-06T03:30:34.687069+0000","last_undegraded":"2023-10-06T03:30:34.687069+0000","last_fullsized":"2023-10-06T03:30:34.687069+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:30:32.615612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:30:32.615612+0000","last_clean_scrub_stamp":"2023-10-06T03:30:32.615612+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T11:17:50.406093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1b","version":"0'0","reported_seq":9,"reported_epoch":41,"state":"active+clean","last_fresh":"2023-10-06T03:30:34.690894+0000","last_change":"2023-10-06T03:30:34.679564+0000","last_active":"2023-10-06T03:30:34.690894+0000","last_peered":"2023-10-06T03:30:34.690894+0000","last_clean":"2023-10-06T03:30:34.690894+0000","last_became_active":"2023-10-06T03:30:34.678029+0000","last_became_peered":"2023-10-06T03:30:34.678029+0000","last_unstale":"2023-10-06T03:30:34.690894+0000","last_undegraded":"2023-10-06T03:30:34.690894+0000","last_fullsized":"2023-10-06T03:30:34.690894+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:30:33.627150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:30:33.627150+0000","last_clean_scrub_stamp":"2023-10-06T03:30:33.627150+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T04:44:04.202063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1a","version":"41'1","reported_seq":11,"reported_epoch":41,"state":"active+clean","last_fresh":"2023-10-06T03:30:34.686405+0000","last_change":"2023-10-06T03:30:33.667834+0000","last_active":"2023-10-06T03:30:34.686405+0000","last_peered":"2023-10-06T03:30:34.686405+0000","last_clean":"2023-10-06T03:30:34.686405+0000","last_became_active":"2023-10-06T03:30:33.667480+0000","last_became_peered":"2023-10-06T03:30:33.667480+0000","last_unstale":"2023-10-06T03:30:34.686405+0000","last_undegraded":"2023-10-06T03:30:34.686405+0000","last_fullsized":"2023-10-06T03:30:34.686405+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:30:32.615612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:30:32.615612+0000","last_clean_scrub_stamp":"2023-10-06T03:30:32.615612+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T07:31:04.934058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":9,"reported_epoch":41,"state":"active+clean","last_fresh":"2023-10-06T03:30:34.691690+0000","last_change":"2023-10-06T03:30:34.679090+0000","last_active":"2023-10-06T03:30:34.691690+0000","last_peered":"2023-10-06T03:30:34.691690+0000","last_clean":"2023-10-06T03:30:34.691690+0000","last_became_active":"2023-10-06T03:30:34.675165+0000","last_became_peered":"2023-10-06T03:30:34.675165+0000","last_unstale":"2023-10-06T03:30:34.691690+0000","last_undegraded":"2023-10-06T03:30:34.691690+0000","last_fullsized":"2023-10-06T03:30:34.691690+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:30:33.627150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:30:33.627150+0000","last_clean_scrub_stamp":"2023-10-06T03:30:33.627150+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T07:31:49.494324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":10,"reported_epoch":41,"state":"active+clean","last_fresh":"2023-10-06T03:30:34.659402+0000","last_change":"2023-10-06T03:30:33.668832+0000","last_active":"2023-10-06T03:30:34.659402+0000","last_peered":"2023-10-06T03:30:34.659402+0000","last_clean":"2023-10-06T03:30:34.659402+0000","last_became_active":"2023-10-06T03:30:33.668454+0000","last_became_peered":"2023-10-06T03:30:33.668454+0000","last_unstale":"2023-10-06T03:30:34.659402+0000","last_undegraded":"2023-10-06T03:30:34.659402+0000","last_fullsized":"2023-10-06T03:30:34.659402+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:30:32.615612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:30:32.615612+0000","last_clean_scrub_stamp":"2023-10-06T03:30:32.615612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T11:13:29.793546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":9,"reported_epoch":41,"state":"active+clean","last_fresh":"2023-10-06T03:30:34.679456+0000","last_change":"2023-10-06T03:30:34.675926+0000","last_active":"2023-10-06T03:30:34.679456+0000","last_peered":"2023-10-06T03:30:34.679456+0000","last_clean":"2023-10-06T03:30:34.679456+0000","last_became_active":"2023-10-06T03:30:34.675583+0000","last_became_peered":"2023-10-06T03:30:34.675583+0000","last_unstale":"2023-10-06T03:30:34.679456+0000","last_undegraded":"2023-10-06T03:30:34.679456+0000","last_fullsized":"2023-10-06T03:30:34.679456+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:30:33.627150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:30:33.627150+0000","last_clean_scrub_stamp":"2023-10-06T03:30:33.627150+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T11:23:03.827062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":10,"reported_epoch":41,"state":"active+clean","last_fresh":"2023-10-06T03:30:34.657394+0000","last_change":"2023-10-06T03:30:33.661851+0000","last_active":"2023-10-06T03:30:34.657394+0000","last_peered":"2023-10-06T03:30:34.657394+0000","last_clean":"2023-10-06T03:30:34.657394+0000","last_became_active":"2023-10-06T03:30:33.661562+0000","last_became_peered":"2023-10-06T03:30:33.661562+0000","last_unstale":"2023-10-06T03:30:34.657394+0000","last_undegraded":"2023-10-06T03:30:34.657394+0000","last_fullsized":"2023-10-06T03:30:34.657394+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:30:32.615612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:30:32.615612+0000","last_clean_scrub_stamp":"2023-10-06T03:30:32.615612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T07:57:53.273675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":9,"reported_epoch":41,"state":"active+clean","last_fresh":"2023-10-06T03:30:34.664821+0000","last_change":"2023-10-06T03:30:34.661077+0000","last_active":"2023-10-06T03:30:34.664821+0000","last_peered":"2023-10-06T03:30:34.664821+0000","last_clean":"2023-10-06T03:30:34.664821+0000","last_became_active":"2023-10-06T03:30:34.660728+0000","last_became_peered":"2023-10-06T03:30:34.660728+0000","last_unstale":"2023-10-06T03:30:34.664821+0000","last_undegraded":"2023-10-06T03:30:34.664821+0000","last_fullsized":"2023-10-06T03:30:34.664821+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:30:33.627150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:30:33.627150+0000","last_clean_scrub_stamp":"2023-10-06T03:30:33.627150+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T12:09:15.937213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":10,"reported_epoch":41,"state":"active+clean","last_fresh":"2023-10-06T03:30:34.654774+0000","last_change":"2023-10-06T03:30:33.664705+0000","last_active":"2023-10-06T03:30:34.654774+0000","last_peered":"2023-10-06T03:30:34.654774+0000","last_clean":"2023-10-06T03:30:34.654774+0000","last_became_active":"2023-10-06T03:30:33.664401+0000","last_became_peered":"2023-10-06T03:30:33.664401+0000","last_unstale":"2023-10-06T03:30:34.654774+0000","last_undegraded":"2023-10-06T03:30:34.654774+0000","last_fullsized":"2023-10-06T03:30:34.654774+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:30:32.615612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:30:32.615612+0000","last_clean_scrub_stamp":"2023-10-06T03:30:32.615612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T07:57:53.273675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":9,"reported_epoch":41,"state":"active+clean","last_fresh":"2023-10-06T03:30:34.677621+0000","last_change":"2023-10-06T03:30:34.670864+0000","last_active":"2023-10-06T03:30:34.677621+0000","last_peered":"2023-10-06T03:30:34.677621+0000","last_clean":"2023-10-06T03:30:34.677621+0000","last_became_active":"2023-10-06T03:30:34.670594+0000","last_became_peered":"2023-10-06T03:30:34.670594+0000","last_unstale":"2023-10-06T03:30:34.677621+0000","last_undegraded":"2023-10-06T03:30:34.677621+0000","last_fullsized":"2023-10-06T03:30:34.677621+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:30:33.627150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:30:33.627150+0000","last_clean_scrub_stamp":"2023-10-06T03:30:33.627150+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T07:31:05.945597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":10,"reported_epoch":41,"state":"active+clean","last_fresh":"2023-10-06T03:30:34.659465+0000","last_change":"2023-10-06T03:30:33.666203+0000","last_active":"2023-10-06T03:30:34.659465+0000","last_peered":"2023-10-06T03:30:34.659465+0000","last_clean":"2023-10-06T03:30:34.659465+0000","last_became_active":"2023-10-06T03:30:33.665849+0000","last_became_peered":"2023-10-06T03:30:33.665849+0000","last_unstale":"2023-10-06T03:30:34.659465+0000","last_undegraded":"2023-10-06T03:30:34.659465+0000","last_fullsized":"2023-10-06T03:30:34.659465+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:30:32.615612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:30:32.615612+0000","last_clean_scrub_stamp":"2023-10-06T03:30:32.615612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T04:56:39.607886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.14","version":"0'0","reported_seq":9,"reported_epoch":41,"state":"active+clean","last_fresh":"2023-10-06T03:30:34.667526+0000","last_change":"2023-10-06T03:30:34.664096+0000","last_active":"2023-10-06T03:30:34.667526+0000","last_peered":"2023-10-06T03:30:34.667526+0000","last_clean":"2023-10-06T03:30:34.667526+0000","last_became_active":"2023-10-06T03:30:34.663768+0000","last_became_peered":"2023-10-06T03:30:34.663768+0000","last_unstale":"2023-10-06T03:30:34.667526+0000","last_undegraded":"2023-10-06T03:30:34.667526+0000","last_fullsized":"2023-10-06T03:30:34.667526+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:30:33.627150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:30:33.627150+0000","last_clean_scrub_stamp":"2023-10-06T03:30:33.627150+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T04:59:46.261266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"41'3","reported_seq":13,"reported_epoch":41,"state":"active+clean","last_fresh":"2023-10-06T03:30:34.686642+0000","last_change":"2023-10-06T03:30:33.673407+0000","last_active":"2023-10-06T03:30:34.686642+0000","last_peered":"2023-10-06T03:30:34.686642+0000","last_clean":"2023-10-06T03:30:34.686642+0000","last_became_active":"2023-10-06T03:30:33.673118+0000","last_became_peered":"2023-10-06T03:30:33.673118+0000","last_unstale":"2023-10-06T03:30:34.686642+0000","last_undegraded":"2023-10-06T03:30:34.686642+0000","last_fullsized":"2023-10-06T03:30:34.686642+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:30:32.615612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:30:32.615612+0000","last_clean_scrub_stamp":"2023-10-06T03:30:32.615612+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T11:23:02.815524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.15","version":"0'0","reported_seq":9,"reported_epoch":41,"state":"active+clean","last_fresh":"2023-10-06T03:30:34.686568+0000","last_change":"2023-10-06T03:30:34.682210+0000","last_active":"2023-10-06T03:30:34.686568+0000","last_peered":"2023-10-06T03:30:34.686568+0000","last_clean":"2023-10-06T03:30:34.686568+0000","last_became_active":"2023-10-06T03:30:34.681822+0000","last_became_peered":"2023-10-06T03:30:34.681822+0000","last_unstale":"2023-10-06T03:30:34.686568+0000","last_undegraded":"2023-10-06T03:30:34.686568+0000","last_fullsized":"2023-10-06T03:30:34.686568+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:30:33.627150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:30:33.627150+0000","last_clean_scrub_stamp":"2023-10-06T03:30:33.627150+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T09:42:24.260904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.14","version":"41'6","reported_seq":15,"reported_epoch":41,"state":"active+clean","last_fresh":"2023-10-06T03:30:34.687563+0000","last_change":"2023-10-06T03:30:33.668573+0000","last_active":"2023-10-06T03:30:34.687563+0000","last_peered":"2023-10-06T03:30:34.687563+0000","last_clean":"2023-10-06T03:30:34.687563+0000","last_became_active":"2023-10-06T03:30:33.668238+0000","last_became_peered":"2023-10-06T03:30:33.668238+0000","last_unstale":"2023-10-06T03:30:34.687563+0000","last_undegraded":"2023-10-06T03:30:34.687563+0000","last_fullsized":"2023-10-06T03:30:34.687563+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:30:32.615612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:30:32.615612+0000","last_clean_scrub_stamp":"2023-10-06T03:30:32.615612+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T14:37:18.115456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1536,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":9,"reported_epoch":41,"state":"active+clean","last_fresh":"2023-10-06T03:30:34.695407+0000","last_change":"2023-10-06T03:30:34.692097+0000","last_active":"2023-10-06T03:30:34.695407+0000","last_peered":"2023-10-06T03:30:34.695407+0000","last_clean":"2023-10-06T03:30:34.695407+0000","last_became_active":"2023-10-06T03:30:34.691745+0000","last_became_peered":"2023-10-06T03:30:34.691745+0000","last_unstale":"2023-10-06T03:30:34.695407+0000","last_undegraded":"2023-10-06T03:30:34.695407+0000","last_fullsized":"2023-10-06T03:30:34.695407+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:30:33.627150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:30:33.627150+0000","last_clean_scrub_stamp":"2023-10-06T03:30:33.627150+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T15:00:13.119482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"41'2","reported_seq":12,"reported_epoch":41,"state":"active+clean","last_fresh":"2023-10-06T03:30:34.692597+0000","last_change":"2023-10-06T03:30:33.664807+0000","last_active":"2023-10-06T03:30:34.692597+0000","last_peered":"2023-10-06T03:30:34.692597+0000","last_clean":"2023-10-06T03:30:34.692597+0000","last_became_active":"2023-10-06T03:30:33.664547+0000","last_became_peered":"2023-10-06T03:30:33.664547+0000","last_unstale":"2023-10-06T03:30:34.692597+0000","last_undegraded":"2023-10-06T03:30:34.692597+0000","last_fullsized":"2023-10-06T03:30:34.692597+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:30:32.615612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:30:32.615612+0000","last_clean_scrub_stamp":"2023-10-06T03:30:32.615612+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T12:12:51.259303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.13","version":"0'0","reported_seq":9,"reported_epoch":41,"state":"active+clean","last_fresh":"2023-10-06T03:30:34.690533+0000","last_change":"2023-10-06T03:30:34.674581+0000","last_active":"2023-10-06T03:30:34.690533+0000","last_peered":"2023-10-06T03:30:34.690533+0000","last_clean":"2023-10-06T03:30:34.690533+0000","last_became_active":"2023-10-06T03:30:34.674121+0000","last_became_peered":"2023-10-06T03:30:34.674121+0000","last_unstale":"2023-10-06T03:30:34.690533+0000","last_undegraded":"2023-10-06T03:30:34.690533+0000","last_fullsized":"2023-10-06T03:30:34.690533+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:30:33.627150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:30:33.627150+0000","last_clean_scrub_stamp":"2023-10-06T03:30:33.627150+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T04:11:20.451136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":10,"reported_epoch":41,"state":"active+clean","last_fresh":"2023-10-06T03:30:34.663747+0000","last_change":"2023-10-06T03:30:33.670388+0000","last_active":"2023-10-06T03:30:34.663747+0000","last_peered":"2023-10-06T03:30:34.663747+0000","last_clean":"2023-10-06T03:30:34.663747+0000","last_became_active":"2023-10-06T03:30:33.670058+0000","last_became_peered":"2023-10-06T03:30:33.670058+0000","last_unstale":"2023-10-06T03:30:34.663747+0000","last_undegraded":"2023-10-06T03:30:34.663747+0000","last_fullsized":"2023-10-06T03:30:34.663747+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:30:32.615612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:30:32.615612+0000","last_clean_scrub_stamp":"2023-10-06T03:30:32.615612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T05:05:41.515609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.10","version":"0'0","reported_seq":9,"reported_epoch":41,"state":"active+clean","last_fresh":"2023-10-06T03:30:34.667620+0000","last_change":"2023-10-06T03:30:34.664448+0000","last_active":"2023-10-06T03:30:34.667620+0000","last_peered":"2023-10-06T03:30:34.667620+0000","last_clean":"2023-10-06T03:30:34.667620+0000","last_became_active":"2023-10-06T03:30:34.664132+0000","last_became_peered":"2023-10-06T03:30:34.664132+0000","last_unstale":"2023-10-06T03:30:34.667620+0000","last_undegraded":"2023-10-06T03:30:34.667620+0000","last_fullsized":"2023-10-06T03:30:34.667620+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:30:33.627150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:30:33.627150+0000","last_clean_scrub_stamp":"2023-10-06T03:30:33.627150+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T08:18:03.563387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":10,"reported_epoch":41,"state":"active+clean","last_fresh":"2023-10-06T03:30:34.651348+0000","last_change":"2023-10-06T03:30:33.662513+0000","last_active":"2023-10-06T03:30:34.651348+0000","last_peered":"2023-10-06T03:30:34.651348+0000","last_clean":"2023-10-06T03:30:34.651348+0000","last_became_active":"2023-10-06T03:30:33.662244+0000","last_became_peered":"2023-10-06T03:30:33.662244+0000","last_unstale":"2023-10-06T03:30:34.651348+0000","last_undegraded":"2023-10-06T03:30:34.651348+0000","last_fullsized":"2023-10-06T03:30:34.651348+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:30:32.615612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:30:32.615612+0000","last_clean_scrub_stamp":"2023-10-06T03:30:32.615612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T15:05:13.223269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":9,"reported_epoch":41,"state":"active+clean","last_fresh":"2023-10-06T03:30:34.680270+0000","last_change":"2023-10-06T03:30:34.677719+0000","last_active":"2023-10-06T03:30:34.680270+0000","last_peered":"2023-10-06T03:30:34.680270+0000","last_clean":"2023-10-06T03:30:34.680270+0000","last_became_active":"2023-10-06T03:30:34.677161+0000","last_became_peered":"2023-10-06T03:30:34.677161+0000","last_unstale":"2023-10-06T03:30:34.680270+0000","last_undegraded":"2023-10-06T03:30:34.680270+0000","last_fullsized":"2023-10-06T03:30:34.680270+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:30:33.627150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:30:33.627150+0000","last_clean_scrub_stamp":"2023-10-06T03:30:33.627150+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T04:42:51.843533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.10","version":"41'8","reported_seq":18,"reported_epoch":41,"state":"active+clean","last_fresh":"2023-10-06T03:30:34.694945+0000","last_change":"2023-10-06T03:30:33.665504+0000","last_active":"2023-10-06T03:30:34.694945+0000","last_peered":"2023-10-06T03:30:34.694945+0000","last_clean":"2023-10-06T03:30:34.694945+0000","last_became_active":"2023-10-06T03:30:33.665161+0000","last_became_peered":"2023-10-06T03:30:33.665161+0000","last_unstale":"2023-10-06T03:30:34.694945+0000","last_undegraded":"2023-10-06T03:30:34.694945+0000","last_fullsized":"2023-10-06T03:30:34.694945+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:30:32.615612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:30:32.615612+0000","last_clean_scrub_stamp":"2023-10-06T03:30:32.615612+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T08:14:04.282451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.e","version":"0'0","reported_seq":8,"reported_epoch":41,"state":"active+clean","last_fresh":"2023-10-06T03:30:34.666880+0000","last_change":"2023-10-06T03:30:34.666880+0000","last_active":"2023-10-06T03:30:34.666880+0000","last_peered":"2023-10-06T03:30:34.666880+0000","last_clean":"2023-10-06T03:30:34.666880+0000","last_became_active":"2023-10-06T03:30:34.666582+0000","last_became_peered":"2023-10-06T03:30:34.666582+0000","last_unstale":"2023-10-06T03:30:34.666880+0000","last_undegraded":"2023-10-06T03:30:34.666880+0000","last_fullsized":"2023-10-06T03:30:34.666880+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:30:33.627150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:30:33.627150+0000","last_clean_scrub_stamp":"2023-10-06T03:30:33.627150+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T11:23:03.827062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"41'1","reported_seq":10,"reported_epoch":41,"state":"active+clean","last_fresh":"2023-10-06T03:30:34.657064+0000","last_change":"2023-10-06T03:30:33.671975+0000","last_active":"2023-10-06T03:30:34.657064+0000","last_peered":"2023-10-06T03:30:34.657064+0000","last_clean":"2023-10-06T03:30:34.657064+0000","last_became_active":"2023-10-06T03:30:33.671652+0000","last_became_peered":"2023-10-06T03:30:33.671652+0000","last_unstale":"2023-10-06T03:30:34.657064+0000","last_undegraded":"2023-10-06T03:30:34.657064+0000","last_fullsized":"2023-10-06T03:30:34.657064+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:30:32.615612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:30:32.615612+0000","last_clean_scrub_stamp":"2023-10-06T03:30:32.615612+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T13:10:54.937694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.f","version":"0'0","reported_seq":9,"reported_epoch":41,"state":"active+clean","last_fresh":"2023-10-06T03:30:34.690700+0000","last_change":"2023-10-06T03:30:34.678963+0000","last_active":"2023-10-06T03:30:34.690700+0000","last_peered":"2023-10-06T03:30:34.690700+0000","last_clean":"2023-10-06T03:30:34.690700+0000","last_became_active":"2023-10-06T03:30:34.678563+0000","last_became_peered":"2023-10-06T03:30:34.678563+0000","last_unstale":"2023-10-06T03:30:34.690700+0000","last_undegraded":"2023-10-06T03:30:34.690700+0000","last_fullsized":"2023-10-06T03:30:34.690700+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:30:33.627150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:30:33.627150+0000","last_clean_scrub_stamp":"2023-10-06T03:30:33.627150+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T05:04:39.755028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.e","version":"41'2","reported_seq":12,"reported_epoch":41,"state":"active+clean","last_fresh":"2023-10-06T03:30:34.686460+0000","last_change":"2023-10-06T03:30:33.666580+0000","last_active":"2023-10-06T03:30:34.686460+0000","last_peered":"2023-10-06T03:30:34.686460+0000","last_clean":"2023-10-06T03:30:34.686460+0000","last_became_active":"2023-10-06T03:30:33.666284+0000","last_became_peered":"2023-10-06T03:30:33.666284+0000","last_unstale":"2023-10-06T03:30:34.686460+0000","last_undegraded":"2023-10-06T03:30:34.686460+0000","last_fullsized":"2023-10-06T03:30:34.686460+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:30:32.615612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:30:32.615612+0000","last_clean_scrub_stamp":"2023-10-06T03:30:32.615612+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T04:42:50.831995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":9,"reported_epoch":41,"state":"active+clean","last_fresh":"2023-10-06T03:30:34.676060+0000","last_change":"2023-10-06T03:30:34.673719+0000","last_active":"2023-10-06T03:30:34.676060+0000","last_peered":"2023-10-06T03:30:34.676060+0000","last_clean":"2023-10-06T03:30:34.676060+0000","last_became_active":"2023-10-06T03:30:34.673383+0000","last_became_peered":"2023-10-06T03:30:34.673383+0000","last_unstale":"2023-10-06T03:30:34.676060+0000","last_undegraded":"2023-10-06T03:30:34.676060+0000","last_fullsized":"2023-10-06T03:30:34.676060+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:30:33.627150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:30:33.627150+0000","last_clean_scrub_stamp":"2023-10-06T03:30:33.627150+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T10:35:09.809336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.d","version":"41'1","reported_seq":11,"reported_epoch":41,"state":"active+clean","last_fresh":"2023-10-06T03:30:34.685527+0000","last_change":"2023-10-06T03:30:33.666263+0000","last_active":"2023-10-06T03:30:34.685527+0000","last_peered":"2023-10-06T03:30:34.685527+0000","last_clean":"2023-10-06T03:30:34.685527+0000","last_became_active":"2023-10-06T03:30:33.665729+0000","last_became_peered":"2023-10-06T03:30:33.665729+0000","last_unstale":"2023-10-06T03:30:34.685527+0000","last_undegraded":"2023-10-06T03:30:34.685527+0000","last_fullsized":"2023-10-06T03:30:34.685527+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:30:32.615612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:30:32.615612+0000","last_clean_scrub_stamp":"2023-10-06T03:30:32.615612+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T08:26:45.574540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.d","version":"0'0","reported_seq":9,"reported_epoch":41,"state":"active+clean","last_fresh":"2023-10-06T03:30:34.675661+0000","last_change":"2023-10-06T03:30:34.673317+0000","last_active":"2023-10-06T03:30:34.675661+0000","last_peered":"2023-10-06T03:30:34.675661+0000","last_clean":"2023-10-06T03:30:34.675661+0000","last_became_active":"2023-10-06T03:30:34.672989+0000","last_became_peered":"2023-10-06T03:30:34.672989+0000","last_unstale":"2023-10-06T03:30:34.675661+0000","last_undegraded":"2023-10-06T03:30:34.675661+0000","last_fullsized":"2023-10-06T03:30:34.675661+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:30:33.627150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:30:33.627150+0000","last_clean_scrub_stamp":"2023-10-06T03:30:33.627150+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T05:27:20.388907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":10,"reported_epoch":41,"state":"active+clean","last_fresh":"2023-10-06T03:30:34.652110+0000","last_change":"2023-10-06T03:30:33.660529+0000","last_active":"2023-10-06T03:30:34.652110+0000","last_peered":"2023-10-06T03:30:34.652110+0000","last_clean":"2023-10-06T03:30:34.652110+0000","last_became_active":"2023-10-06T03:30:33.660205+0000","last_became_peered":"2023-10-06T03:30:33.660205+0000","last_unstale":"2023-10-06T03:30:34.652110+0000","last_undegraded":"2023-10-06T03:30:34.652110+0000","last_fullsized":"2023-10-06T03:30:34.652110+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:30:32.615612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:30:32.615612+0000","last_clean_scrub_stamp":"2023-10-06T03:30:32.615612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T08:43:57.890425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":9,"reported_epoch":41,"state":"active+clean","last_fresh":"2023-10-06T03:30:34.668289+0000","last_change":"2023-10-06T03:30:34.665508+0000","last_active":"2023-10-06T03:30:34.668289+0000","last_peered":"2023-10-06T03:30:34.668289+0000","last_clean":"2023-10-06T03:30:34.668289+0000","last_became_active":"2023-10-06T03:30:34.664787+0000","last_became_peered":"2023-10-06T03:30:34.664787+0000","last_unstale":"2023-10-06T03:30:34.668289+0000","last_undegraded":"2023-10-06T03:30:34.668289+0000","last_fullsized":"2023-10-06T03:30:34.668289+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:30:33.627150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:30:33.627150+0000","last_clean_scrub_stamp":"2023-10-06T03:30:33.627150+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T15:11:06.944301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.b","version":"41'2","reported_seq":12,"reported_epoch":41,"state":"active+clean","last_fresh":"2023-10-06T03:30:34.687133+0000","last_change":"2023-10-06T03:30:33.667463+0000","last_active":"2023-10-06T03:30:34.687133+0000","last_peered":"2023-10-06T03:30:34.687133+0000","last_clean":"2023-10-06T03:30:34.687133+0000","last_became_active":"2023-10-06T03:30:33.665107+0000","last_became_peered":"2023-10-06T03:30:33.665107+0000","last_unstale":"2023-10-06T03:30:34.687133+0000","last_undegraded":"2023-10-06T03:30:34.687133+0000","last_fullsized":"2023-10-06T03:30:34.687133+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:30:32.615612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:30:32.615612+0000","last_clean_scrub_stamp":"2023-10-06T03:30:32.615612+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T07:30:17.435201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"0'0","reported_seq":8,"reported_epoch":41,"state":"active+clean","last_fresh":"2023-10-06T03:30:34.666719+0000","last_change":"2023-10-06T03:30:34.666719+0000","last_active":"2023-10-06T03:30:34.666719+0000","last_peered":"2023-10-06T03:30:34.666719+0000","last_clean":"2023-10-06T03:30:34.666719+0000","last_became_active":"2023-10-06T03:30:34.665873+0000","last_became_peered":"2023-10-06T03:30:34.665873+0000","last_unstale":"2023-10-06T03:30:34.666719+0000","last_undegraded":"2023-10-06T03:30:34.666719+0000","last_fullsized":"2023-10-06T03:30:34.666719+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:30:33.627150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:30:33.627150+0000","last_clean_scrub_stamp":"2023-10-06T03:30:33.627150+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T03:45:43.730583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":10,"reported_epoch":41,"state":"active+clean","last_fresh":"2023-10-06T03:30:34.661855+0000","last_change":"2023-10-06T03:30:33.665970+0000","last_active":"2023-10-06T03:30:34.661855+0000","last_peered":"2023-10-06T03:30:34.661855+0000","last_clean":"2023-10-06T03:30:34.661855+0000","last_became_active":"2023-10-06T03:30:33.665179+0000","last_became_peered":"2023-10-06T03:30:33.665179+0000","last_unstale":"2023-10-06T03:30:34.661855+0000","last_undegraded":"2023-10-06T03:30:34.661855+0000","last_fullsized":"2023-10-06T03:30:34.661855+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:30:32.615612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:30:32.615612+0000","last_clean_scrub_stamp":"2023-10-06T03:30:32.615612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T05:05:41.515609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":8,"reported_epoch":41,"state":"active+clean","last_fresh":"2023-10-06T03:30:34.668145+0000","last_change":"2023-10-06T03:30:34.668145+0000","last_active":"2023-10-06T03:30:34.668145+0000","last_peered":"2023-10-06T03:30:34.668145+0000","last_clean":"2023-10-06T03:30:34.668145+0000","last_became_active":"2023-10-06T03:30:34.667870+0000","last_became_peered":"2023-10-06T03:30:34.667870+0000","last_unstale":"2023-10-06T03:30:34.668145+0000","last_undegraded":"2023-10-06T03:30:34.668145+0000","last_fullsized":"2023-10-06T03:30:34.668145+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:30:33.627150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:30:33.627150+0000","last_clean_scrub_stamp":"2023-10-06T03:30:33.627150+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T05:05:42.527147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":10,"reported_epoch":41,"state":"active+clean","last_fresh":"2023-10-06T03:30:34.657770+0000","last_change":"2023-10-06T03:30:33.661848+0000","last_active":"2023-10-06T03:30:34.657770+0000","last_peered":"2023-10-06T03:30:34.657770+0000","last_clean":"2023-10-06T03:30:34.657770+0000","last_became_active":"2023-10-06T03:30:33.661539+0000","last_became_peered":"2023-10-06T03:30:33.661539+0000","last_unstale":"2023-10-06T03:30:34.657770+0000","last_undegraded":"2023-10-06T03:30:34.657770+0000","last_fullsized":"2023-10-06T03:30:34.657770+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:30:32.615612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:30:32.615612+0000","last_clean_scrub_stamp":"2023-10-06T03:30:32.615612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T05:43:04.324789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":9,"reported_epoch":41,"state":"active+clean","last_fresh":"2023-10-06T03:30:34.690358+0000","last_change":"2023-10-06T03:30:34.671684+0000","last_active":"2023-10-06T03:30:34.690358+0000","last_peered":"2023-10-06T03:30:34.690358+0000","last_clean":"2023-10-06T03:30:34.690358+0000","last_became_active":"2023-10-06T03:30:34.671415+0000","last_became_peered":"2023-10-06T03:30:34.671415+0000","last_unstale":"2023-10-06T03:30:34.690358+0000","last_undegraded":"2023-10-06T03:30:34.690358+0000","last_fullsized":"2023-10-06T03:30:34.690358+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:30:33.627150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:30:33.627150+0000","last_clean_scrub_stamp":"2023-10-06T03:30:33.627150+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T10:12:42.343138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":10,"reported_epoch":41,"state":"active+clean","last_fresh":"2023-10-06T03:30:34.663576+0000","last_change":"2023-10-06T03:30:33.670054+0000","last_active":"2023-10-06T03:30:34.663576+0000","last_peered":"2023-10-06T03:30:34.663576+0000","last_clean":"2023-10-06T03:30:34.663576+0000","last_became_active":"2023-10-06T03:30:33.669689+0000","last_became_peered":"2023-10-06T03:30:33.669689+0000","last_unstale":"2023-10-06T03:30:34.663576+0000","last_undegraded":"2023-10-06T03:30:34.663576+0000","last_fullsized":"2023-10-06T03:30:34.663576+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:30:32.615612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:30:32.615612+0000","last_clean_scrub_stamp":"2023-10-06T03:30:32.615612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T10:53:06.172743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":9,"reported_epoch":41,"state":"active+clean","last_fresh":"2023-10-06T03:30:34.667228+0000","last_change":"2023-10-06T03:30:34.662684+0000","last_active":"2023-10-06T03:30:34.667228+0000","last_peered":"2023-10-06T03:30:34.667228+0000","last_clean":"2023-10-06T03:30:34.667228+0000","last_became_active":"2023-10-06T03:30:34.662365+0000","last_became_peered":"2023-10-06T03:30:34.662365+0000","last_unstale":"2023-10-06T03:30:34.667228+0000","last_undegraded":"2023-10-06T03:30:34.667228+0000","last_fullsized":"2023-10-06T03:30:34.667228+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:30:33.627150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:30:33.627150+0000","last_clean_scrub_stamp":"2023-10-06T03:30:33.627150+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T04:33:34.388983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":10,"reported_epoch":41,"state":"active+clean","last_fresh":"2023-10-06T03:30:34.664128+0000","last_change":"2023-10-06T03:30:33.673394+0000","last_active":"2023-10-06T03:30:34.664128+0000","last_peered":"2023-10-06T03:30:34.664128+0000","last_clean":"2023-10-06T03:30:34.664128+0000","last_became_active":"2023-10-06T03:30:33.673097+0000","last_became_peered":"2023-10-06T03:30:33.673097+0000","last_unstale":"2023-10-06T03:30:34.664128+0000","last_undegraded":"2023-10-06T03:30:34.664128+0000","last_fullsized":"2023-10-06T03:30:34.664128+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:30:32.615612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:30:32.615612+0000","last_clean_scrub_stamp":"2023-10-06T03:30:32.615612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T07:39:22.429891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":27,"reported_epoch":41,"state":"active+clean","last_fresh":"2023-10-06T03:30:34.667505+0000","last_change":"2023-10-06T03:29:29.387402+0000","last_active":"2023-10-06T03:30:34.667505+0000","last_peered":"2023-10-06T03:30:34.667505+0000","last_clean":"2023-10-06T03:30:34.667505+0000","last_became_active":"2023-10-06T03:29:29.386812+0000","last_became_peered":"2023-10-06T03:29:29.386812+0000","last_unstale":"2023-10-06T03:30:34.667505+0000","last_undegraded":"2023-10-06T03:30:34.667505+0000","last_fullsized":"2023-10-06T03:30:34.667505+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:28.357506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:28.357506+0000","last_clean_scrub_stamp":"2023-10-06T03:29:28.357506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T12:09:43.895810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":9,"reported_epoch":41,"state":"active+clean","last_fresh":"2023-10-06T03:30:34.667592+0000","last_change":"2023-10-06T03:30:34.664854+0000","last_active":"2023-10-06T03:30:34.667592+0000","last_peered":"2023-10-06T03:30:34.667592+0000","last_clean":"2023-10-06T03:30:34.667592+0000","last_became_active":"2023-10-06T03:30:34.664466+0000","last_became_peered":"2023-10-06T03:30:34.664466+0000","last_unstale":"2023-10-06T03:30:34.667592+0000","last_undegraded":"2023-10-06T03:30:34.667592+0000","last_fullsized":"2023-10-06T03:30:34.667592+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:30:33.627150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:30:33.627150+0000","last_clean_scrub_stamp":"2023-10-06T03:30:33.627150+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T09:21:33.343368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":10,"reported_epoch":41,"state":"active+clean","last_fresh":"2023-10-06T03:30:34.654345+0000","last_change":"2023-10-06T03:30:33.664929+0000","last_active":"2023-10-06T03:30:34.654345+0000","last_peered":"2023-10-06T03:30:34.654345+0000","last_clean":"2023-10-06T03:30:34.654345+0000","last_became_active":"2023-10-06T03:30:33.664682+0000","last_became_peered":"2023-10-06T03:30:33.664682+0000","last_unstale":"2023-10-06T03:30:34.654345+0000","last_undegraded":"2023-10-06T03:30:34.654345+0000","last_fullsized":"2023-10-06T03:30:34.654345+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:30:32.615612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:30:32.615612+0000","last_clean_scrub_stamp":"2023-10-06T03:30:32.615612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T08:14:04.282451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":27,"reported_epoch":40,"state":"active+clean","last_fresh":"2023-10-06T03:30:33.658461+0000","last_change":"2023-10-06T03:29:29.386238+0000","last_active":"2023-10-06T03:30:33.658461+0000","last_peered":"2023-10-06T03:30:33.658461+0000","last_clean":"2023-10-06T03:30:33.658461+0000","last_became_active":"2023-10-06T03:29:29.385865+0000","last_became_peered":"2023-10-06T03:29:29.385865+0000","last_unstale":"2023-10-06T03:30:33.658461+0000","last_undegraded":"2023-10-06T03:30:33.658461+0000","last_fullsized":"2023-10-06T03:30:33.658461+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:28.357506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:28.357506+0000","last_clean_scrub_stamp":"2023-10-06T03:29:28.357506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T12:43:34.714730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":9,"reported_epoch":41,"state":"active+clean","last_fresh":"2023-10-06T03:30:34.671016+0000","last_change":"2023-10-06T03:30:34.666777+0000","last_active":"2023-10-06T03:30:34.671016+0000","last_peered":"2023-10-06T03:30:34.671016+0000","last_clean":"2023-10-06T03:30:34.671016+0000","last_became_active":"2023-10-06T03:30:34.666488+0000","last_became_peered":"2023-10-06T03:30:34.666488+0000","last_unstale":"2023-10-06T03:30:34.671016+0000","last_undegraded":"2023-10-06T03:30:34.671016+0000","last_fullsized":"2023-10-06T03:30:34.671016+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:30:33.627150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:30:33.627150+0000","last_clean_scrub_stamp":"2023-10-06T03:30:33.627150+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T13:40:41.879317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":9,"reported_epoch":40,"state":"active+clean","last_fresh":"2023-10-06T03:30:33.669068+0000","last_change":"2023-10-06T03:30:33.664832+0000","last_active":"2023-10-06T03:30:33.669068+0000","last_peered":"2023-10-06T03:30:33.669068+0000","last_clean":"2023-10-06T03:30:33.669068+0000","last_became_active":"2023-10-06T03:30:33.664537+0000","last_became_peered":"2023-10-06T03:30:33.664537+0000","last_unstale":"2023-10-06T03:30:33.669068+0000","last_undegraded":"2023-10-06T03:30:33.669068+0000","last_fullsized":"2023-10-06T03:30:33.669068+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:30:32.615612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:30:32.615612+0000","last_clean_scrub_stamp":"2023-10-06T03:30:32.615612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T13:44:37.657361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":28,"reported_epoch":41,"state":"active+clean","last_fresh":"2023-10-06T03:30:34.664437+0000","last_change":"2023-10-06T03:29:29.384279+0000","last_active":"2023-10-06T03:30:34.664437+0000","last_peered":"2023-10-06T03:30:34.664437+0000","last_clean":"2023-10-06T03:30:34.664437+0000","last_became_active":"2023-10-06T03:29:29.383533+0000","last_became_peered":"2023-10-06T03:29:29.383533+0000","last_unstale":"2023-10-06T03:30:34.664437+0000","last_undegraded":"2023-10-06T03:30:34.664437+0000","last_fullsized":"2023-10-06T03:30:34.664437+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:28.357506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:28.357506+0000","last_clean_scrub_stamp":"2023-10-06T03:29:28.357506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T08:34:19.641525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":9,"reported_epoch":41,"state":"active+clean","last_fresh":"2023-10-06T03:30:34.675087+0000","last_change":"2023-10-06T03:30:34.671599+0000","last_active":"2023-10-06T03:30:34.675087+0000","last_peered":"2023-10-06T03:30:34.675087+0000","last_clean":"2023-10-06T03:30:34.675087+0000","last_became_active":"2023-10-06T03:30:34.671158+0000","last_became_peered":"2023-10-06T03:30:34.671158+0000","last_unstale":"2023-10-06T03:30:34.675087+0000","last_undegraded":"2023-10-06T03:30:34.675087+0000","last_fullsized":"2023-10-06T03:30:34.675087+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:30:33.627150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:30:33.627150+0000","last_clean_scrub_stamp":"2023-10-06T03:30:33.627150+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T04:01:07.493426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":10,"reported_epoch":41,"state":"active+clean","last_fresh":"2023-10-06T03:30:34.657619+0000","last_change":"2023-10-06T03:30:33.658163+0000","last_active":"2023-10-06T03:30:34.657619+0000","last_peered":"2023-10-06T03:30:34.657619+0000","last_clean":"2023-10-06T03:30:34.657619+0000","last_became_active":"2023-10-06T03:30:33.657761+0000","last_became_peered":"2023-10-06T03:30:33.657761+0000","last_unstale":"2023-10-06T03:30:34.657619+0000","last_undegraded":"2023-10-06T03:30:34.657619+0000","last_fullsized":"2023-10-06T03:30:34.657619+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:30:32.615612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:30:32.615612+0000","last_clean_scrub_stamp":"2023-10-06T03:30:32.615612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T14:57:48.864706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":27,"reported_epoch":40,"state":"active+clean","last_fresh":"2023-10-06T03:30:33.657268+0000","last_change":"2023-10-06T03:29:29.382183+0000","last_active":"2023-10-06T03:30:33.657268+0000","last_peered":"2023-10-06T03:30:33.657268+0000","last_clean":"2023-10-06T03:30:33.657268+0000","last_became_active":"2023-10-06T03:29:29.381918+0000","last_became_peered":"2023-10-06T03:29:29.381918+0000","last_unstale":"2023-10-06T03:30:33.657268+0000","last_undegraded":"2023-10-06T03:30:33.657268+0000","last_fullsized":"2023-10-06T03:30:33.657268+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:28.357506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:28.357506+0000","last_clean_scrub_stamp":"2023-10-06T03:29:28.357506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T11:56:26.013716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.7","version":"0'0","reported_seq":9,"reported_epoch":41,"state":"active+clean","last_fresh":"2023-10-06T03:30:34.695394+0000","last_change":"2023-10-06T03:30:34.691973+0000","last_active":"2023-10-06T03:30:34.695394+0000","last_peered":"2023-10-06T03:30:34.695394+0000","last_clean":"2023-10-06T03:30:34.695394+0000","last_became_active":"2023-10-06T03:30:34.691651+0000","last_became_peered":"2023-10-06T03:30:34.691651+0000","last_unstale":"2023-10-06T03:30:34.695394+0000","last_undegraded":"2023-10-06T03:30:34.695394+0000","last_fullsized":"2023-10-06T03:30:34.695394+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:30:33.627150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:30:33.627150+0000","last_clean_scrub_stamp":"2023-10-06T03:30:33.627150+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T05:05:42.527147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":10,"reported_epoch":41,"state":"active+clean","last_fresh":"2023-10-06T03:30:34.650414+0000","last_change":"2023-10-06T03:30:33.660915+0000","last_active":"2023-10-06T03:30:34.650414+0000","last_peered":"2023-10-06T03:30:34.650414+0000","last_clean":"2023-10-06T03:30:34.650414+0000","last_became_active":"2023-10-06T03:30:33.660639+0000","last_became_peered":"2023-10-06T03:30:33.660639+0000","last_unstale":"2023-10-06T03:30:34.650414+0000","last_undegraded":"2023-10-06T03:30:34.650414+0000","last_fullsized":"2023-10-06T03:30:34.650414+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:30:32.615612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:30:32.615612+0000","last_clean_scrub_stamp":"2023-10-06T03:30:32.615612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T09:13:46.237757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":28,"reported_epoch":41,"state":"active+clean","last_fresh":"2023-10-06T03:30:34.662497+0000","last_change":"2023-10-06T03:29:29.384415+0000","last_active":"2023-10-06T03:30:34.662497+0000","last_peered":"2023-10-06T03:30:34.662497+0000","last_clean":"2023-10-06T03:30:34.662497+0000","last_became_active":"2023-10-06T03:29:29.383894+0000","last_became_peered":"2023-10-06T03:29:29.383894+0000","last_unstale":"2023-10-06T03:30:34.662497+0000","last_undegraded":"2023-10-06T03:30:34.662497+0000","last_fullsized":"2023-10-06T03:30:34.662497+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:28.357506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:28.357506+0000","last_clean_scrub_stamp":"2023-10-06T03:29:28.357506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T13:09:00.657107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":9,"reported_epoch":41,"state":"active+clean","last_fresh":"2023-10-06T03:30:34.690625+0000","last_change":"2023-10-06T03:30:34.678582+0000","last_active":"2023-10-06T03:30:34.690625+0000","last_peered":"2023-10-06T03:30:34.690625+0000","last_clean":"2023-10-06T03:30:34.690625+0000","last_became_active":"2023-10-06T03:30:34.675243+0000","last_became_peered":"2023-10-06T03:30:34.675243+0000","last_unstale":"2023-10-06T03:30:34.690625+0000","last_undegraded":"2023-10-06T03:30:34.690625+0000","last_fullsized":"2023-10-06T03:30:34.690625+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:30:33.627150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:30:33.627150+0000","last_clean_scrub_stamp":"2023-10-06T03:30:33.627150+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T05:38:02.968967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":10,"reported_epoch":41,"state":"active+clean","last_fresh":"2023-10-06T03:30:34.664677+0000","last_change":"2023-10-06T03:30:33.673514+0000","last_active":"2023-10-06T03:30:34.664677+0000","last_peered":"2023-10-06T03:30:34.664677+0000","last_clean":"2023-10-06T03:30:34.664677+0000","last_became_active":"2023-10-06T03:30:33.673212+0000","last_became_peered":"2023-10-06T03:30:33.673212+0000","last_unstale":"2023-10-06T03:30:34.664677+0000","last_undegraded":"2023-10-06T03:30:34.664677+0000","last_fullsized":"2023-10-06T03:30:34.664677+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:30:32.615612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:30:32.615612+0000","last_clean_scrub_stamp":"2023-10-06T03:30:32.615612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T10:35:08.797798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":28,"reported_epoch":40,"state":"active+clean","last_fresh":"2023-10-06T03:30:33.652979+0000","last_change":"2023-10-06T03:29:29.606195+0000","last_active":"2023-10-06T03:30:33.652979+0000","last_peered":"2023-10-06T03:30:33.652979+0000","last_clean":"2023-10-06T03:30:33.652979+0000","last_became_active":"2023-10-06T03:29:29.605489+0000","last_became_peered":"2023-10-06T03:29:29.605489+0000","last_unstale":"2023-10-06T03:30:33.652979+0000","last_undegraded":"2023-10-06T03:30:33.652979+0000","last_fullsized":"2023-10-06T03:30:33.652979+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:28.357506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:28.357506+0000","last_clean_scrub_stamp":"2023-10-06T03:29:28.357506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T10:21:35.864895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":9,"reported_epoch":41,"state":"active+clean","last_fresh":"2023-10-06T03:30:34.672575+0000","last_change":"2023-10-06T03:30:34.668419+0000","last_active":"2023-10-06T03:30:34.672575+0000","last_peered":"2023-10-06T03:30:34.672575+0000","last_clean":"2023-10-06T03:30:34.672575+0000","last_became_active":"2023-10-06T03:30:34.668073+0000","last_became_peered":"2023-10-06T03:30:34.668073+0000","last_unstale":"2023-10-06T03:30:34.672575+0000","last_undegraded":"2023-10-06T03:30:34.672575+0000","last_fullsized":"2023-10-06T03:30:34.672575+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:30:33.627150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:30:33.627150+0000","last_clean_scrub_stamp":"2023-10-06T03:30:33.627150+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T07:43:00.839285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":9,"reported_epoch":40,"state":"active+clean","last_fresh":"2023-10-06T03:30:33.671676+0000","last_change":"2023-10-06T03:30:33.670575+0000","last_active":"2023-10-06T03:30:33.671676+0000","last_peered":"2023-10-06T03:30:33.671676+0000","last_clean":"2023-10-06T03:30:33.671676+0000","last_became_active":"2023-10-06T03:30:33.670190+0000","last_became_peered":"2023-10-06T03:30:33.670190+0000","last_unstale":"2023-10-06T03:30:33.671676+0000","last_undegraded":"2023-10-06T03:30:33.671676+0000","last_fullsized":"2023-10-06T03:30:33.671676+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:30:32.615612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:30:32.615612+0000","last_clean_scrub_stamp":"2023-10-06T03:30:32.615612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T11:15:18.455519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":28,"reported_epoch":41,"state":"active+clean","last_fresh":"2023-10-06T03:30:34.663436+0000","last_change":"2023-10-06T03:29:29.384353+0000","last_active":"2023-10-06T03:30:34.663436+0000","last_peered":"2023-10-06T03:30:34.663436+0000","last_clean":"2023-10-06T03:30:34.663436+0000","last_became_active":"2023-10-06T03:29:29.383884+0000","last_became_peered":"2023-10-06T03:29:29.383884+0000","last_unstale":"2023-10-06T03:30:34.663436+0000","last_undegraded":"2023-10-06T03:30:34.663436+0000","last_fullsized":"2023-10-06T03:30:34.663436+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:28.357506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:28.357506+0000","last_clean_scrub_stamp":"2023-10-06T03:29:28.357506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T14:43:09.107140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":9,"reported_epoch":41,"state":"active+clean","last_fresh":"2023-10-06T03:30:34.690312+0000","last_change":"2023-10-06T03:30:34.671403+0000","last_active":"2023-10-06T03:30:34.690312+0000","last_peered":"2023-10-06T03:30:34.690312+0000","last_clean":"2023-10-06T03:30:34.690312+0000","last_became_active":"2023-10-06T03:30:34.671107+0000","last_became_peered":"2023-10-06T03:30:34.671107+0000","last_unstale":"2023-10-06T03:30:34.690312+0000","last_undegraded":"2023-10-06T03:30:34.690312+0000","last_fullsized":"2023-10-06T03:30:34.690312+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:30:33.627150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:30:33.627150+0000","last_clean_scrub_stamp":"2023-10-06T03:30:33.627150+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T07:50:34.678143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.7","version":"41'2","reported_seq":12,"reported_epoch":41,"state":"active+clean","last_fresh":"2023-10-06T03:30:34.684804+0000","last_change":"2023-10-06T03:30:33.665445+0000","last_active":"2023-10-06T03:30:34.684804+0000","last_peered":"2023-10-06T03:30:34.684804+0000","last_clean":"2023-10-06T03:30:34.684804+0000","last_became_active":"2023-10-06T03:30:33.663712+0000","last_became_peered":"2023-10-06T03:30:33.663712+0000","last_unstale":"2023-10-06T03:30:34.684804+0000","last_undegraded":"2023-10-06T03:30:34.684804+0000","last_fullsized":"2023-10-06T03:30:34.684804+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:30:32.615612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:30:32.615612+0000","last_clean_scrub_stamp":"2023-10-06T03:30:32.615612+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T10:56:56.553590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":28,"reported_epoch":41,"state":"active+clean","last_fresh":"2023-10-06T03:30:34.658740+0000","last_change":"2023-10-06T03:29:29.386120+0000","last_active":"2023-10-06T03:30:34.658740+0000","last_peered":"2023-10-06T03:30:34.658740+0000","last_clean":"2023-10-06T03:30:34.658740+0000","last_became_active":"2023-10-06T03:29:29.385692+0000","last_became_peered":"2023-10-06T03:29:29.385692+0000","last_unstale":"2023-10-06T03:30:34.658740+0000","last_undegraded":"2023-10-06T03:30:34.658740+0000","last_fullsized":"2023-10-06T03:30:34.658740+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:28.357506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:28.357506+0000","last_clean_scrub_stamp":"2023-10-06T03:29:28.357506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T10:21:35.864895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":181,"reported_epoch":41,"state":"active+clean","last_fresh":"2023-10-06T03:30:34.663545+0000","last_change":"2023-10-06T03:29:21.978311+0000","last_active":"2023-10-06T03:30:34.663545+0000","last_peered":"2023-10-06T03:30:34.663545+0000","last_clean":"2023-10-06T03:30:34.663545+0000","last_became_active":"2023-10-06T03:29:21.978017+0000","last_became_peered":"2023-10-06T03:29:21.978017+0000","last_unstale":"2023-10-06T03:30:34.663545+0000","last_undegraded":"2023-10-06T03:30:34.663545+0000","last_fullsized":"2023-10-06T03:30:34.663545+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:20.957955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:20.957955+0000","last_clean_scrub_stamp":"2023-10-06T03:29:20.957955+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T07:33:10.429084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":2306,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":6,"up_from":16,"seq":68719476756,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297760,"kb_used_data":472,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074080,"statfs":{"total":96636764160,"available":96331857920,"internally_reserved":0,"allocated":483328,"data_stored":213215,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Oct 6 03:30:26 2023","interfaces":[{"interface":"back","average":{"1min":0.876,"5min":0.876,"15min":0.876},"min":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"max":{"1min":1.149,"5min":1.149,"15min":1.149},"last":0.94499999999999995},{"interface":"front","average":{"1min":0.90400000000000003,"5min":0.90400000000000003,"15min":0.90400000000000003},"min":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"max":{"1min":1.254,"5min":1.254,"15min":1.254},"last":1.141}]},{"osd":1,"last update":"Fri Oct 6 03:30:26 2023","interfaces":[{"interface":"back","average":{"1min":0.81299999999999994,"5min":0.81299999999999994,"15min":0.81299999999999994},"min":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"max":{"1min":1.4730000000000001,"5min":1.4730000000000001,"15min":1.4730000000000001},"last":0.71899999999999997},{"interface":"front","average":{"1min":0.97099999999999997,"5min":0.97099999999999997,"15min":0.97099999999999997},"min":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"max":{"1min":1.3440000000000001,"5min":1.3440000000000001,"15min":1.3440000000000001},"last":1.1160000000000001}]},{"osd":2,"last update":"Fri Oct 6 03:30:26 2023","interfaces":[{"interface":"back","average":{"1min":0.94099999999999995,"5min":0.94099999999999995,"15min":0.94099999999999995},"min":{"1min":0.629,"5min":0.629,"15min":0.629},"max":{"1min":1.236,"5min":1.236,"15min":1.236},"last":1.0740000000000001},{"interface":"front","average":{"1min":0.95599999999999996,"5min":0.95599999999999996,"15min":0.95599999999999996},"min":{"1min":0.623,"5min":0.623,"15min":0.623},"max":{"1min":1.3660000000000001,"5min":1.3660000000000001,"15min":1.3660000000000001},"last":1.0469999999999999}]},{"osd":3,"last update":"Fri Oct 6 03:30:26 2023","interfaces":[{"interface":"back","average":{"1min":0.99299999999999999,"5min":0.99299999999999999,"15min":0.99299999999999999},"min":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":1.3660000000000001,"5min":1.3660000000000001,"15min":1.3660000000000001},"last":0.84999999999999998},{"interface":"front","average":{"1min":0.95599999999999996,"5min":0.95599999999999996,"15min":0.95599999999999996},"min":{"1min":0.76300000000000001,"5min":0.76300000000000001,"15min":0.76300000000000001},"max":{"1min":1.2609999999999999,"5min":1.2609999999999999,"15min":1.2609999999999999},"last":0.89900000000000002}]},{"osd":4,"last update":"Fri Oct 6 03:30:26 2023","interfaces":[{"interface":"back","average":{"1min":1.1040000000000001,"5min":1.1040000000000001,"15min":1.1040000000000001},"min":{"1min":0.875,"5min":0.875,"15min":0.875},"max":{"1min":1.2869999999999999,"5min":1.2869999999999999,"15min":1.2869999999999999},"last":1.0149999999999999},{"interface":"front","average":{"1min":1.1140000000000001,"5min":1.1140000000000001,"15min":1.1140000000000001},"min":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.73399999999999999},"max":{"1min":1.4450000000000001,"5min":1.4450000000000001,"15min":1.4450000000000001},"last":0.97699999999999998}]},{"osd":5,"last update":"Fri Oct 6 03:30:26 2023","interfaces":[{"interface":"back","average":{"1min":1.105,"5min":1.105,"15min":1.105},"min":{"1min":0.86799999999999999,"5min":0.86799999999999999,"15min":0.86799999999999999},"max":{"1min":1.613,"5min":1.613,"15min":1.613},"last":0.82199999999999995},{"interface":"front","average":{"1min":1.1699999999999999,"5min":1.1699999999999999,"15min":1.1699999999999999},"min":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"max":{"1min":1.351,"5min":1.351,"15min":1.351},"last":1.1579999999999999}]},{"osd":7,"last update":"Fri Oct 6 03:30:26 2023","interfaces":[{"interface":"back","average":{"1min":1.2370000000000001,"5min":1.2370000000000001,"15min":1.2370000000000001},"min":{"1min":1.012,"5min":1.012,"15min":1.012},"max":{"1min":1.5409999999999999,"5min":1.5409999999999999,"15min":1.5409999999999999},"last":1.0900000000000001},{"interface":"front","average":{"1min":1.212,"5min":1.212,"15min":1.212},"min":{"1min":0.96099999999999997,"5min":0.96099999999999997,"15min":0.96099999999999997},"max":{"1min":1.704,"5min":1.704,"15min":1.704},"last":0.995}]}]},{"osd":5,"up_from":16,"seq":68719476756,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297756,"kb_used_data":468,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074084,"statfs":{"total":96636764160,"available":96331862016,"internally_reserved":0,"allocated":479232,"data_stored":212641,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Oct 6 03:30:23 2023","interfaces":[{"interface":"back","average":{"1min":0.86399999999999999,"5min":0.86399999999999999,"15min":0.86399999999999999},"min":{"1min":0.505,"5min":0.505,"15min":0.505},"max":{"1min":1.4079999999999999,"5min":1.4079999999999999,"15min":1.4079999999999999},"last":1.214},{"interface":"front","average":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":0.89800000000000002},"min":{"1min":0.64500000000000002,"5min":0.64500000000000002,"15min":0.64500000000000002},"max":{"1min":1.3779999999999999,"5min":1.3779999999999999,"15min":1.3779999999999999},"last":0.81999999999999995}]},{"osd":1,"last update":"Fri Oct 6 03:30:23 2023","interfaces":[{"interface":"back","average":{"1min":0.97799999999999998,"5min":0.97799999999999998,"15min":0.97799999999999998},"min":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":1.2829999999999999,"5min":1.2829999999999999,"15min":1.2829999999999999},"last":0.90800000000000003},{"interface":"front","average":{"1min":0.85999999999999999,"5min":0.85999999999999999,"15min":0.85999999999999999},"min":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"max":{"1min":1.581,"5min":1.581,"15min":1.581},"last":0.88500000000000001}]},{"osd":2,"last update":"Fri Oct 6 03:30:23 2023","interfaces":[{"interface":"back","average":{"1min":1,"5min":1,"15min":1},"min":{"1min":0.67300000000000004,"5min":0.67300000000000004,"15min":0.67300000000000004},"max":{"1min":1.258,"5min":1.258,"15min":1.258},"last":0.77100000000000002},{"interface":"front","average":{"1min":0.90600000000000003,"5min":0.90600000000000003,"15min":0.90600000000000003},"min":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.67100000000000004},"max":{"1min":1.367,"5min":1.367,"15min":1.367},"last":0.68200000000000005}]},{"osd":3,"last update":"Fri Oct 6 03:30:23 2023","interfaces":[{"interface":"back","average":{"1min":1.034,"5min":1.034,"15min":1.034},"min":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.77200000000000002},"max":{"1min":1.4079999999999999,"5min":1.4079999999999999,"15min":1.4079999999999999},"last":1.004},{"interface":"front","average":{"1min":1.034,"5min":1.034,"15min":1.034},"min":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"max":{"1min":1.244,"5min":1.244,"15min":1.244},"last":0.96899999999999997}]},{"osd":4,"last update":"Fri Oct 6 03:30:23 2023","interfaces":[{"interface":"back","average":{"1min":1.0920000000000001,"5min":1.0920000000000001,"15min":1.0920000000000001},"min":{"1min":0.752,"5min":0.752,"15min":0.752},"max":{"1min":2.2050000000000001,"5min":2.2050000000000001,"15min":2.2050000000000001},"last":1.0980000000000001},{"interface":"front","average":{"1min":1.1870000000000001,"5min":1.1870000000000001,"15min":1.1870000000000001},"min":{"1min":0.90900000000000003,"5min":0.90900000000000003,"15min":0.90900000000000003},"max":{"1min":1.518,"5min":1.518,"15min":1.518},"last":1.0740000000000001}]},{"osd":6,"last update":"Fri Oct 6 03:30:23 2023","interfaces":[{"interface":"back","average":{"1min":1.1040000000000001,"5min":1.1040000000000001,"15min":1.1040000000000001},"min":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"max":{"1min":1.3169999999999999,"5min":1.3169999999999999,"15min":1.3169999999999999},"last":1.0429999999999999},{"interface":"front","average":{"1min":1.2370000000000001,"5min":1.2370000000000001,"15min":1.2370000000000001},"min":{"1min":0.94599999999999995,"5min":0.94599999999999995,"15min":0.94599999999999995},"max":{"1min":2.3809999999999998,"5min":2.3809999999999998,"15min":2.3809999999999998},"last":1.2729999999999999}]},{"osd":7,"last update":"Fri Oct 6 03:30:23 2023","interfaces":[{"interface":"back","average":{"1min":1.2370000000000001,"5min":1.2370000000000001,"15min":1.2370000000000001},"min":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.79500000000000004},"max":{"1min":1.637,"5min":1.637,"15min":1.637},"last":1.177},{"interface":"front","average":{"1min":1.2929999999999999,"5min":1.2929999999999999,"15min":1.2929999999999999},"min":{"1min":1.0429999999999999,"5min":1.0429999999999999,"15min":1.0429999999999999},"max":{"1min":2.3330000000000002,"5min":2.3330000000000002,"15min":2.3330000000000002},"last":1.145}]}]},{"osd":4,"up_from":16,"seq":68719476755,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297764,"kb_used_data":476,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074076,"statfs":{"total":96636764160,"available":96331853824,"internally_reserved":0,"allocated":487424,"data_stored":214107,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Oct 6 03:30:25 2023","interfaces":[{"interface":"back","average":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"min":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"max":{"1min":1.129,"5min":1.129,"15min":1.129},"last":0.66900000000000004},{"interface":"front","average":{"1min":0.96299999999999997,"5min":0.96299999999999997,"15min":0.96299999999999997},"min":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"max":{"1min":1.629,"5min":1.629,"15min":1.629},"last":0.998}]},{"osd":1,"last update":"Fri Oct 6 03:30:25 2023","interfaces":[{"interface":"back","average":{"1min":0.89300000000000002,"5min":0.89300000000000002,"15min":0.89300000000000002},"min":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"max":{"1min":1.23,"5min":1.23,"15min":1.23},"last":1.04},{"interface":"front","average":{"1min":0.86799999999999999,"5min":0.86799999999999999,"15min":0.86799999999999999},"min":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"max":{"1min":1.3680000000000001,"5min":1.3680000000000001,"15min":1.3680000000000001},"last":0.91600000000000004}]},{"osd":2,"last update":"Fri Oct 6 03:30:25 2023","interfaces":[{"interface":"back","average":{"1min":0.98999999999999999,"5min":0.98999999999999999,"15min":0.98999999999999999},"min":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":1.5549999999999999,"5min":1.5549999999999999,"15min":1.5549999999999999},"last":0.83799999999999997},{"interface":"front","average":{"1min":0.92100000000000004,"5min":0.92100000000000004,"15min":0.92100000000000004},"min":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.60799999999999998},"max":{"1min":1.306,"5min":1.306,"15min":1.306},"last":0.96499999999999997}]},{"osd":3,"last update":"Fri Oct 6 03:30:25 2023","interfaces":[{"interface":"back","average":{"1min":1.1040000000000001,"5min":1.1040000000000001,"15min":1.1040000000000001},"min":{"1min":0.84499999999999997,"5min":0.84499999999999997,"15min":0.84499999999999997},"max":{"1min":1.5580000000000001,"5min":1.5580000000000001,"15min":1.5580000000000001},"last":1.159},{"interface":"front","average":{"1min":1.04,"5min":1.04,"15min":1.04},"min":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.67500000000000004},"max":{"1min":1.6699999999999999,"5min":1.6699999999999999,"15min":1.6699999999999999},"last":1.127}]},{"osd":5,"last update":"Fri Oct 6 03:30:25 2023","interfaces":[{"interface":"back","average":{"1min":1.044,"5min":1.044,"15min":1.044},"min":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.73399999999999999},"max":{"1min":1.5169999999999999,"5min":1.5169999999999999,"15min":1.5169999999999999},"last":1.0800000000000001},{"interface":"front","average":{"1min":1.266,"5min":1.266,"15min":1.266},"min":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":0.82799999999999996},"max":{"1min":2.2029999999999998,"5min":2.2029999999999998,"15min":2.2029999999999998},"last":1.2669999999999999}]},{"osd":6,"last update":"Fri Oct 6 03:30:25 2023","interfaces":[{"interface":"back","average":{"1min":1.181,"5min":1.181,"15min":1.181},"min":{"1min":0.79900000000000004,"5min":0.79900000000000004,"15min":0.79900000000000004},"max":{"1min":2.0209999999999999,"5min":2.0209999999999999,"15min":2.0209999999999999},"last":1.2410000000000001},{"interface":"front","average":{"1min":1.179,"5min":1.179,"15min":1.179},"min":{"1min":0.71399999999999997,"5min":0.71399999999999997,"15min":0.71399999999999997},"max":{"1min":2.1440000000000001,"5min":2.1440000000000001,"15min":2.1440000000000001},"last":1.208}]},{"osd":7,"last update":"Fri Oct 6 03:30:25 2023","interfaces":[{"interface":"back","average":{"1min":1.337,"5min":1.337,"15min":1.337},"min":{"1min":0.94399999999999995,"5min":0.94399999999999995,"15min":0.94399999999999995},"max":{"1min":2.0939999999999999,"5min":2.0939999999999999,"15min":2.0939999999999999},"last":1.3280000000000001},{"interface":"front","average":{"1min":1.3089999999999999,"5min":1.3089999999999999,"15min":1.3089999999999999},"min":{"1min":0.95899999999999996,"5min":0.95899999999999996,"15min":0.95899999999999996},"max":{"1min":1.968,"5min":1.968,"15min":1.968},"last":1.5229999999999999}]}]},{"osd":7,"up_from":15,"seq":64424509459,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298336,"kb_used_data":1048,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94073504,"statfs":{"total":96636764160,"available":96331268096,"internally_reserved":0,"allocated":1073152,"data_stored":803029,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Oct 6 03:30:26 2023","interfaces":[{"interface":"back","average":{"1min":0.878,"5min":0.878,"15min":0.878},"min":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"max":{"1min":1.228,"5min":1.228,"15min":1.228},"last":0.83599999999999997},{"interface":"front","average":{"1min":0.878,"5min":0.878,"15min":0.878},"min":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":1.181,"5min":1.181,"15min":1.181},"last":0.98899999999999999}]},{"osd":1,"last update":"Fri Oct 6 03:30:26 2023","interfaces":[{"interface":"back","average":{"1min":0.93000000000000005,"5min":0.93000000000000005,"15min":0.93000000000000005},"min":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"max":{"1min":1.2889999999999999,"5min":1.2889999999999999,"15min":1.2889999999999999},"last":0.92600000000000005},{"interface":"front","average":{"1min":0.94299999999999995,"5min":0.94299999999999995,"15min":0.94299999999999995},"min":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"max":{"1min":1.341,"5min":1.341,"15min":1.341},"last":1.052}]},{"osd":2,"last update":"Fri Oct 6 03:30:26 2023","interfaces":[{"interface":"back","average":{"1min":0.95299999999999996,"5min":0.95299999999999996,"15min":0.95299999999999996},"min":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.72599999999999998},"max":{"1min":1.2050000000000001,"5min":1.2050000000000001,"15min":1.2050000000000001},"last":0.871},{"interface":"front","average":{"1min":1.006,"5min":1.006,"15min":1.006},"min":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"max":{"1min":1.409,"5min":1.409,"15min":1.409},"last":1.022}]},{"osd":3,"last update":"Fri Oct 6 03:30:26 2023","interfaces":[{"interface":"back","average":{"1min":0.86299999999999999,"5min":0.86299999999999999,"15min":0.86299999999999999},"min":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"max":{"1min":1.1339999999999999,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.79200000000000004},{"interface":"front","average":{"1min":0.96899999999999997,"5min":0.96899999999999997,"15min":0.96899999999999997},"min":{"1min":0.63400000000000001,"5min":0.63400000000000001,"15min":0.63400000000000001},"max":{"1min":1.256,"5min":1.256,"15min":1.256},"last":0.749}]},{"osd":4,"last update":"Fri Oct 6 03:30:26 2023","interfaces":[{"interface":"back","average":{"1min":0.91900000000000004,"5min":0.91900000000000004,"15min":0.91900000000000004},"min":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"max":{"1min":1.2230000000000001,"5min":1.2230000000000001,"15min":1.2230000000000001},"last":0.94899999999999995},{"interface":"front","average":{"1min":1.0529999999999999,"5min":1.0529999999999999,"15min":1.0529999999999999},"min":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"max":{"1min":1.4750000000000001,"5min":1.4750000000000001,"15min":1.4750000000000001},"last":0.61899999999999999}]},{"osd":5,"last update":"Fri Oct 6 03:30:26 2023","interfaces":[{"interface":"back","average":{"1min":1.0229999999999999,"5min":1.0229999999999999,"15min":1.0229999999999999},"min":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":0.77900000000000003},"max":{"1min":1.431,"5min":1.431,"15min":1.431},"last":0.90100000000000002},{"interface":"front","average":{"1min":1.105,"5min":1.105,"15min":1.105},"min":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"max":{"1min":1.407,"5min":1.407,"15min":1.407},"last":1.0760000000000001}]},{"osd":6,"last update":"Fri Oct 6 03:30:26 2023","interfaces":[{"interface":"back","average":{"1min":1.1399999999999999,"5min":1.1399999999999999,"15min":1.1399999999999999},"min":{"1min":0.86799999999999999,"5min":0.86799999999999999,"15min":0.86799999999999999},"max":{"1min":1.526,"5min":1.526,"15min":1.526},"last":1.1259999999999999},{"interface":"front","average":{"1min":1.198,"5min":1.198,"15min":1.198},"min":{"1min":0.98399999999999999,"5min":0.98399999999999999,"15min":0.98399999999999999},"max":{"1min":1.6719999999999999,"5min":1.6719999999999999,"15min":1.6719999999999999},"last":1.103}]}]},{"osd":0,"up_from":15,"seq":64424509460,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302432,"kb_used_data":1048,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94069408,"statfs":{"total":96636764160,"available":96327073792,"internally_reserved":0,"allocated":1073152,"data_stored":802973,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Oct 6 03:30:22 2023","interfaces":[{"interface":"back","average":{"1min":0.629,"5min":0.629,"15min":0.629},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":1.0029999999999999,"5min":1.0029999999999999,"15min":1.0029999999999999},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.74199999999999999},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":1.179,"5min":1.179,"15min":1.179},"last":0.89500000000000002}]},{"osd":2,"last update":"Fri Oct 6 03:30:22 2023","interfaces":[{"interface":"back","average":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":1.1879999999999999,"5min":1.1879999999999999,"15min":1.1879999999999999},"last":0.64100000000000001},{"interface":"front","average":{"1min":0.72499999999999998,"5min":0.72499999999999998,"15min":0.72499999999999998},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":1.3879999999999999,"5min":1.3879999999999999,"15min":1.3879999999999999},"last":0.78100000000000003}]},{"osd":3,"last update":"Fri Oct 6 03:30:22 2023","interfaces":[{"interface":"back","average":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":1.1930000000000001,"5min":1.1930000000000001,"15min":1.1930000000000001},"last":0.82899999999999996},{"interface":"front","average":{"1min":0.86399999999999999,"5min":0.86399999999999999,"15min":0.86399999999999999},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":1.1970000000000001,"5min":1.1970000000000001,"15min":1.1970000000000001},"last":0.73999999999999999}]},{"osd":4,"last update":"Fri Oct 6 03:30:22 2023","interfaces":[{"interface":"back","average":{"1min":1.0189999999999999,"5min":1.0189999999999999,"15min":1.0189999999999999},"min":{"1min":0.72899999999999998,"5min":0.72899999999999998,"15min":0.72899999999999998},"max":{"1min":1.55,"5min":1.55,"15min":1.55},"last":1.004},{"interface":"front","average":{"1min":0.99099999999999999,"5min":0.99099999999999999,"15min":0.99099999999999999},"min":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"max":{"1min":1.5269999999999999,"5min":1.5269999999999999,"15min":1.5269999999999999},"last":0.97699999999999998}]},{"osd":5,"last update":"Fri Oct 6 03:30:22 2023","interfaces":[{"interface":"back","average":{"1min":0.98499999999999999,"5min":0.98499999999999999,"15min":0.98499999999999999},"min":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"max":{"1min":1.5409999999999999,"5min":1.5409999999999999,"15min":1.5409999999999999},"last":1.1060000000000001},{"interface":"front","average":{"1min":1.087,"5min":1.087,"15min":1.087},"min":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"max":{"1min":1.353,"5min":1.353,"15min":1.353},"last":0.94999999999999996}]},{"osd":6,"last update":"Fri Oct 6 03:30:22 2023","interfaces":[{"interface":"back","average":{"1min":1.145,"5min":1.145,"15min":1.145},"min":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"max":{"1min":1.502,"5min":1.502,"15min":1.502},"last":1.1379999999999999},{"interface":"front","average":{"1min":1.1299999999999999,"5min":1.1299999999999999,"15min":1.1299999999999999},"min":{"1min":0.64900000000000002,"5min":0.64900000000000002,"15min":0.64900000000000002},"max":{"1min":1.6439999999999999,"5min":1.6439999999999999,"15min":1.6439999999999999},"last":1.1679999999999999}]},{"osd":7,"last update":"Fri Oct 6 03:30:22 2023","interfaces":[{"interface":"back","average":{"1min":1.099,"5min":1.099,"15min":1.099},"min":{"1min":0.874,"5min":0.874,"15min":0.874},"max":{"1min":1.369,"5min":1.369,"15min":1.369},"last":1.2030000000000001},{"interface":"front","average":{"1min":1.2030000000000001,"5min":1.2030000000000001,"15min":1.2030000000000001},"min":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"max":{"1min":1.573,"5min":1.573,"15min":1.573},"last":1.24}]}]},{"osd":3,"up_from":14,"seq":60129542163,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":301848,"kb_used_data":464,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94069992,"statfs":{"total":96636764160,"available":96327671808,"internally_reserved":0,"allocated":475136,"data_stored":206220,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Oct 6 03:30:26 2023","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.81899999999999995,"5min":0.81899999999999995,"15min":0.81899999999999995},"last":0.70299999999999996},{"interface":"front","average":{"1min":0.754,"5min":0.754,"15min":0.754},"min":{"1min":0.505,"5min":0.505,"15min":0.505},"max":{"1min":1.0920000000000001,"5min":1.0920000000000001,"15min":1.0920000000000001},"last":0.85399999999999998}]},{"osd":1,"last update":"Fri Oct 6 03:30:20 2023","interfaces":[{"interface":"back","average":{"1min":0.83199999999999996,"5min":0.83199999999999996,"15min":0.83199999999999996},"min":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":1.2210000000000001,"5min":1.2210000000000001,"15min":1.2210000000000001},"last":0.621},{"interface":"front","average":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":1.321,"5min":1.321,"15min":1.321},"last":0.88700000000000001}]},{"osd":2,"last update":"Fri Oct 6 03:30:20 2023","interfaces":[{"interface":"back","average":{"1min":0.86799999999999999,"5min":0.86799999999999999,"15min":0.86799999999999999},"min":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.61199999999999999},"max":{"1min":1.228,"5min":1.228,"15min":1.228},"last":0.73499999999999999},{"interface":"front","average":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"min":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":1.3680000000000001,"5min":1.3680000000000001,"15min":1.3680000000000001},"last":0.753}]},{"osd":4,"last update":"Fri Oct 6 03:30:26 2023","interfaces":[{"interface":"back","average":{"1min":1.0129999999999999,"5min":1.0129999999999999,"15min":1.0129999999999999},"min":{"1min":0.76800000000000002,"5min":0.76800000000000002,"15min":0.76800000000000002},"max":{"1min":1.6359999999999999,"5min":1.6359999999999999,"15min":1.6359999999999999},"last":0.80700000000000005},{"interface":"front","average":{"1min":0.96699999999999997,"5min":0.96699999999999997,"15min":0.96699999999999997},"min":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"max":{"1min":1.409,"5min":1.409,"15min":1.409},"last":0.97499999999999998}]},{"osd":5,"last update":"Fri Oct 6 03:30:26 2023","interfaces":[{"interface":"back","average":{"1min":0.98999999999999999,"5min":0.98999999999999999,"15min":0.98999999999999999},"min":{"1min":0.746,"5min":0.746,"15min":0.746},"max":{"1min":1.387,"5min":1.387,"15min":1.387},"last":1.1599999999999999},{"interface":"front","average":{"1min":1.1359999999999999,"5min":1.1359999999999999,"15min":1.1359999999999999},"min":{"1min":0.89000000000000001,"5min":0.89000000000000001,"15min":0.89000000000000001},"max":{"1min":1.615,"5min":1.615,"15min":1.615},"last":0.94499999999999995}]},{"osd":6,"last update":"Fri Oct 6 03:30:26 2023","interfaces":[{"interface":"back","average":{"1min":1.079,"5min":1.079,"15min":1.079},"min":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"max":{"1min":1.643,"5min":1.643,"15min":1.643},"last":1.056},{"interface":"front","average":{"1min":1.105,"5min":1.105,"15min":1.105},"min":{"1min":0.74399999999999999,"5min":0.74399999999999999,"15min":0.74399999999999999},"max":{"1min":1.6819999999999999,"5min":1.6819999999999999,"15min":1.6819999999999999},"last":1.232}]},{"osd":7,"last update":"Fri Oct 6 03:30:26 2023","interfaces":[{"interface":"back","average":{"1min":1.1850000000000001,"5min":1.1850000000000001,"15min":1.1850000000000001},"min":{"1min":0.92000000000000004,"5min":0.92000000000000004,"15min":0.92000000000000004},"max":{"1min":1.597,"5min":1.597,"15min":1.597},"last":1.0149999999999999},{"interface":"front","average":{"1min":1.111,"5min":1.111,"15min":1.111},"min":{"1min":0.86599999999999999,"5min":0.86599999999999999,"15min":0.86599999999999999},"max":{"1min":1.446,"5min":1.446,"15min":1.446},"last":1.123}]}]},{"osd":2,"up_from":14,"seq":60129542162,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":301852,"kb_used_data":468,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94069988,"statfs":{"total":96636764160,"available":96327667712,"internally_reserved":0,"allocated":479232,"data_stored":207168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Oct 6 03:30:21 2023","interfaces":[{"interface":"back","average":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":1.1240000000000001,"5min":1.1240000000000001,"15min":1.1240000000000001},"last":0.88700000000000001},{"interface":"front","average":{"1min":0.70399999999999996,"5min":0.70399999999999996,"15min":0.70399999999999996},"min":{"1min":0.495,"5min":0.495,"15min":0.495},"max":{"1min":1.1559999999999999,"5min":1.1559999999999999,"15min":1.1559999999999999},"last":0.82099999999999995}]},{"osd":1,"last update":"Fri Oct 6 03:30:21 2023","interfaces":[{"interface":"back","average":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"min":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":1.1060000000000001,"5min":1.1060000000000001,"15min":1.1060000000000001},"last":0.67400000000000004},{"interface":"front","average":{"1min":0.73499999999999999,"5min":0.73499999999999999,"15min":0.73499999999999999},"min":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":1.2390000000000001,"5min":1.2390000000000001,"15min":1.2390000000000001},"last":0.65300000000000002}]},{"osd":3,"last update":"Fri Oct 6 03:30:21 2023","interfaces":[{"interface":"back","average":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":0.78500000000000003},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":1.165,"5min":1.165,"15min":1.165},"last":0.79200000000000004},{"interface":"front","average":{"1min":0.873,"5min":0.873,"15min":0.873},"min":{"1min":0.626,"5min":0.626,"15min":0.626},"max":{"1min":1.2949999999999999,"5min":1.2949999999999999,"15min":1.2949999999999999},"last":0.435}]},{"osd":4,"last update":"Fri Oct 6 03:30:23 2023","interfaces":[{"interface":"back","average":{"1min":1.095,"5min":1.095,"15min":1.095},"min":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"max":{"1min":2.3450000000000002,"5min":2.3450000000000002,"15min":2.3450000000000002},"last":1.0069999999999999},{"interface":"front","average":{"1min":0.95599999999999996,"5min":0.95599999999999996,"15min":0.95599999999999996},"min":{"1min":0.74399999999999999,"5min":0.74399999999999999,"15min":0.74399999999999999},"max":{"1min":1.1950000000000001,"5min":1.1950000000000001,"15min":1.1950000000000001},"last":0.85699999999999998}]},{"osd":5,"last update":"Fri Oct 6 03:30:23 2023","interfaces":[{"interface":"back","average":{"1min":1.139,"5min":1.139,"15min":1.139},"min":{"1min":0.79900000000000004,"5min":0.79900000000000004,"15min":0.79900000000000004},"max":{"1min":2.5169999999999999,"5min":2.5169999999999999,"15min":2.5169999999999999},"last":0.96099999999999997},{"interface":"front","average":{"1min":1.107,"5min":1.107,"15min":1.107},"min":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":0.81799999999999995},"max":{"1min":1.7250000000000001,"5min":1.7250000000000001,"15min":1.7250000000000001},"last":1.032}]},{"osd":6,"last update":"Fri Oct 6 03:30:23 2023","interfaces":[{"interface":"back","average":{"1min":1.014,"5min":1.014,"15min":1.014},"min":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"max":{"1min":1.218,"5min":1.218,"15min":1.218},"last":0.76600000000000001},{"interface":"front","average":{"1min":1.1899999999999999,"5min":1.1899999999999999,"15min":1.1899999999999999},"min":{"1min":0.85899999999999999,"5min":0.85899999999999999,"15min":0.85899999999999999},"max":{"1min":2.2450000000000001,"5min":2.2450000000000001,"15min":2.2450000000000001},"last":1.1539999999999999}]},{"osd":7,"last update":"Fri Oct 6 03:30:21 2023","interfaces":[{"interface":"back","average":{"1min":1.1639999999999999,"5min":1.1639999999999999,"15min":1.1639999999999999},"min":{"1min":0.81599999999999995,"5min":0.81599999999999995,"15min":0.81599999999999995},"max":{"1min":1.8140000000000001,"5min":1.8140000000000001,"15min":1.8140000000000001},"last":0.98499999999999999},{"interface":"front","average":{"1min":1.0860000000000001,"5min":1.0860000000000001,"15min":1.0860000000000001},"min":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.79500000000000004},"max":{"1min":1.2749999999999999,"5min":1.2749999999999999,"15min":1.2749999999999999},"last":0.92500000000000004}]}]},{"osd":1,"up_from":14,"seq":60129542162,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":301840,"kb_used_data":456,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94070000,"statfs":{"total":96636764160,"available":96327680000,"internally_reserved":0,"allocated":466944,"data_stored":205680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Oct 6 03:30:26 2023","interfaces":[{"interface":"back","average":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.60799999999999998},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.78500000000000003},{"interface":"front","average":{"1min":0.71599999999999997,"5min":0.71599999999999997,"15min":0.71599999999999997},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":1.5329999999999999,"5min":1.5329999999999999,"15min":1.5329999999999999},"last":0.74199999999999999}]},{"osd":2,"last update":"Fri Oct 6 03:30:20 2023","interfaces":[{"interface":"back","average":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.93899999999999995,"5min":0.93899999999999995,"15min":0.93899999999999995},"last":0.625},{"interface":"front","average":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"min":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"max":{"1min":1.278,"5min":1.278,"15min":1.278},"last":0.51000000000000001}]},{"osd":3,"last update":"Fri Oct 6 03:30:20 2023","interfaces":[{"interface":"back","average":{"1min":0.72899999999999998,"5min":0.72899999999999998,"15min":0.72899999999999998},"min":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"max":{"1min":0.90500000000000003,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.54900000000000004},{"interface":"front","average":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.3700000000000001,"5min":1.3700000000000001,"15min":1.3700000000000001},"last":0.83099999999999996}]},{"osd":4,"last update":"Fri Oct 6 03:30:26 2023","interfaces":[{"interface":"back","average":{"1min":0.94299999999999995,"5min":0.94299999999999995,"15min":0.94299999999999995},"min":{"1min":0.66400000000000003,"5min":0.66400000000000003,"15min":0.66400000000000003},"max":{"1min":1.7549999999999999,"5min":1.7549999999999999,"15min":1.7549999999999999},"last":0.91300000000000003},{"interface":"front","average":{"1min":0.97499999999999998,"5min":0.97499999999999998,"15min":0.97499999999999998},"min":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":1.415,"5min":1.415,"15min":1.415},"last":1.103}]},{"osd":5,"last update":"Fri Oct 6 03:30:26 2023","interfaces":[{"interface":"back","average":{"1min":0.93999999999999995,"5min":0.93999999999999995,"15min":0.93999999999999995},"min":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"max":{"1min":1.4910000000000001,"5min":1.4910000000000001,"15min":1.4910000000000001},"last":0.96999999999999997},{"interface":"front","average":{"1min":1.0049999999999999,"5min":1.0049999999999999,"15min":1.0049999999999999},"min":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":0.70499999999999996},"max":{"1min":1.4370000000000001,"5min":1.4370000000000001,"15min":1.4370000000000001},"last":1.137}]},{"osd":6,"last update":"Fri Oct 6 03:30:26 2023","interfaces":[{"interface":"back","average":{"1min":1.022,"5min":1.022,"15min":1.022},"min":{"1min":0.69199999999999995,"5min":0.69199999999999995,"15min":0.69199999999999995},"max":{"1min":1.603,"5min":1.603,"15min":1.603},"last":1.159},{"interface":"front","average":{"1min":1.1080000000000001,"5min":1.1080000000000001,"15min":1.1080000000000001},"min":{"1min":0.69199999999999995,"5min":0.69199999999999995,"15min":0.69199999999999995},"max":{"1min":1.5069999999999999,"5min":1.5069999999999999,"15min":1.5069999999999999},"last":0.998}]},{"osd":7,"last update":"Fri Oct 6 03:30:26 2023","interfaces":[{"interface":"back","average":{"1min":1.04,"5min":1.04,"15min":1.04},"min":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.60799999999999998},"max":{"1min":1.6160000000000001,"5min":1.6160000000000001,"15min":1.6160000000000001},"last":1.0249999999999999},{"interface":"front","average":{"1min":1.071,"5min":1.071,"15min":1.071},"min":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.67500000000000004},"max":{"1min":1.5700000000000001,"5min":1.5700000000000001,"15min":1.5700000000000001},"last":1.0489999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1558,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":610,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-06T03:30:39.004 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2023-10-06T03:30:39.004 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2023-10-06T03:30:39.119 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:39.112+0000 7f7d4d259700 1 Processor -- start 2023-10-06T03:30:39.119 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:39.112+0000 7f7d4d259700 1 -- start start 2023-10-06T03:30:39.120 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:39.116+0000 7f7d4d259700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f7d4806c2a0 0x7f7d480b0a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:39.120 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:39.116+0000 7f7d4d259700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d480cfa00 con 0x7f7d4806b7c0 2023-10-06T03:30:39.120 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:39.116+0000 7f7d4d259700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d480cedf0 con 0x7f7d480ce880 2023-10-06T03:30:39.120 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:39.116+0000 7f7d4d259700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7d480cb1c0 con 0x7f7d4806c2a0 2023-10-06T03:30:39.121 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:39.116+0000 7f7d47fff700 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f7d4806b7c0 0x7f7d480ce170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:54654/0 (socket says 172.21.15.110:54654) 2023-10-06T03:30:39.121 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:39.116+0000 7f7d47fff700 1 -- 172.21.15.110:0/1685637478 learned_addr learned my addr 172.21.15.110:0/1685637478 (peer_addr_for_me v1:172.21.15.110:0/0) 2023-10-06T03:30:39.121 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:39.116+0000 7f7d4ca58700 1 --2- 172.21.15.110:0/1685637478 >> v2:172.21.15.192:3300/0 conn(0x7f7d4806c2a0 0x7f7d480b0a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:39.121 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:39.116+0000 7f7d46ffd700 1 -- 172.21.15.110:0/1685637478 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4052312161 0 0) 0x7f7d480cfa00 con 0x7f7d4806b7c0 2023-10-06T03:30:39.121 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:39.116+0000 7f7d46ffd700 1 -- 172.21.15.110:0/1685637478 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7d30003570 con 0x7f7d4806b7c0 2023-10-06T03:30:39.122 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:39.116+0000 7f7d4ca58700 1 -- 172.21.15.110:0/1685637478 >> v1:172.21.15.192:6789/0 conn(0x7f7d480ce880 legacy=0x7f7d4806bb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:39.122 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:39.116+0000 7f7d4ca58700 1 -- 172.21.15.110:0/1685637478 >> v1:172.21.15.110:6789/0 conn(0x7f7d4806b7c0 legacy=0x7f7d480ce170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:39.122 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:39.116+0000 7f7d4ca58700 1 -- 172.21.15.110:0/1685637478 --> v2:172.21.15.192:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d480cf0d0 con 0x7f7d4806c2a0 2023-10-06T03:30:39.122 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:39.116+0000 7f7d4ca58700 1 --2- 172.21.15.110:0/1685637478 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7d4806c2a0 0x7f7d480b0a30 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f7d40009e60 tx=0x7f7d40009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ca71b1d2b2dfc03 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:30:39.123 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:39.116+0000 7f7d46ffd700 1 -- 172.21.15.110:0/1685637478 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d4000e070 con 0x7f7d4806c2a0 2023-10-06T03:30:39.123 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:39.116+0000 7f7d46ffd700 1 -- 172.21.15.110:0/1685637478 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d4000bb60 con 0x7f7d4806c2a0 2023-10-06T03:30:39.123 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:39.116+0000 7f7d46ffd700 1 -- 172.21.15.110:0/1685637478 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d40012430 con 0x7f7d4806c2a0 2023-10-06T03:30:39.123 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:39.116+0000 7f7d4d259700 1 -- 172.21.15.110:0/1685637478 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7d4806c2a0 msgr2=0x7f7d480b0a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:39.123 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:39.116+0000 7f7d4d259700 1 --2- 172.21.15.110:0/1685637478 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7d4806c2a0 0x7f7d480b0a30 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f7d40009e60 tx=0x7f7d40009780 comp rx=0 tx=0).stop 2023-10-06T03:30:39.123 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:39.116+0000 7f7d4d259700 1 -- 172.21.15.110:0/1685637478 shutdown_connections 2023-10-06T03:30:39.123 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:39.116+0000 7f7d4d259700 1 --2- 172.21.15.110:0/1685637478 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7d4806c2a0 0x7f7d480b0a30 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:39.124 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:39.116+0000 7f7d4d259700 1 -- 172.21.15.110:0/1685637478 >> 172.21.15.110:0/1685637478 conn(0x7f7d480cfea0 msgr2=0x7f7d480d02a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:30:39.124 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:39.116+0000 7f7d4d259700 1 -- 172.21.15.110:0/1685637478 shutdown_connections 2023-10-06T03:30:39.124 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:39.116+0000 7f7d4d259700 1 -- 172.21.15.110:0/1685637478 wait complete. 2023-10-06T03:30:39.124 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:39.120+0000 7f7d4d259700 1 Processor -- start 2023-10-06T03:30:39.124 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:39.120+0000 7f7d4d259700 1 -- start start 2023-10-06T03:30:39.127 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:39.120+0000 7f7d4d259700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7d4806b7c0 0x7f7d48149a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:39.127 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:39.120+0000 7f7d4d259700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7d4806c2a0 0x7f7d480b4ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:39.127 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:39.120+0000 7f7d4d259700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7d480ce880 0x7f7d48157ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:39.127 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:39.120+0000 7f7d4d259700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d480c7970 con 0x7f7d4806b7c0 2023-10-06T03:30:39.127 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:39.120+0000 7f7d4d259700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d480b54b0 con 0x7f7d4806c2a0 2023-10-06T03:30:39.128 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:39.120+0000 7f7d4d259700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d480c7080 con 0x7f7d480ce880 2023-10-06T03:30:39.128 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:39.120+0000 7f7d47fff700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7d4806b7c0 0x7f7d48149a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:39.128 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:39.120+0000 7f7d47fff700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7d4806b7c0 0x7f7d48149a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:48186/0 (socket says 172.21.15.110:48186) 2023-10-06T03:30:39.128 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:39.120+0000 7f7d47fff700 1 -- 172.21.15.110:0/240861796 learned_addr learned my addr 172.21.15.110:0/240861796 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:30:39.128 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:39.120+0000 7f7d477fe700 1 --2- 172.21.15.110:0/240861796 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7d4806c2a0 0x7f7d480b4ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:39.128 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:39.120+0000 7f7d4ca58700 1 --2- 172.21.15.110:0/240861796 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7d480ce880 0x7f7d48157ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:39.128 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:39.120+0000 7f7d47fff700 1 -- 172.21.15.110:0/240861796 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7d480ce880 msgr2=0x7f7d48157ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:39.128 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:39.120+0000 7f7d47fff700 1 --2- 172.21.15.110:0/240861796 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7d480ce880 0x7f7d48157ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:39.128 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:39.120+0000 7f7d47fff700 1 -- 172.21.15.110:0/240861796 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7d4806c2a0 msgr2=0x7f7d480b4ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:39.128 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:39.120+0000 7f7d47fff700 1 --2- 172.21.15.110:0/240861796 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7d4806c2a0 0x7f7d480b4ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:39.128 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:39.120+0000 7f7d47fff700 1 -- 172.21.15.110:0/240861796 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d40009a20 con 0x7f7d4806b7c0 2023-10-06T03:30:39.129 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:39.120+0000 7f7d4ca58700 1 --2- 172.21.15.110:0/240861796 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7d480ce880 0x7f7d48157ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-06T03:30:39.129 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:39.120+0000 7f7d47fff700 1 --2- 172.21.15.110:0/240861796 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7d4806b7c0 0x7f7d48149a50 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f7d3c00c7c0 tx=0x7f7d3c007e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:30:39.129 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:39.120+0000 7f7d2ffff700 1 -- 172.21.15.110:0/240861796 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d3c005b30 con 0x7f7d4806b7c0 2023-10-06T03:30:39.129 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:39.120+0000 7f7d4d259700 1 -- 172.21.15.110:0/240861796 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d480c7320 con 0x7f7d4806b7c0 2023-10-06T03:30:39.129 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:39.120+0000 7f7d4d259700 1 -- 172.21.15.110:0/240861796 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d48149f90 con 0x7f7d4806b7c0 2023-10-06T03:30:39.129 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:39.120+0000 7f7d2ffff700 1 -- 172.21.15.110:0/240861796 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d3c005cb0 con 0x7f7d4806b7c0 2023-10-06T03:30:39.129 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:39.120+0000 7f7d2ffff700 1 -- 172.21.15.110:0/240861796 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d3c007320 con 0x7f7d4806b7c0 2023-10-06T03:30:39.130 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:39.124+0000 7f7d4d259700 1 -- 172.21.15.110:0/240861796 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d280052b0 con 0x7f7d4806b7c0 2023-10-06T03:30:39.130 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:39.124+0000 7f7d2ffff700 1 -- 172.21.15.110:0/240861796 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7f7d3c01a360 con 0x7f7d4806b7c0 2023-10-06T03:30:39.130 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:39.124+0000 7f7d2ffff700 1 --2- 172.21.15.110:0/240861796 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f7d34060550 0x7f7d34062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:39.130 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:39.124+0000 7f7d2ffff700 1 -- 172.21.15.110:0/240861796 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6378+0+0 (secure 0 0 0) 0x7f7d3c07bfc0 con 0x7f7d4806b7c0 2023-10-06T03:30:39.131 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:39.124+0000 7f7d477fe700 1 --2- 172.21.15.110:0/240861796 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f7d34060550 0x7f7d34062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:39.131 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:39.124+0000 7f7d477fe700 1 --2- 172.21.15.110:0/240861796 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f7d34060550 0x7f7d34062a00 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f7d48158850 tx=0x7f7d38007ec0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:30:39.139 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:39.136+0000 7f7d2ffff700 1 -- 172.21.15.110:0/240861796 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f7d3c04c270 con 0x7f7d4806b7c0 2023-10-06T03:30:39.353 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:39.348+0000 7f7d4d259700 1 -- 172.21.15.110:0/240861796 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f7d28005560 con 0x7f7d4806b7c0 2023-10-06T03:30:39.360 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:39.356+0000 7f7d2ffff700 1 -- 172.21.15.110:0/240861796 <== mon.0 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v9) v1 ==== 190+0+0 (secure 0 0 0) 0x7f7d3c0170e0 con 0x7f7d4806b7c0 2023-10-06T03:30:39.363 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:39.356+0000 7f7d4d259700 1 -- 172.21.15.110:0/240861796 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f7d34060550 msgr2=0x7f7d34062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:39.363 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:39.356+0000 7f7d4d259700 1 --2- 172.21.15.110:0/240861796 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f7d34060550 0x7f7d34062a00 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f7d48158850 tx=0x7f7d38007ec0 comp rx=0 tx=0).stop 2023-10-06T03:30:39.363 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:39.356+0000 7f7d4d259700 1 -- 172.21.15.110:0/240861796 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7d4806b7c0 msgr2=0x7f7d48149a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:39.363 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:39.356+0000 7f7d4d259700 1 --2- 172.21.15.110:0/240861796 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7d4806b7c0 0x7f7d48149a50 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f7d3c00c7c0 tx=0x7f7d3c007e00 comp rx=0 tx=0).stop 2023-10-06T03:30:39.363 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:39.356+0000 7f7d4d259700 1 -- 172.21.15.110:0/240861796 shutdown_connections 2023-10-06T03:30:39.363 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:39.356+0000 7f7d4d259700 1 --2- 172.21.15.110:0/240861796 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f7d34060550 0x7f7d34062a00 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:39.364 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:39.356+0000 7f7d4d259700 1 --2- 172.21.15.110:0/240861796 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7d480ce880 0x7f7d48157ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:39.364 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:39.356+0000 7f7d4d259700 1 --2- 172.21.15.110:0/240861796 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7d4806c2a0 0x7f7d480b4ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:39.364 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:39.356+0000 7f7d4d259700 1 --2- 172.21.15.110:0/240861796 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7d4806b7c0 0x7f7d48149a50 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:39.364 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:39.356+0000 7f7d4d259700 1 -- 172.21.15.110:0/240861796 >> 172.21.15.110:0/240861796 conn(0x7f7d480cfea0 msgr2=0x7f7d48155530 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:30:39.364 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:39.356+0000 7f7d4d259700 1 -- 172.21.15.110:0/240861796 shutdown_connections 2023-10-06T03:30:39.364 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:39.356+0000 7f7d4d259700 1 -- 172.21.15.110:0/240861796 wait complete. 2023-10-06T03:30:39.364 INFO:teuthology.orchestra.run.smithi110.stderr:updated caps for client.0 2023-10-06T03:30:39.375 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-06T03:30:39.481 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:39.476+0000 7f949e06e700 1 Processor -- start 2023-10-06T03:30:39.482 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:39.476+0000 7f949e06e700 1 -- start start 2023-10-06T03:30:39.483 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:39.476+0000 7f949e06e700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f9498156b80 0x7f94980c68d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:39.483 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:39.476+0000 7f949e06e700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f94980c6e10 0x7f94980ad560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:39.483 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:39.476+0000 7f949e06e700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94980c7f30 con 0x7f9498156f50 2023-10-06T03:30:39.483 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:39.476+0000 7f949e06e700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f94980c7320 con 0x7f9498156b80 2023-10-06T03:30:39.483 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:39.476+0000 7f949e06e700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94980c7600 con 0x7f94980c6e10 2023-10-06T03:30:39.484 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:39.476+0000 7f949c86b700 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f9498156f50 0x7f94980b0e40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:54656/0 (socket says 172.21.15.110:54656) 2023-10-06T03:30:39.484 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:39.476+0000 7f949c86b700 1 -- 172.21.15.110:0/712447552 learned_addr learned my addr 172.21.15.110:0/712447552 (peer_addr_for_me v1:172.21.15.110:0/0) 2023-10-06T03:30:39.484 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:39.476+0000 7f949d06c700 1 --2- 172.21.15.110:0/712447552 >> v2:172.21.15.110:3300/0 conn(0x7f9498156b80 0x7f94980c68d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:39.484 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:39.476+0000 7f949d86d700 1 --2- 172.21.15.110:0/712447552 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f94980c6e10 0x7f94980ad560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:39.484 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:39.476+0000 7f9497fff700 1 -- 172.21.15.110:0/712447552 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1971293 0 0) 0x7f94980c7f30 con 0x7f9498156f50 2023-10-06T03:30:39.484 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:39.476+0000 7f9497fff700 1 -- 172.21.15.110:0/712447552 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9480003570 con 0x7f9498156f50 2023-10-06T03:30:39.484 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:39.480+0000 7f949d06c700 1 -- 172.21.15.110:0/712447552 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f94980c6e10 msgr2=0x7f94980ad560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:30:39.484 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:39.480+0000 7f949d06c700 1 --2- 172.21.15.110:0/712447552 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f94980c6e10 0x7f94980ad560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:39.484 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:39.480+0000 7f949d06c700 1 -- 172.21.15.110:0/712447552 >> v1:172.21.15.110:6789/0 conn(0x7f9498156f50 legacy=0x7f94980b0e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:39.484 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:39.480+0000 7f949d06c700 1 -- 172.21.15.110:0/712447552 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94980c78e0 con 0x7f9498156b80 2023-10-06T03:30:39.485 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:39.480+0000 7f949d06c700 1 --2- 172.21.15.110:0/712447552 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9498156b80 0x7f94980c68d0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f948c0098e0 tx=0x7f948c0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c43eb2e3cfad7a0c server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:30:39.485 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:39.480+0000 7f9497fff700 1 -- 172.21.15.110:0/712447552 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f948c011630 con 0x7f9498156b80 2023-10-06T03:30:39.485 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:39.480+0000 7f9497fff700 1 -- 172.21.15.110:0/712447552 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f948c011790 con 0x7f9498156b80 2023-10-06T03:30:39.486 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:39.480+0000 7f9497fff700 1 -- 172.21.15.110:0/712447552 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f948c011b10 con 0x7f9498156b80 2023-10-06T03:30:39.486 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:39.480+0000 7f949e06e700 1 -- 172.21.15.110:0/712447552 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9498156b80 msgr2=0x7f94980c68d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:39.486 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:39.480+0000 7f949e06e700 1 --2- 172.21.15.110:0/712447552 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9498156b80 0x7f94980c68d0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f948c0098e0 tx=0x7f948c0091a0 comp rx=0 tx=0).stop 2023-10-06T03:30:39.486 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:39.480+0000 7f949e06e700 1 -- 172.21.15.110:0/712447552 shutdown_connections 2023-10-06T03:30:39.486 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:39.480+0000 7f949e06e700 1 --2- 172.21.15.110:0/712447552 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f94980c6e10 0x7f94980ad560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:39.486 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:39.480+0000 7f949e06e700 1 --2- 172.21.15.110:0/712447552 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9498156b80 0x7f94980c68d0 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:39.486 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:39.480+0000 7f949e06e700 1 -- 172.21.15.110:0/712447552 >> 172.21.15.110:0/712447552 conn(0x7f94980cc470 msgr2=0x7f94980c8410 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:30:39.486 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:39.480+0000 7f949e06e700 1 -- 172.21.15.110:0/712447552 shutdown_connections 2023-10-06T03:30:39.486 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:39.480+0000 7f949e06e700 1 -- 172.21.15.110:0/712447552 wait complete. 2023-10-06T03:30:39.488 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:39.480+0000 7f949e06e700 1 Processor -- start 2023-10-06T03:30:39.488 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:39.480+0000 7f949e06e700 1 -- start start 2023-10-06T03:30:39.488 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:39.480+0000 7f949e06e700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f94980c6e10 0x7f949814bf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:39.488 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:39.480+0000 7f949e06e700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9498156b80 0x7f949815a7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:39.488 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:39.480+0000 7f949e06e700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9498156f50 0x7f949815ad30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:39.488 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:39.480+0000 7f949e06e700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f949814c470 con 0x7f9498156f50 2023-10-06T03:30:39.488 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:39.480+0000 7f949e06e700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94980c1e80 con 0x7f9498156b80 2023-10-06T03:30:39.489 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:39.480+0000 7f949e06e700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94980caf50 con 0x7f94980c6e10 2023-10-06T03:30:39.489 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:39.480+0000 7f949d86d700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9498156f50 0x7f949815ad30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:39.489 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:39.480+0000 7f949d86d700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9498156f50 0x7f949815ad30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:48196/0 (socket says 172.21.15.110:48196) 2023-10-06T03:30:39.489 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:39.480+0000 7f949d86d700 1 -- 172.21.15.110:0/3355874672 learned_addr learned my addr 172.21.15.110:0/3355874672 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:30:39.489 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:39.480+0000 7f949c86b700 1 --2- 172.21.15.110:0/3355874672 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9498156b80 0x7f949815a7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:39.489 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:39.480+0000 7f949d06c700 1 --2- 172.21.15.110:0/3355874672 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f94980c6e10 0x7f949814bf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:39.489 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:39.484+0000 7f949d86d700 1 -- 172.21.15.110:0/3355874672 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f94980c6e10 msgr2=0x7f949814bf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:39.489 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:39.484+0000 7f949d86d700 1 --2- 172.21.15.110:0/3355874672 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f94980c6e10 0x7f949814bf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:39.489 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:39.484+0000 7f949d86d700 1 -- 172.21.15.110:0/3355874672 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9498156b80 msgr2=0x7f949815a7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:39.489 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:39.484+0000 7f949d86d700 1 --2- 172.21.15.110:0/3355874672 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9498156b80 0x7f949815a7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:39.489 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:39.484+0000 7f949d86d700 1 -- 172.21.15.110:0/3355874672 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f948c009420 con 0x7f9498156f50 2023-10-06T03:30:39.490 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:39.484+0000 7f949c86b700 1 --2- 172.21.15.110:0/3355874672 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9498156b80 0x7f949815a7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-06T03:30:39.490 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:39.484+0000 7f949d86d700 1 --2- 172.21.15.110:0/3355874672 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9498156f50 0x7f949815ad30 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f948800ea20 tx=0x7f94880091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:30:39.490 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:39.484+0000 7f94957fa700 1 -- 172.21.15.110:0/3355874672 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f948800c7f0 con 0x7f9498156f50 2023-10-06T03:30:39.490 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:39.484+0000 7f94957fa700 1 -- 172.21.15.110:0/3355874672 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f948800c970 con 0x7f9498156f50 2023-10-06T03:30:39.490 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:39.484+0000 7f949e06e700 1 -- 172.21.15.110:0/3355874672 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94980ca340 con 0x7f9498156f50 2023-10-06T03:30:39.490 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:39.484+0000 7f94957fa700 1 -- 172.21.15.110:0/3355874672 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f948800cd30 con 0x7f9498156f50 2023-10-06T03:30:39.490 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:39.484+0000 7f949e06e700 1 -- 172.21.15.110:0/3355874672 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94980ca900 con 0x7f9498156f50 2023-10-06T03:30:39.491 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:39.484+0000 7f949e06e700 1 -- 172.21.15.110:0/3355874672 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f94980afea0 con 0x7f9498156f50 2023-10-06T03:30:39.491 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:39.484+0000 7f94957fa700 1 -- 172.21.15.110:0/3355874672 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7f9488013680 con 0x7f9498156f50 2023-10-06T03:30:39.491 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:39.484+0000 7f94957fa700 1 --2- 172.21.15.110:0/3355874672 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f9484060550 0x7f9484062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:39.491 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:39.484+0000 7f94957fa700 1 -- 172.21.15.110:0/3355874672 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6378+0+0 (secure 0 0 0) 0x7f948807ccb0 con 0x7f9498156f50 2023-10-06T03:30:39.491 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:39.484+0000 7f949d06c700 1 --2- 172.21.15.110:0/3355874672 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f9484060550 0x7f9484062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:39.491 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:39.484+0000 7f949d06c700 1 --2- 172.21.15.110:0/3355874672 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f9484060550 0x7f9484062a00 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f948c0091e0 tx=0x7f948c017000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:30:39.498 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:39.492+0000 7f94957fa700 1 -- 172.21.15.110:0/3355874672 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f94980afea0 con 0x7f9498156f50 2023-10-06T03:30:39.693 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:39.688+0000 7f949e06e700 1 -- 172.21.15.110:0/3355874672 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f94980afea0 con 0x7f9498156f50 2023-10-06T03:30:39.694 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:39.688+0000 7f94957fa700 1 -- 172.21.15.110:0/3355874672 <== mon.0 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) v1 ==== 94+0+4603 (secure 0 0 0) 0x7f94980afea0 con 0x7f9498156f50 2023-10-06T03:30:39.694 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-10-06T03:30:39.694 INFO:teuthology.orchestra.run.smithi110.stdout:{"epoch":16,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5107,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6834/884349347","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6833","nonce":884349347},{"type":"v1","addr":"172.21.15.192:6834","nonce":884349347}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":5110,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6835/4077204152","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6832","nonce":4077204152},{"type":"v1","addr":"172.21.15.192:6835","nonce":4077204152}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":5160,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/504583843","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6834","nonce":504583843},{"type":"v1","addr":"172.21.15.110:6835","nonce":504583843}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-06T03:30:34.014644+0000","modified":"2023-10-06T03:30:37.662700+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5163},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5163":{"gid":5163,"name":"a","rank":0,"incarnation":13,"state":"up:active","state_seq":4,"addr":"172.21.15.110:6837/1379510962","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6836","nonce":1379510962},{"type":"v1","addr":"172.21.15.110:6837","nonce":1379510962}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":2}]} 2023-10-06T03:30:39.698 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:39.692+0000 7f949e06e700 1 -- 172.21.15.110:0/3355874672 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f9484060550 msgr2=0x7f9484062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:39.698 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:39.692+0000 7f949e06e700 1 --2- 172.21.15.110:0/3355874672 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f9484060550 0x7f9484062a00 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f948c0091e0 tx=0x7f948c017000 comp rx=0 tx=0).stop 2023-10-06T03:30:39.698 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:39.692+0000 7f949e06e700 1 -- 172.21.15.110:0/3355874672 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9498156f50 msgr2=0x7f949815ad30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:39.698 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:39.692+0000 7f949e06e700 1 --2- 172.21.15.110:0/3355874672 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9498156f50 0x7f949815ad30 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f948800ea20 tx=0x7f94880091a0 comp rx=0 tx=0).stop 2023-10-06T03:30:39.698 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:39.692+0000 7f949e06e700 1 -- 172.21.15.110:0/3355874672 shutdown_connections 2023-10-06T03:30:39.698 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:39.692+0000 7f949e06e700 1 --2- 172.21.15.110:0/3355874672 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f9484060550 0x7f9484062a00 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:39.698 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:39.692+0000 7f949e06e700 1 --2- 172.21.15.110:0/3355874672 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9498156f50 0x7f949815ad30 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:39.699 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:39.692+0000 7f949e06e700 1 --2- 172.21.15.110:0/3355874672 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9498156b80 0x7f949815a7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:39.699 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:39.692+0000 7f949e06e700 1 --2- 172.21.15.110:0/3355874672 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f94980c6e10 0x7f949814bf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:39.699 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:39.692+0000 7f949e06e700 1 -- 172.21.15.110:0/3355874672 >> 172.21.15.110:0/3355874672 conn(0x7f94980cc470 msgr2=0x7f94980c49c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:30:39.699 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:39.692+0000 7f949e06e700 1 -- 172.21.15.110:0/3355874672 shutdown_connections 2023-10-06T03:30:39.699 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:39.692+0000 7f949e06e700 1 -- 172.21.15.110:0/3355874672 wait complete. 2023-10-06T03:30:39.699 INFO:teuthology.orchestra.run.smithi110.stderr:dumped fsmap epoch 16 2023-10-06T03:30:40.114 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 16, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-10-06T03:30:34.014644+0000', 'modified': '2023-10-06T03:30:37.662700+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5163}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5163': {'gid': 5163, 'name': 'a', 'rank': 0, 'incarnation': 13, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.110:6837/1379510962', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.110:6836', 'nonce': 1379510962}, {'type': 'v1', 'addr': '172.21.15.110:6837', 'nonce': 1379510962}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138320759226367, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [7], 'metadata_pool': 6, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2023-10-06T03:30:40.114 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2023-10-06T03:30:40.115 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2023-10-06T03:30:40.234 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.228+0000 7fc085edd700 1 Processor -- start 2023-10-06T03:30:40.235 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.228+0000 7fc085edd700 1 -- start start 2023-10-06T03:30:40.236 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.232+0000 7fc085edd700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fc080157130 0x7fc0800695e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:40.236 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.232+0000 7fc085edd700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fc080069b20 0x7fc08015ac70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:40.236 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.232+0000 7fc085edd700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc080158090 con 0x7fc080156d60 2023-10-06T03:30:40.237 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.232+0000 7fc085edd700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc080157580 con 0x7fc080157130 2023-10-06T03:30:40.237 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.232+0000 7fc085edd700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc0801577a0 con 0x7fc080069b20 2023-10-06T03:30:40.237 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.232+0000 7fc077fff700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fc080157130 0x7fc0800695e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:40.237 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.232+0000 7fc077fff700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fc080157130 0x7fc0800695e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:48200/0 (socket says 172.21.15.110:48200) 2023-10-06T03:30:40.237 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.232+0000 7fc077fff700 1 -- 172.21.15.110:0/395412171 learned_addr learned my addr 172.21.15.110:0/395412171 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:30:40.237 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.232+0000 7fc07f7fe700 1 --1- 172.21.15.110:0/395412171 >> v1:172.21.15.110:6789/0 conn(0x7fc080156d60 0x7fc0800c6a90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:54664/0 (socket says 172.21.15.110:54664) 2023-10-06T03:30:40.237 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.232+0000 7fc07ffff700 1 --2- 172.21.15.110:0/395412171 >> v2:172.21.15.192:3300/0 conn(0x7fc080069b20 0x7fc08015ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:40.238 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.232+0000 7fc077fff700 1 -- 172.21.15.110:0/395412171 >> v2:172.21.15.192:3300/0 conn(0x7fc080069b20 msgr2=0x7fc08015ac70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:30:40.238 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.232+0000 7fc077fff700 1 --2- 172.21.15.110:0/395412171 >> v2:172.21.15.192:3300/0 conn(0x7fc080069b20 0x7fc08015ac70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:40.238 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.232+0000 7fc077fff700 1 -- 172.21.15.110:0/395412171 >> v1:172.21.15.110:6789/0 conn(0x7fc080156d60 legacy=0x7fc0800c6a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:40.238 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.232+0000 7fc077fff700 1 -- 172.21.15.110:0/395412171 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc080069f10 con 0x7fc080157130 2023-10-06T03:30:40.238 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.232+0000 7fc077fff700 1 --2- 172.21.15.110:0/395412171 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc080157130 0x7fc0800695e0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7fc080067ea0 tx=0x7fc068009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af3b05bddb02a3db server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:30:40.239 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.232+0000 7fc07effd700 1 -- 172.21.15.110:0/395412171 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc068009e70 con 0x7fc080157130 2023-10-06T03:30:40.239 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.232+0000 7fc07effd700 1 -- 172.21.15.110:0/395412171 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc06800b860 con 0x7fc080157130 2023-10-06T03:30:40.239 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.232+0000 7fc07effd700 1 -- 172.21.15.110:0/395412171 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc06800bc00 con 0x7fc080157130 2023-10-06T03:30:40.239 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.232+0000 7fc085edd700 1 -- 172.21.15.110:0/395412171 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc080157130 msgr2=0x7fc0800695e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:40.240 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.232+0000 7fc085edd700 1 --2- 172.21.15.110:0/395412171 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc080157130 0x7fc0800695e0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7fc080067ea0 tx=0x7fc068009250 comp rx=0 tx=0).stop 2023-10-06T03:30:40.240 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.232+0000 7fc085edd700 1 -- 172.21.15.110:0/395412171 shutdown_connections 2023-10-06T03:30:40.240 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.232+0000 7fc085edd700 1 --2- 172.21.15.110:0/395412171 >> v2:172.21.15.192:3300/0 conn(0x7fc080069b20 0x7fc08015ac70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:40.240 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.232+0000 7fc085edd700 1 --2- 172.21.15.110:0/395412171 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc080157130 0x7fc0800695e0 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:40.240 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.232+0000 7fc085edd700 1 -- 172.21.15.110:0/395412171 >> 172.21.15.110:0/395412171 conn(0x7fc0800cc710 msgr2=0x7fc0800c8670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:30:40.240 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.232+0000 7fc085edd700 1 -- 172.21.15.110:0/395412171 shutdown_connections 2023-10-06T03:30:40.240 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.232+0000 7fc085edd700 1 -- 172.21.15.110:0/395412171 wait complete. 2023-10-06T03:30:40.242 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.236+0000 7fc085edd700 1 Processor -- start 2023-10-06T03:30:40.242 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.236+0000 7fc085edd700 1 -- start start 2023-10-06T03:30:40.245 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.240+0000 7fc085edd700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc080069b20 0x7fc080149a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:40.245 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.240+0000 7fc085edd700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc080156d60 0x7fc0800ad880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:40.245 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.240+0000 7fc085edd700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc080157130 0x7fc0800aeab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:40.245 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.240+0000 7fc085edd700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0800c58c0 con 0x7fc080157130 2023-10-06T03:30:40.245 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.240+0000 7fc085edd700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0800ad3f0 con 0x7fc080069b20 2023-10-06T03:30:40.245 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.240+0000 7fc085edd700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0800ac500 con 0x7fc080156d60 2023-10-06T03:30:40.245 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.240+0000 7fc07ffff700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc080157130 0x7fc0800aeab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:40.245 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.240+0000 7fc07ffff700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc080157130 0x7fc0800aeab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:48204/0 (socket says 172.21.15.110:48204) 2023-10-06T03:30:40.246 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.240+0000 7fc07ffff700 1 -- 172.21.15.110:0/2769863201 learned_addr learned my addr 172.21.15.110:0/2769863201 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:30:40.246 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.240+0000 7fc07f7fe700 1 --2- 172.21.15.110:0/2769863201 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc080069b20 0x7fc080149a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:40.246 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.240+0000 7fc077fff700 1 --2- 172.21.15.110:0/2769863201 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc080156d60 0x7fc0800ad880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:40.246 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.240+0000 7fc07ffff700 1 -- 172.21.15.110:0/2769863201 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc080156d60 msgr2=0x7fc0800ad880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:40.246 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.240+0000 7fc07ffff700 1 --2- 172.21.15.110:0/2769863201 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc080156d60 0x7fc0800ad880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:40.246 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.240+0000 7fc07ffff700 1 -- 172.21.15.110:0/2769863201 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc080069b20 msgr2=0x7fc080149a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:40.246 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.240+0000 7fc07ffff700 1 --2- 172.21.15.110:0/2769863201 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc080069b20 0x7fc080149a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:40.246 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.240+0000 7fc07ffff700 1 -- 172.21.15.110:0/2769863201 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0680094f0 con 0x7fc080157130 2023-10-06T03:30:40.247 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.240+0000 7fc07ffff700 1 --2- 172.21.15.110:0/2769863201 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc080157130 0x7fc0800aeab0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7fc078009e90 tx=0x7fc07800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:30:40.247 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.240+0000 7fc0777fe700 1 -- 172.21.15.110:0/2769863201 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc07800cb70 con 0x7fc080157130 2023-10-06T03:30:40.247 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.240+0000 7fc0777fe700 1 -- 172.21.15.110:0/2769863201 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc07800f040 con 0x7fc080157130 2023-10-06T03:30:40.247 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.240+0000 7fc085edd700 1 -- 172.21.15.110:0/2769863201 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0800ac7e0 con 0x7fc080157130 2023-10-06T03:30:40.248 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.240+0000 7fc0777fe700 1 -- 172.21.15.110:0/2769863201 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc078014480 con 0x7fc080157130 2023-10-06T03:30:40.248 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.240+0000 7fc085edd700 1 -- 172.21.15.110:0/2769863201 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0800acda0 con 0x7fc080157130 2023-10-06T03:30:40.248 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.244+0000 7fc0777fe700 1 -- 172.21.15.110:0/2769863201 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7fc0780147f0 con 0x7fc080157130 2023-10-06T03:30:40.249 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.244+0000 7fc0777fe700 1 --2- 172.21.15.110:0/2769863201 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fc060060550 0x7fc060062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:40.250 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.244+0000 7fc07f7fe700 1 --2- 172.21.15.110:0/2769863201 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fc060060550 0x7fc060062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:40.250 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.244+0000 7fc0777fe700 1 -- 172.21.15.110:0/2769863201 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6378+0+0 (secure 0 0 0) 0x7fc07807d790 con 0x7fc080157130 2023-10-06T03:30:40.250 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.244+0000 7fc07f7fe700 1 --2- 172.21.15.110:0/2769863201 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fc060060550 0x7fc060062a00 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fc070004fd0 tx=0x7fc070007eb0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:30:40.255 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.248+0000 7fc088186740 1 Processor -- start 2023-10-06T03:30:40.255 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.248+0000 7fc088186740 1 -- start start 2023-10-06T03:30:40.255 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.248+0000 7fc088186740 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0xa1cae0 0xc99e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:40.255 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.248+0000 7fc088186740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0xa1c710 0xc9a390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:40.255 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.248+0000 7fc088186740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0xa67790 0xc9a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:40.255 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.248+0000 7fc088186740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0xbc7f30 con 0xa67790 2023-10-06T03:30:40.256 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.248+0000 7fc088186740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0xbc3ed0 con 0xa1c710 2023-10-06T03:30:40.256 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.248+0000 7fc088186740 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0xb8a0b0 con 0xa1cae0 2023-10-06T03:30:40.256 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.248+0000 7fc07ffff700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0xa67790 0xc9a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:40.256 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.248+0000 7fc07ffff700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0xa67790 0xc9a8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:48214/0 (socket says 172.21.15.110:48214) 2023-10-06T03:30:40.256 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.248+0000 7fc07ffff700 1 -- 172.21.15.110:0/1389602939 learned_addr learned my addr 172.21.15.110:0/1389602939 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:30:40.256 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.248+0000 7fc077fff700 1 --2- 172.21.15.110:0/1389602939 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0xa1c710 0xc9a390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:40.256 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.248+0000 7fc07f7fe700 1 --2- 172.21.15.110:0/1389602939 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0xa1cae0 0xc99e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:40.256 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.248+0000 7fc07ffff700 1 -- 172.21.15.110:0/1389602939 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0xa1cae0 msgr2=0xc99e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:30:40.256 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.248+0000 7fc07ffff700 1 --2- 172.21.15.110:0/1389602939 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0xa1cae0 0xc99e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:40.256 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.248+0000 7fc07ffff700 1 -- 172.21.15.110:0/1389602939 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0xa1c710 msgr2=0xc9a390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:30:40.256 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.248+0000 7fc07ffff700 1 --2- 172.21.15.110:0/1389602939 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0xa1c710 0xc9a390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:40.257 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.248+0000 7fc07ffff700 1 -- 172.21.15.110:0/1389602939 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0xa689e0 con 0xa67790 2023-10-06T03:30:40.257 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.248+0000 7fc07f7fe700 1 --2- 172.21.15.110:0/1389602939 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0xa1cae0 0xc99e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-06T03:30:40.257 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.248+0000 7fc07ffff700 1 --2- 172.21.15.110:0/1389602939 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0xa67790 0xc9a8d0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7fc07804dc00 tx=0x7fc07807b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c4aa373df37fa34 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:30:40.257 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.248+0000 7fc074ff9700 1 -- 172.21.15.110:0/1389602939 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc078066070 con 0xa67790 2023-10-06T03:30:40.257 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.248+0000 7fc074ff9700 1 -- 172.21.15.110:0/1389602939 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0780586e0 con 0xa67790 2023-10-06T03:30:40.257 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.248+0000 7fc074ff9700 1 -- 172.21.15.110:0/1389602939 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc07804f840 con 0xa67790 2023-10-06T03:30:40.260 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.252+0000 7fc088186740 1 -- 172.21.15.110:0/1389602939 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0xa67790 msgr2=0xc9a8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:40.260 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.252+0000 7fc088186740 1 --2- 172.21.15.110:0/1389602939 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0xa67790 0xc9a8d0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7fc07804dc00 tx=0x7fc07807b8d0 comp rx=0 tx=0).stop 2023-10-06T03:30:40.261 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.252+0000 7fc088186740 1 -- 172.21.15.110:0/1389602939 shutdown_connections 2023-10-06T03:30:40.261 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.252+0000 7fc088186740 1 --2- 172.21.15.110:0/1389602939 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0xa67790 0xc9a8d0 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:40.261 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.252+0000 7fc088186740 1 --2- 172.21.15.110:0/1389602939 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0xa1c710 0xc9a390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:40.261 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.252+0000 7fc088186740 1 --2- 172.21.15.110:0/1389602939 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0xa1cae0 0xc99e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:40.261 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.252+0000 7fc088186740 1 -- 172.21.15.110:0/1389602939 >> 172.21.15.110:0/1389602939 conn(0xa1ceb0 msgr2=0xc56880 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:30:40.261 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.252+0000 7fc088186740 1 -- 172.21.15.110:0/1389602939 shutdown_connections 2023-10-06T03:30:40.261 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.252+0000 7fc088186740 1 -- 172.21.15.110:0/1389602939 wait complete. 2023-10-06T03:30:40.261 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.252+0000 7fc088186740 1 -- start start 2023-10-06T03:30:40.261 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.252+0000 7fc088186740 1 Processor -- start 2023-10-06T03:30:40.261 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.252+0000 7fc088186740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0xa1c710 0xc56740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:40.261 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.252+0000 7fc088186740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0xa1cae0 0xc7fc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:40.261 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.252+0000 7fc088186740 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0xa67790 0xc80140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:40.262 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.252+0000 7fc088186740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0xb89d50 con 0xa1c710 2023-10-06T03:30:40.262 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.252+0000 7fc088186740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0xb62970 con 0xa1cae0 2023-10-06T03:30:40.262 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.252+0000 7fc088186740 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0xbc4770 con 0xa67790 2023-10-06T03:30:40.262 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.252+0000 7fc07f7fe700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0xa1c710 0xc56740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:40.262 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.252+0000 7fc088186740 10 client.0 _ll_register_callbacks cb 0xc0db00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-06T03:30:40.262 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.252+0000 7fc07f7fe700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0xa1c710 0xc56740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:48216/0 (socket says 172.21.15.110:48216) 2023-10-06T03:30:40.262 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.252+0000 7fc07f7fe700 1 -- 172.21.15.110:0/2008003426 learned_addr learned my addr 172.21.15.110:0/2008003426 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:30:40.262 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.252+0000 7fc077fff700 1 --2- 172.21.15.110:0/2008003426 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0xa1cae0 0xc7fc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:40.262 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.252+0000 7fc07ffff700 1 --2- 172.21.15.110:0/2008003426 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0xa67790 0xc80140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:40.262 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.252+0000 7fc07f7fe700 1 -- 172.21.15.110:0/2008003426 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0xa67790 msgr2=0xc80140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:30:40.263 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.252+0000 7fc07f7fe700 1 --2- 172.21.15.110:0/2008003426 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0xa67790 0xc80140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:40.263 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.252+0000 7fc07f7fe700 1 -- 172.21.15.110:0/2008003426 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0xa1cae0 msgr2=0xc7fc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:30:40.263 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.252+0000 7fc07f7fe700 1 --2- 172.21.15.110:0/2008003426 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0xa1cae0 0xc7fc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:40.263 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.252+0000 7fc07f7fe700 1 -- 172.21.15.110:0/2008003426 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0800acda0 con 0xa1c710 2023-10-06T03:30:40.263 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.252+0000 7fc07f7fe700 1 --2- 172.21.15.110:0/2008003426 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0xa1c710 0xc56740 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7fc0700137a0 tx=0x7fc070014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=536396b410ea0ed server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:30:40.263 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.252+0000 7fc05dffb700 10 client.0 ms_handle_connect on v2:172.21.15.110:3300/0 2023-10-06T03:30:40.263 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.256+0000 7fc05dffb700 1 -- 172.21.15.110:0/2008003426 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc070004cc0 con 0xa1c710 2023-10-06T03:30:40.263 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.256+0000 7fc05dffb700 1 -- 172.21.15.110:0/2008003426 <== mon.0 v2:172.21.15.110:3300/0 2 ==== osd_map(42..42 src has 1..42) v4 ==== 6378+0+0 (secure 0 0 0) 0x7fc07001b070 con 0xa1c710 2023-10-06T03:30:40.263 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.256+0000 7fc088186740 1 -- 172.21.15.110:0/2008003426 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0xb62cc0 con 0xa1c710 2023-10-06T03:30:40.263 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.256+0000 7fc05dffb700 1 -- 172.21.15.110:0/2008003426 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_get_version_reply(handle=1 version=16) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc07002c3b0 con 0xa1c710 2023-10-06T03:30:40.263 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.256+0000 7fc088186740 10 client.5352 fetch_fsmap learned FSMap version 16 2023-10-06T03:30:40.263 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.256+0000 7fc088186740 1 -- 172.21.15.110:0/2008003426 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,fsmap=16,monmap=2+}) v3 -- 0xb92370 con 0xa1c710 2023-10-06T03:30:40.264 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.256+0000 7fc05dffb700 1 -- 172.21.15.110:0/2008003426 <== mon.0 v2:172.21.15.110:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0xb62cc0 con 0xa1c710 2023-10-06T03:30:40.264 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.256+0000 7fc05dffb700 1 -- 172.21.15.110:0/2008003426 <== mon.0 v2:172.21.15.110:3300/0 5 ==== fsmap(e 16) v1 ==== 3100+0+0 (secure 0 0 0) 0x7fc07002c510 con 0xa1c710 2023-10-06T03:30:40.264 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.256+0000 7fc088186740 10 client.5352 fetch_fsmap finished waiting for FSMap version 16 2023-10-06T03:30:40.264 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.256+0000 7fc088186740 10 client.5352 resolve_mds: resolved name 'a' to daemon mds.a 2023-10-06T03:30:40.264 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.256+0000 7fc088186740 20 client.5352 populate_metadata read hostname 'smithi110' 2023-10-06T03:30:40.264 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.256+0000 7fc088186740 1 --2- 172.21.15.110:0/2008003426 >> [v2:172.21.15.110:6836/1379510962,v1:172.21.15.110:6837/1379510962] conn(0xca66b0 0xca8a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:40.264 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.256+0000 7fc088186740 4 client.5352 mds_command: new command op to 5163 tid=0[{"prefix": "get_command_descriptions"}] 2023-10-06T03:30:40.264 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.256+0000 7fc088186740 1 -- 172.21.15.110:0/2008003426 --> [v2:172.21.15.110:6836/1379510962,v1:172.21.15.110:6837/1379510962] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0xb663e0 con 0xca66b0 2023-10-06T03:30:40.264 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.256+0000 7fc077fff700 1 --2- 172.21.15.110:0/2008003426 >> [v2:172.21.15.110:6836/1379510962,v1:172.21.15.110:6837/1379510962] conn(0xca66b0 0xca8a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:40.265 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.256+0000 7fc077fff700 1 --2- 172.21.15.110:0/2008003426 >> [v2:172.21.15.110:6836/1379510962,v1:172.21.15.110:6837/1379510962] conn(0xca66b0 0xca8a90 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ba56226a70853e9c server_cookie=730556d5fafc5a16 in_seq=0 out_seq=0 2023-10-06T03:30:40.266 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.256+0000 7fc05dffb700 10 client.5352 ms_handle_connect on v2:172.21.15.110:6836/1379510962 2023-10-06T03:30:40.266 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.256+0000 7fc05dffb700 1 -- 172.21.15.110:0/2008003426 <== mds.0 v2:172.21.15.110:6836/1379510962 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+16954 (crc 0 0 0) 0xb663e0 con 0xca66b0 2023-10-06T03:30:40.266 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.256+0000 7fc05dffb700 10 client.5352 handle_command_reply: tid=0 2023-10-06T03:30:40.266 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.256+0000 7fc088186740 1 client.5352 shutdown 2023-10-06T03:30:40.266 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.256+0000 7fc088186740 1 -- 172.21.15.110:0/2008003426 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0xa1c710 msgr2=0xc56740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:40.266 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.256+0000 7fc088186740 1 --2- 172.21.15.110:0/2008003426 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0xa1c710 0xc56740 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7fc0700137a0 tx=0x7fc070014000 comp rx=0 tx=0).stop 2023-10-06T03:30:40.266 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.256+0000 7fc088186740 1 -- 172.21.15.110:0/2008003426 shutdown_connections 2023-10-06T03:30:40.266 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.256+0000 7fc088186740 1 --2- 172.21.15.110:0/2008003426 >> [v2:172.21.15.110:6836/1379510962,v1:172.21.15.110:6837/1379510962] conn(0xca66b0 0xca8a90 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:40.266 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.256+0000 7fc088186740 1 --2- 172.21.15.110:0/2008003426 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0xa67790 0xc80140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:40.266 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.256+0000 7fc05dffb700 0 client.5352 ms_handle_reset on v2:172.21.15.110:6836/1379510962 2023-10-06T03:30:40.267 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.256+0000 7fc088186740 1 --2- 172.21.15.110:0/2008003426 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0xa1cae0 0xc7fc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:40.267 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.256+0000 7fc088186740 1 --2- 172.21.15.110:0/2008003426 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0xa1c710 0xc56740 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:40.267 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.256+0000 7fc088186740 1 -- 172.21.15.110:0/2008003426 >> 172.21.15.110:0/2008003426 conn(0xa1ceb0 msgr2=0xc4eed0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:30:40.267 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.256+0000 7fc088186740 1 -- 172.21.15.110:0/2008003426 shutdown_connections 2023-10-06T03:30:40.267 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.256+0000 7fc088186740 1 -- 172.21.15.110:0/2008003426 wait complete. 2023-10-06T03:30:40.267 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.256+0000 7fc088186740 20 client.5352 trim_cache size 0 max 16384 2023-10-06T03:30:40.280 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.276+0000 7fc088186740 1 Processor -- start 2023-10-06T03:30:40.280 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.276+0000 7fc088186740 1 -- start start 2023-10-06T03:30:40.281 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.276+0000 7fc088186740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0xa1c710 0xca8a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:40.282 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.276+0000 7fc088186740 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0xa1cae0 0xca8fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:40.282 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.276+0000 7fc088186740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0xa67790 0xca9510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:40.282 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.276+0000 7fc088186740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0xbc8490 con 0xa67790 2023-10-06T03:30:40.282 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.276+0000 7fc088186740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0xbc1890 con 0xa1c710 2023-10-06T03:30:40.282 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.276+0000 7fc088186740 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0xbc8ef0 con 0xa1cae0 2023-10-06T03:30:40.282 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.276+0000 7fc07ffff700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0xa67790 0xca9510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:40.282 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.276+0000 7fc07ffff700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0xa67790 0xca9510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:48224/0 (socket says 172.21.15.110:48224) 2023-10-06T03:30:40.282 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.276+0000 7fc07ffff700 1 -- 172.21.15.110:0/868967658 learned_addr learned my addr 172.21.15.110:0/868967658 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:30:40.282 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.276+0000 7fc07f7fe700 1 --2- 172.21.15.110:0/868967658 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0xa1c710 0xca8a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:40.282 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.276+0000 7fc077fff700 1 --2- 172.21.15.110:0/868967658 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0xa1cae0 0xca8fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:40.283 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.276+0000 7fc07ffff700 1 -- 172.21.15.110:0/868967658 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0xa1cae0 msgr2=0xca8fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:30:40.283 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.276+0000 7fc07ffff700 1 --2- 172.21.15.110:0/868967658 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0xa1cae0 0xca8fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:40.284 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.276+0000 7fc07ffff700 1 -- 172.21.15.110:0/868967658 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0xa1c710 msgr2=0xca8a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:30:40.284 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.276+0000 7fc07ffff700 1 --2- 172.21.15.110:0/868967658 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0xa1c710 0xca8a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:40.284 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.276+0000 7fc07ffff700 1 -- 172.21.15.110:0/868967658 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0800af350 con 0xa67790 2023-10-06T03:30:40.284 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.276+0000 7fc07ffff700 1 --2- 172.21.15.110:0/868967658 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0xa67790 0xca9510 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7fc07804dc00 tx=0x7fc078054000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=27378599bf6ad665 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:30:40.284 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.276+0000 7fc0757fa700 1 -- 172.21.15.110:0/868967658 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc07806f070 con 0xa67790 2023-10-06T03:30:40.284 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.276+0000 7fc0757fa700 1 -- 172.21.15.110:0/868967658 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc07800c960 con 0xa67790 2023-10-06T03:30:40.284 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.276+0000 7fc0757fa700 1 -- 172.21.15.110:0/868967658 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc078066040 con 0xa67790 2023-10-06T03:30:40.285 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.280+0000 7fc088186740 1 -- 172.21.15.110:0/868967658 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0xa67790 msgr2=0xca9510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:40.285 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.280+0000 7fc088186740 1 --2- 172.21.15.110:0/868967658 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0xa67790 0xca9510 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7fc07804dc00 tx=0x7fc078054000 comp rx=0 tx=0).stop 2023-10-06T03:30:40.285 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.280+0000 7fc088186740 1 -- 172.21.15.110:0/868967658 shutdown_connections 2023-10-06T03:30:40.285 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.280+0000 7fc088186740 1 --2- 172.21.15.110:0/868967658 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0xa67790 0xca9510 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:40.285 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.280+0000 7fc088186740 1 --2- 172.21.15.110:0/868967658 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0xa1cae0 0xca8fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:40.285 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.280+0000 7fc088186740 1 --2- 172.21.15.110:0/868967658 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0xa1c710 0xca8a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:40.285 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.280+0000 7fc088186740 1 -- 172.21.15.110:0/868967658 >> 172.21.15.110:0/868967658 conn(0xa1ceb0 msgr2=0xc7dac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:30:40.286 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.280+0000 7fc088186740 1 -- 172.21.15.110:0/868967658 shutdown_connections 2023-10-06T03:30:40.286 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.280+0000 7fc088186740 1 -- 172.21.15.110:0/868967658 wait complete. 2023-10-06T03:30:40.286 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.280+0000 7fc088186740 1 -- start start 2023-10-06T03:30:40.286 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.280+0000 7fc088186740 1 Processor -- start 2023-10-06T03:30:40.289 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.280+0000 7fc088186740 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0xa1c710 0xcb3190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:40.289 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.284+0000 7fc088186740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0xa1cae0 0xcb36d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:40.289 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.284+0000 7fc088186740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0xa67790 0xcb3c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:40.289 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.284+0000 7fc088186740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0xb9c910 con 0xa67790 2023-10-06T03:30:40.289 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.284+0000 7fc088186740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0xb6d510 con 0xa1cae0 2023-10-06T03:30:40.289 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.284+0000 7fc088186740 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0xc138e0 con 0xa1c710 2023-10-06T03:30:40.289 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.284+0000 7fc088186740 10 client.0 _ll_register_callbacks cb 0xae00e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-06T03:30:40.289 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.284+0000 7fc07ffff700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0xa67790 0xcb3c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:40.289 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.284+0000 7fc077fff700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0xa1cae0 0xcb36d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:40.289 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.284+0000 7fc07f7fe700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0xa1c710 0xcb3190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:40.289 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.284+0000 7fc077fff700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0xa1cae0 0xcb36d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.110:48752/0 (socket says 172.21.15.110:48752) 2023-10-06T03:30:40.290 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.284+0000 7fc077fff700 1 -- 172.21.15.110:0/3530678346 learned_addr learned my addr 172.21.15.110:0/3530678346 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:30:40.290 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.284+0000 7fc07f7fe700 1 -- 172.21.15.110:0/3530678346 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0xa1cae0 msgr2=0xcb36d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:30:40.290 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.284+0000 7fc07f7fe700 1 --2- 172.21.15.110:0/3530678346 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0xa1cae0 0xcb36d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:40.290 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.284+0000 7fc07f7fe700 1 -- 172.21.15.110:0/3530678346 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0xa67790 msgr2=0xcb3c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:30:40.290 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.284+0000 7fc07f7fe700 1 --2- 172.21.15.110:0/3530678346 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0xa67790 0xcb3c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:40.290 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.284+0000 7fc07f7fe700 1 -- 172.21.15.110:0/3530678346 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0xa689e0 con 0xa1c710 2023-10-06T03:30:40.290 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.284+0000 7fc07f7fe700 1 --2- 172.21.15.110:0/3530678346 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0xa1c710 0xcb3190 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fc070001990 tx=0x7fc070018000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cf06995138602942 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:30:40.290 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.284+0000 7fc05dffb700 10 client.0 ms_handle_connect on v2:172.21.15.192:3301/0 2023-10-06T03:30:40.290 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.284+0000 7fc05dffb700 1 -- 172.21.15.110:0/3530678346 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc070001410 con 0xa1c710 2023-10-06T03:30:40.290 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.284+0000 7fc088186740 1 -- 172.21.15.110:0/3530678346 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fc0780586e0 con 0xa1c710 2023-10-06T03:30:40.291 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.284+0000 7fc05dffb700 1 -- 172.21.15.110:0/3530678346 <== mon.2 v2:172.21.15.192:3301/0 2 ==== osd_map(42..42 src has 1..42) v4 ==== 6378+0+0 (secure 0 0 0) 0x7fc070004830 con 0xa1c710 2023-10-06T03:30:40.293 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.284+0000 7fc05dffb700 1 -- 172.21.15.110:0/3530678346 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_get_version_reply(handle=1 version=16) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc070004c60 con 0xa1c710 2023-10-06T03:30:40.293 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.284+0000 7fc088186740 10 client.5066 fetch_fsmap learned FSMap version 16 2023-10-06T03:30:40.293 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.284+0000 7fc088186740 1 -- 172.21.15.110:0/3530678346 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,fsmap=16,monmap=2+}) v3 -- 0xb78d80 con 0xa1c710 2023-10-06T03:30:40.293 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.284+0000 7fc05dffb700 1 -- 172.21.15.110:0/3530678346 <== mon.2 v2:172.21.15.192:3301/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0780586e0 con 0xa1c710 2023-10-06T03:30:40.293 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.284+0000 7fc05dffb700 1 -- 172.21.15.110:0/3530678346 <== mon.2 v2:172.21.15.192:3301/0 5 ==== fsmap(e 16) v1 ==== 3100+0+0 (secure 0 0 0) 0x7fc07002c3b0 con 0xa1c710 2023-10-06T03:30:40.293 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.284+0000 7fc088186740 10 client.5066 fetch_fsmap finished waiting for FSMap version 16 2023-10-06T03:30:40.293 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.284+0000 7fc088186740 10 client.5066 resolve_mds: resolved name 'a' to daemon mds.a 2023-10-06T03:30:40.293 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.284+0000 7fc088186740 20 client.5066 populate_metadata read hostname 'smithi110' 2023-10-06T03:30:40.293 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.284+0000 7fc088186740 1 --2- 172.21.15.110:0/3530678346 >> [v2:172.21.15.110:6836/1379510962,v1:172.21.15.110:6837/1379510962] conn(0xcfe160 0xd00540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:40.293 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.284+0000 7fc088186740 4 client.5066 mds_command: new command op to 5163 tid=0[{"prefix": "status"}] 2023-10-06T03:30:40.293 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.284+0000 7fc088186740 1 -- 172.21.15.110:0/3530678346 --> [v2:172.21.15.110:6836/1379510962,v1:172.21.15.110:6837/1379510962] -- command(tid 0: {"prefix": "status"}) v1 -- 0xb98410 con 0xcfe160 2023-10-06T03:30:40.294 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.284+0000 7fc077fff700 1 --2- 172.21.15.110:0/3530678346 >> [v2:172.21.15.110:6836/1379510962,v1:172.21.15.110:6837/1379510962] conn(0xcfe160 0xd00540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:40.294 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.284+0000 7fc05dffb700 10 client.5066 ms_handle_connect on v2:172.21.15.110:6836/1379510962 2023-10-06T03:30:40.294 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.284+0000 7fc077fff700 1 --2- 172.21.15.110:0/3530678346 >> [v2:172.21.15.110:6836/1379510962,v1:172.21.15.110:6837/1379510962] conn(0xcfe160 0xd00540 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9381eb831f147fb7 server_cookie=f18ca19449597c55 in_seq=0 out_seq=0 2023-10-06T03:30:40.294 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.288+0000 7fc05dffb700 1 -- 172.21.15.110:0/3530678346 <== mds.0 v2:172.21.15.110:6836/1379510962 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+325 (crc 0 0 0) 0xb98410 con 0xcfe160 2023-10-06T03:30:40.294 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.288+0000 7fc05dffb700 10 client.5066 handle_command_reply: tid=0 2023-10-06T03:30:40.294 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.288+0000 7fc088186740 1 client.5066 shutdown 2023-10-06T03:30:40.294 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.288+0000 7fc088186740 1 -- 172.21.15.110:0/3530678346 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0xa1c710 msgr2=0xcb3190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:40.294 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.288+0000 7fc088186740 1 --2- 172.21.15.110:0/3530678346 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0xa1c710 0xcb3190 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fc070001990 tx=0x7fc070018000 comp rx=0 tx=0).stop 2023-10-06T03:30:40.294 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.288+0000 7fc088186740 1 -- 172.21.15.110:0/3530678346 shutdown_connections 2023-10-06T03:30:40.294 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.288+0000 7fc088186740 1 --2- 172.21.15.110:0/3530678346 >> [v2:172.21.15.110:6836/1379510962,v1:172.21.15.110:6837/1379510962] conn(0xcfe160 0xd00540 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:40.295 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.288+0000 7fc088186740 1 --2- 172.21.15.110:0/3530678346 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0xa67790 0xcb3c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:40.295 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.288+0000 7fc088186740 1 --2- 172.21.15.110:0/3530678346 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0xa1cae0 0xcb36d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:40.295 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.288+0000 7fc088186740 1 --2- 172.21.15.110:0/3530678346 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0xa1c710 0xcb3190 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:40.295 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.288+0000 7fc05dffb700 0 client.5066 ms_handle_reset on v2:172.21.15.110:6836/1379510962 2023-10-06T03:30:40.295 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.288+0000 7fc088186740 1 -- 172.21.15.110:0/3530678346 >> 172.21.15.110:0/3530678346 conn(0xa1ceb0 msgr2=0xc4fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:30:40.295 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.288+0000 7fc088186740 1 -- 172.21.15.110:0/3530678346 shutdown_connections 2023-10-06T03:30:40.295 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.288+0000 7fc088186740 1 -- 172.21.15.110:0/3530678346 wait complete. 2023-10-06T03:30:40.295 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.288+0000 7fc088186740 20 client.5066 trim_cache size 0 max 16384 2023-10-06T03:30:40.295 INFO:teuthology.orchestra.run.smithi110.stdout:{ 2023-10-06T03:30:40.295 INFO:teuthology.orchestra.run.smithi110.stdout: "cluster_fsid": "93b8f6ac-4c92-4b45-b8e4-b59617024aa2", 2023-10-06T03:30:40.296 INFO:teuthology.orchestra.run.smithi110.stdout: "whoami": 0, 2023-10-06T03:30:40.296 INFO:teuthology.orchestra.run.smithi110.stdout: "id": 5163, 2023-10-06T03:30:40.296 INFO:teuthology.orchestra.run.smithi110.stdout: "want_state": "up:active", 2023-10-06T03:30:40.296 INFO:teuthology.orchestra.run.smithi110.stdout: "state": "up:active", 2023-10-06T03:30:40.296 INFO:teuthology.orchestra.run.smithi110.stdout: "fs_name": "cephfs", 2023-10-06T03:30:40.296 INFO:teuthology.orchestra.run.smithi110.stdout: "rank_uptime": 5.6236566769999996, 2023-10-06T03:30:40.296 INFO:teuthology.orchestra.run.smithi110.stdout: "mdsmap_epoch": 16, 2023-10-06T03:30:40.296 INFO:teuthology.orchestra.run.smithi110.stdout: "osdmap_epoch": 42, 2023-10-06T03:30:40.296 INFO:teuthology.orchestra.run.smithi110.stdout: "osdmap_epoch_barrier": 42, 2023-10-06T03:30:40.296 INFO:teuthology.orchestra.run.smithi110.stdout: "uptime": 13.597903911 2023-10-06T03:30:40.296 INFO:teuthology.orchestra.run.smithi110.stdout:} 2023-10-06T03:30:40.297 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.288+0000 7fc085edd700 1 -- 172.21.15.110:0/2769863201 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fc060060550 msgr2=0x7fc060062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:40.297 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.288+0000 7fc085edd700 1 --2- 172.21.15.110:0/2769863201 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fc060060550 0x7fc060062a00 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fc070004fd0 tx=0x7fc070007eb0 comp rx=0 tx=0).stop 2023-10-06T03:30:40.297 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.288+0000 7fc085edd700 1 -- 172.21.15.110:0/2769863201 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc080157130 msgr2=0x7fc0800aeab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:40.297 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.288+0000 7fc085edd700 1 --2- 172.21.15.110:0/2769863201 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc080157130 0x7fc0800aeab0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7fc078009e90 tx=0x7fc07800b040 comp rx=0 tx=0).stop 2023-10-06T03:30:40.298 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.288+0000 7fc085edd700 1 -- 172.21.15.110:0/2769863201 shutdown_connections 2023-10-06T03:30:40.298 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.288+0000 7fc085edd700 1 --2- 172.21.15.110:0/2769863201 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fc060060550 0x7fc060062a00 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:40.298 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.288+0000 7fc085edd700 1 --2- 172.21.15.110:0/2769863201 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc080157130 0x7fc0800aeab0 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:40.298 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.288+0000 7fc085edd700 1 --2- 172.21.15.110:0/2769863201 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc080156d60 0x7fc0800ad880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:40.298 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.288+0000 7fc085edd700 1 --2- 172.21.15.110:0/2769863201 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc080069b20 0x7fc080149a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:40.298 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.288+0000 7fc085edd700 1 -- 172.21.15.110:0/2769863201 >> 172.21.15.110:0/2769863201 conn(0x7fc0800cc710 msgr2=0x7fc08015a070 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:30:40.298 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.288+0000 7fc085edd700 1 -- 172.21.15.110:0/2769863201 shutdown_connections 2023-10-06T03:30:40.298 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.288+0000 7fc085edd700 1 -- 172.21.15.110:0/2769863201 wait complete. 2023-10-06T03:30:40.309 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-10-06T03:30:40.421 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.416+0000 7fd4a493d700 1 Processor -- start 2023-10-06T03:30:40.422 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.416+0000 7fd4a493d700 1 -- start start 2023-10-06T03:30:40.423 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.416+0000 7fd4a493d700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fd4a015d420 0x7fd4a00c7b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:40.423 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.416+0000 7fd4a493d700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fd4a0069d70 0x7fd4a00ca270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:40.423 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.416+0000 7fd4a493d700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd4a015d7f0 0x7fd4a00ca7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:40.424 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.416+0000 7fd4a493d700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd4a006aef0 con 0x7fd4a015d420 2023-10-06T03:30:40.424 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.416+0000 7fd4a493d700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd4a006a2e0 con 0x7fd4a0069d70 2023-10-06T03:30:40.424 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.416+0000 7fd4a493d700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4a006a5c0 con 0x7fd4a015d7f0 2023-10-06T03:30:40.424 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.416+0000 7fd49e59c700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fd4a015d420 0x7fd4a00c7b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:40.424 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.420+0000 7fd49e59c700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fd4a015d420 0x7fd4a00c7b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:48242/0 (socket says 172.21.15.110:48242) 2023-10-06T03:30:40.424 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.420+0000 7fd49e59c700 1 -- 172.21.15.110:0/2885295382 learned_addr learned my addr 172.21.15.110:0/2885295382 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:30:40.425 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.420+0000 7fd49dd9b700 1 --2- 172.21.15.110:0/2885295382 >> v2:172.21.15.192:3300/0 conn(0x7fd4a0069d70 0x7fd4a00ca270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:40.425 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.420+0000 7fd49ed9d700 1 --2- 172.21.15.110:0/2885295382 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd4a015d7f0 0x7fd4a00ca7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:40.426 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.420+0000 7fd49e59c700 1 -- 172.21.15.110:0/2885295382 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd4a015d7f0 msgr2=0x7fd4a00ca7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:30:40.427 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.420+0000 7fd49e59c700 1 --2- 172.21.15.110:0/2885295382 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd4a015d7f0 0x7fd4a00ca7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:40.427 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.420+0000 7fd49e59c700 1 -- 172.21.15.110:0/2885295382 >> v2:172.21.15.192:3300/0 conn(0x7fd4a0069d70 msgr2=0x7fd4a00ca270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:30:40.427 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.420+0000 7fd49e59c700 1 --2- 172.21.15.110:0/2885295382 >> v2:172.21.15.192:3300/0 conn(0x7fd4a0069d70 0x7fd4a00ca270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:40.427 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.420+0000 7fd49e59c700 1 -- 172.21.15.110:0/2885295382 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4a006a8a0 con 0x7fd4a015d420 2023-10-06T03:30:40.427 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.420+0000 7fd49dd9b700 1 --2- 172.21.15.110:0/2885295382 >> v2:172.21.15.192:3300/0 conn(0x7fd4a0069d70 0x7fd4a00ca270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-06T03:30:40.428 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.420+0000 7fd49e59c700 1 --2- 172.21.15.110:0/2885295382 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd4a015d420 0x7fd4a00c7b90 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7fd4a0067ed0 tx=0x7fd490009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=66c306c080d626cc server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:30:40.428 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.420+0000 7fd49d59a700 1 -- 172.21.15.110:0/2885295382 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd490009e70 con 0x7fd4a015d420 2023-10-06T03:30:40.428 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.420+0000 7fd49d59a700 1 -- 172.21.15.110:0/2885295382 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd49000b860 con 0x7fd4a015d420 2023-10-06T03:30:40.428 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.420+0000 7fd49d59a700 1 -- 172.21.15.110:0/2885295382 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd49000bc00 con 0x7fd4a015d420 2023-10-06T03:30:40.428 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.420+0000 7fd4a493d700 1 -- 172.21.15.110:0/2885295382 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd4a015d420 msgr2=0x7fd4a00c7b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:40.428 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.420+0000 7fd4a493d700 1 --2- 172.21.15.110:0/2885295382 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd4a015d420 0x7fd4a00c7b90 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7fd4a0067ed0 tx=0x7fd490009250 comp rx=0 tx=0).stop 2023-10-06T03:30:40.428 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.420+0000 7fd4a493d700 1 -- 172.21.15.110:0/2885295382 shutdown_connections 2023-10-06T03:30:40.429 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.420+0000 7fd4a493d700 1 --2- 172.21.15.110:0/2885295382 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd4a015d7f0 0x7fd4a00ca7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:40.429 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.420+0000 7fd4a493d700 1 --2- 172.21.15.110:0/2885295382 >> v2:172.21.15.192:3300/0 conn(0x7fd4a0069d70 0x7fd4a00ca270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:40.429 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.420+0000 7fd4a493d700 1 --2- 172.21.15.110:0/2885295382 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd4a015d420 0x7fd4a00c7b90 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:40.429 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.420+0000 7fd4a493d700 1 -- 172.21.15.110:0/2885295382 >> 172.21.15.110:0/2885295382 conn(0x7fd4a00cbd00 msgr2=0x7fd4a00cc100 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:30:40.429 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.420+0000 7fd4a493d700 1 -- 172.21.15.110:0/2885295382 shutdown_connections 2023-10-06T03:30:40.429 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.420+0000 7fd4a493d700 1 -- 172.21.15.110:0/2885295382 wait complete. 2023-10-06T03:30:40.431 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.424+0000 7fd4a493d700 1 Processor -- start 2023-10-06T03:30:40.432 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.424+0000 7fd4a493d700 1 -- start start 2023-10-06T03:30:40.432 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.424+0000 7fd4a493d700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd4a0069d70 0x7fd4a015bdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:40.432 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.424+0000 7fd4a493d700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd4a015d420 0x7fd4a00aeb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:40.432 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.424+0000 7fd4a493d700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd4a015d7f0 0x7fd4a00af060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:40.432 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.424+0000 7fd4a493d700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4a00af9e0 con 0x7fd4a015d420 2023-10-06T03:30:40.432 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.424+0000 7fd4a493d700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4a00b12f0 con 0x7fd4a0069d70 2023-10-06T03:30:40.432 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.424+0000 7fd4a493d700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4a0158a80 con 0x7fd4a015d7f0 2023-10-06T03:30:40.432 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.424+0000 7fd49dd9b700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd4a015d420 0x7fd4a00aeb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:40.432 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.424+0000 7fd49dd9b700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd4a015d420 0x7fd4a00aeb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:48256/0 (socket says 172.21.15.110:48256) 2023-10-06T03:30:40.432 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.424+0000 7fd49dd9b700 1 -- 172.21.15.110:0/1024640706 learned_addr learned my addr 172.21.15.110:0/1024640706 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:30:40.433 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.424+0000 7fd49e59c700 1 --2- 172.21.15.110:0/1024640706 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd4a0069d70 0x7fd4a015bdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:40.433 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.424+0000 7fd49ed9d700 1 --2- 172.21.15.110:0/1024640706 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd4a015d7f0 0x7fd4a00af060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:40.433 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.424+0000 7fd49dd9b700 1 -- 172.21.15.110:0/1024640706 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd4a015d7f0 msgr2=0x7fd4a00af060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:40.433 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.424+0000 7fd49dd9b700 1 --2- 172.21.15.110:0/1024640706 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd4a015d7f0 0x7fd4a00af060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:40.433 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.424+0000 7fd49dd9b700 1 -- 172.21.15.110:0/1024640706 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd4a0069d70 msgr2=0x7fd4a015bdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:40.433 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.424+0000 7fd49dd9b700 1 --2- 172.21.15.110:0/1024640706 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd4a0069d70 0x7fd4a015bdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:40.433 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.424+0000 7fd49dd9b700 1 -- 172.21.15.110:0/1024640706 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4900094f0 con 0x7fd4a015d420 2023-10-06T03:30:40.433 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.424+0000 7fd49ed9d700 1 --2- 172.21.15.110:0/1024640706 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd4a015d7f0 0x7fd4a00af060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-06T03:30:40.433 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.424+0000 7fd49dd9b700 1 --2- 172.21.15.110:0/1024640706 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd4a015d420 0x7fd4a00aeb20 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7fd49400e900 tx=0x7fd494008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:30:40.433 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.424+0000 7fd48e7fc700 1 -- 172.21.15.110:0/1024640706 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd494013410 con 0x7fd4a015d420 2023-10-06T03:30:40.433 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.424+0000 7fd48e7fc700 1 -- 172.21.15.110:0/1024640706 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd494013570 con 0x7fd4a015d420 2023-10-06T03:30:40.434 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.424+0000 7fd48e7fc700 1 -- 172.21.15.110:0/1024640706 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd494013910 con 0x7fd4a015d420 2023-10-06T03:30:40.434 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.424+0000 7fd4a493d700 1 -- 172.21.15.110:0/1024640706 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4a00af700 con 0x7fd4a015d420 2023-10-06T03:30:40.434 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.424+0000 7fd4a493d700 1 -- 172.21.15.110:0/1024640706 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4a00b0f80 con 0x7fd4a015d420 2023-10-06T03:30:40.434 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.428+0000 7fd4a493d700 1 -- 172.21.15.110:0/1024640706 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd4a00b0b00 con 0x7fd4a015d420 2023-10-06T03:30:40.435 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.428+0000 7fd48e7fc700 1 -- 172.21.15.110:0/1024640706 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7fd494013d00 con 0x7fd4a015d420 2023-10-06T03:30:40.435 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.428+0000 7fd48e7fc700 1 --2- 172.21.15.110:0/1024640706 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fd480060550 0x7fd480062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:40.435 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.428+0000 7fd48e7fc700 1 -- 172.21.15.110:0/1024640706 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6378+0+0 (secure 0 0 0) 0x7fd49407c780 con 0x7fd4a015d420 2023-10-06T03:30:40.435 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.428+0000 7fd49e59c700 1 --2- 172.21.15.110:0/1024640706 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fd480060550 0x7fd480062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:40.435 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.428+0000 7fd49e59c700 1 --2- 172.21.15.110:0/1024640706 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fd480060550 0x7fd480062a00 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fd4a0067ed0 tx=0x7fd490017000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:30:40.439 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.432+0000 7fd48e7fc700 1 -- 172.21.15.110:0/1024640706 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fd4a00b0b00 con 0x7fd4a015d420 2023-10-06T03:30:40.632 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.628+0000 7fd4a493d700 1 -- 172.21.15.110:0/1024640706 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fd4a00b0b00 con 0x7fd4a015d420 2023-10-06T03:30:40.633 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.628+0000 7fd48e7fc700 1 -- 172.21.15.110:0/1024640706 <== mon.0 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v16) v1 ==== 53+0+73 (secure 0 0 0) 0x7fd4a00b0b00 con 0x7fd4a015d420 2023-10-06T03:30:40.633 INFO:teuthology.orchestra.run.smithi110.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-10-06T03:30:40.635 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.628+0000 7fd4a493d700 1 -- 172.21.15.110:0/1024640706 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fd480060550 msgr2=0x7fd480062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:40.635 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.628+0000 7fd4a493d700 1 --2- 172.21.15.110:0/1024640706 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fd480060550 0x7fd480062a00 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fd4a0067ed0 tx=0x7fd490017000 comp rx=0 tx=0).stop 2023-10-06T03:30:40.635 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.628+0000 7fd4a493d700 1 -- 172.21.15.110:0/1024640706 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd4a015d420 msgr2=0x7fd4a00aeb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:40.635 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.628+0000 7fd4a493d700 1 --2- 172.21.15.110:0/1024640706 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd4a015d420 0x7fd4a00aeb20 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7fd49400e900 tx=0x7fd494008040 comp rx=0 tx=0).stop 2023-10-06T03:30:40.635 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.628+0000 7fd4a493d700 1 -- 172.21.15.110:0/1024640706 shutdown_connections 2023-10-06T03:30:40.636 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.628+0000 7fd4a493d700 1 --2- 172.21.15.110:0/1024640706 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fd480060550 0x7fd480062a00 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:40.636 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.628+0000 7fd4a493d700 1 --2- 172.21.15.110:0/1024640706 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd4a015d7f0 0x7fd4a00af060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:40.636 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.628+0000 7fd4a493d700 1 --2- 172.21.15.110:0/1024640706 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd4a015d420 0x7fd4a00aeb20 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:40.636 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.628+0000 7fd4a493d700 1 --2- 172.21.15.110:0/1024640706 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd4a0069d70 0x7fd4a015bdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:40.636 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.628+0000 7fd4a493d700 1 -- 172.21.15.110:0/1024640706 >> 172.21.15.110:0/1024640706 conn(0x7fd4a00cbd00 msgr2=0x7fd4a00c1340 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:30:40.636 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.628+0000 7fd4a493d700 1 -- 172.21.15.110:0/1024640706 shutdown_connections 2023-10-06T03:30:40.636 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:40.628+0000 7fd4a493d700 1 -- 172.21.15.110:0/1024640706 wait complete. 2023-10-06T03:30:40.646 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2023-10-06T03:30:40.647 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-10-06T03:30:40.647 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi110.front.sepia.ceph.com 2023-10-06T03:30:40.647 INFO:tasks.cephfs.mount:self.client.name = client.0 2023-10-06T03:30:40.647 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-10-06T03:30:40.647 INFO:tasks.cephfs.mount:self.cephfs_name = None 2023-10-06T03:30:40.647 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2023-10-06T03:30:40.647 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-10-06T03:30:40.647 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-10-06T03:30:40.647 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-06T03:30:40.648 DEBUG:teuthology.orchestra.run.smithi110:> ip addr 2023-10-06T03:30:40.653 INFO:teuthology.orchestra.run.smithi110.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-10-06T03:30:40.653 INFO:teuthology.orchestra.run.smithi110.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-10-06T03:30:40.654 INFO:teuthology.orchestra.run.smithi110.stdout: inet 127.0.0.1/8 scope host lo 2023-10-06T03:30:40.654 INFO:teuthology.orchestra.run.smithi110.stdout: valid_lft forever preferred_lft forever 2023-10-06T03:30:40.654 INFO:teuthology.orchestra.run.smithi110.stdout: inet6 ::1/128 scope host 2023-10-06T03:30:40.654 INFO:teuthology.orchestra.run.smithi110.stdout: valid_lft forever preferred_lft forever 2023-10-06T03:30:40.654 INFO:teuthology.orchestra.run.smithi110.stdout:2: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2023-10-06T03:30:40.654 INFO:teuthology.orchestra.run.smithi110.stdout: link/ether 0c:c4:7a:d8:db:14 brd ff:ff:ff:ff:ff:ff 2023-10-06T03:30:40.654 INFO:teuthology.orchestra.run.smithi110.stdout:3: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000 2023-10-06T03:30:40.654 INFO:teuthology.orchestra.run.smithi110.stdout: link/ether 0c:c4:7a:d8:db:15 brd ff:ff:ff:ff:ff:ff 2023-10-06T03:30:40.654 INFO:teuthology.orchestra.run.smithi110.stdout:4: enp3s0f0: mtu 1500 qdisc noop state DOWN group default qlen 1000 2023-10-06T03:30:40.655 INFO:teuthology.orchestra.run.smithi110.stdout: link/ether 0c:c4:7a:8f:cc:34 brd ff:ff:ff:ff:ff:ff 2023-10-06T03:30:40.655 INFO:teuthology.orchestra.run.smithi110.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2023-10-06T03:30:40.655 INFO:teuthology.orchestra.run.smithi110.stdout: link/ether 0c:c4:7a:8f:cc:35 brd ff:ff:ff:ff:ff:ff 2023-10-06T03:30:40.655 INFO:teuthology.orchestra.run.smithi110.stdout: inet 172.21.15.110/20 brd 172.21.15.255 scope global dynamic enp3s0f1 2023-10-06T03:30:40.655 INFO:teuthology.orchestra.run.smithi110.stdout: valid_lft 42549sec preferred_lft 42549sec 2023-10-06T03:30:40.655 INFO:teuthology.orchestra.run.smithi110.stdout: inet6 fe80::ec4:7aff:fe8f:cc35/64 scope link 2023-10-06T03:30:40.655 INFO:teuthology.orchestra.run.smithi110.stdout: valid_lft forever preferred_lft forever 2023-10-06T03:30:40.655 INFO:teuthology.orchestra.run.smithi110.stdout:6: docker0: mtu 1500 qdisc noqueue state DOWN group default 2023-10-06T03:30:40.655 INFO:teuthology.orchestra.run.smithi110.stdout: link/ether 02:42:dc:de:aa:fb brd ff:ff:ff:ff:ff:ff 2023-10-06T03:30:40.656 INFO:teuthology.orchestra.run.smithi110.stdout: inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 2023-10-06T03:30:40.656 INFO:teuthology.orchestra.run.smithi110.stdout: valid_lft forever preferred_lft forever 2023-10-06T03:30:40.656 INFO:teuthology.orchestra.run.smithi110.stdout:9: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-10-06T03:30:40.656 INFO:teuthology.orchestra.run.smithi110.stdout: link/ether 5e:32:2b:b4:0f:e6 brd ff:ff:ff:ff:ff:ff 2023-10-06T03:30:40.656 INFO:teuthology.orchestra.run.smithi110.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-10-06T03:30:40.656 INFO:teuthology.orchestra.run.smithi110.stdout: valid_lft forever preferred_lft forever 2023-10-06T03:30:40.656 INFO:teuthology.orchestra.run.smithi110.stdout: inet6 fe80::34e3:85ff:feb6:e924/64 scope link 2023-10-06T03:30:40.656 INFO:teuthology.orchestra.run.smithi110.stdout: valid_lft forever preferred_lft forever 2023-10-06T03:30:40.656 INFO:teuthology.orchestra.run.smithi110.stdout:10: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-10-06T03:30:40.656 INFO:teuthology.orchestra.run.smithi110.stdout: link/ether 5e:32:2b:b4:0f:e6 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2023-10-06T03:30:40.657 INFO:teuthology.orchestra.run.smithi110.stdout: inet6 fe80::5c32:2bff:feb4:fe6/64 scope link 2023-10-06T03:30:40.657 INFO:teuthology.orchestra.run.smithi110.stdout: valid_lft forever preferred_lft forever 2023-10-06T03:30:40.657 INFO:teuthology.orchestra.run.smithi110.stdout:11: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-10-06T03:30:40.657 INFO:teuthology.orchestra.run.smithi110.stdout: link/ether 76:f9:1f:3d:03:15 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2023-10-06T03:30:40.657 INFO:teuthology.orchestra.run.smithi110.stdout: inet6 fe80::74f9:1fff:fe3d:315/64 scope link 2023-10-06T03:30:40.657 INFO:teuthology.orchestra.run.smithi110.stdout: valid_lft forever preferred_lft forever 2023-10-06T03:30:40.657 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-06T03:30:40.657 DEBUG:teuthology.orchestra.run.smithi110:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-10-06T03:30:40.710 INFO:teuthology.orchestra.run.smithi110.stdout:1 2023-10-06T03:30:40.711 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-06T03:30:40.711 DEBUG:teuthology.orchestra.run.smithi110:> route 2023-10-06T03:30:40.720 INFO:teuthology.orchestra.run.smithi110.stdout:Kernel IP routing table 2023-10-06T03:30:40.720 INFO:teuthology.orchestra.run.smithi110.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-10-06T03:30:40.720 INFO:teuthology.orchestra.run.smithi110.stdout:default _gateway 0.0.0.0 UG 0 0 0 enp3s0f1 2023-10-06T03:30:40.720 INFO:teuthology.orchestra.run.smithi110.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2023-10-06T03:30:40.720 INFO:teuthology.orchestra.run.smithi110.stdout:172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 2023-10-06T03:30:40.720 INFO:teuthology.orchestra.run.smithi110.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 0 0 0 enp3s0f1 2023-10-06T03:30:40.720 INFO:teuthology.orchestra.run.smithi110.stdout:_gateway 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2023-10-06T03:30:40.720 INFO:teuthology.orchestra.run.smithi110.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-10-06T03:30:40.721 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-06T03:30:40.721 DEBUG:teuthology.orchestra.run.smithi110:> (cd / && exec bash -c ' 2023-10-06T03:30:40.721 DEBUG:teuthology.orchestra.run.smithi110:> set -e 2023-10-06T03:30:40.721 DEBUG:teuthology.orchestra.run.smithi110:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2023-10-06T03:30:40.721 DEBUG:teuthology.orchestra.run.smithi110:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2023-10-06T03:30:40.721 DEBUG:teuthology.orchestra.run.smithi110:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2023-10-06T03:30:40.721 DEBUG:teuthology.orchestra.run.smithi110:> ') 2023-10-06T03:30:40.762 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-06T03:30:40.762 DEBUG:teuthology.orchestra.run.smithi110:> ip netns list 2023-10-06T03:30:40.769 INFO:teuthology.orchestra.run.smithi110.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2023-10-06T03:30:40.769 INFO:teuthology.orchestra.run.smithi110.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2023-10-06T03:30:40.769 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-10-06T03:30:40.770 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-06T03:30:40.770 DEBUG:teuthology.orchestra.run.smithi110:> sudo ip link set brx.0 up 2023-10-06T03:30:40.785 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2023-10-06T03:30:40.785 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-06T03:30:40.785 DEBUG:teuthology.orchestra.run.smithi110:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2023-10-06T03:30:40.838 INFO:teuthology.orchestra.run.smithi110.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2023-10-06T03:30:40.838 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-06T03:30:40.838 DEBUG:teuthology.orchestra.run.smithi110:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2023-10-06T03:30:40.844 DEBUG:teuthology.orchestra.run.smithi110:> sudo modprobe fuse 2023-10-06T03:30:40.861 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-10-06T03:30:40.861 DEBUG:teuthology.orchestra.run.smithi110:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-10-06T03:30:40.918 INFO:teuthology.orchestra.run.smithi110.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-10-06T03:30:40.919 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-10-06T03:30:40.920 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-06T03:30:40.920 DEBUG:teuthology.orchestra.run.smithi110:> ls /sys/fs/fuse/connections 2023-10-06T03:30:40.924 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2023-10-06T03:30:40.924 DEBUG:teuthology.orchestra.run.smithi110:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2023-10-06T03:30:40.927 DEBUG:teuthology.orchestra.run.smithi110:> sudo modprobe fuse 2023-10-06T03:30:40.945 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-10-06T03:30:40.945 DEBUG:teuthology.orchestra.run.smithi110:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-10-06T03:30:40.955 INFO:teuthology.orchestra.run.smithi110.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-10-06T03:30:40.956 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-10-06T03:30:40.957 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-06T03:30:40.957 DEBUG:teuthology.orchestra.run.smithi110:> ls /sys/fs/fuse/connections 2023-10-06T03:30:41.007 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi110.stderr:2023-10-06T03:30:41.000+0000 7f9717a1f380 -1 init, newargv = 0x55f49f687ac0 newargc=15 2023-10-06T03:30:41.007 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi110.stderr:ceph-fuse[42294]: starting ceph client 2023-10-06T03:30:41.016 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi110.stderr:ceph-fuse[42294]: starting fuse 2023-10-06T03:30:42.063 DEBUG:teuthology.orchestra.run.smithi110:> sudo modprobe fuse 2023-10-06T03:30:42.080 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-10-06T03:30:42.080 DEBUG:teuthology.orchestra.run.smithi110:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-10-06T03:30:42.139 INFO:teuthology.orchestra.run.smithi110.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-10-06T03:30:42.139 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-10-06T03:30:42.140 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-06T03:30:42.140 DEBUG:teuthology.orchestra.run.smithi110:> ls /sys/fs/fuse/connections 2023-10-06T03:30:42.147 INFO:teuthology.orchestra.run.smithi110.stdout:52 2023-10-06T03:30:42.147 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [52] 2023-10-06T03:30:42.148 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2023-10-06T03:30:42.148 DEBUG:teuthology.orchestra.run.smithi110:> import glob 2023-10-06T03:30:42.148 DEBUG:teuthology.orchestra.run.smithi110:> import re 2023-10-06T03:30:42.148 DEBUG:teuthology.orchestra.run.smithi110:> import os 2023-10-06T03:30:42.148 DEBUG:teuthology.orchestra.run.smithi110:> import subprocess 2023-10-06T03:30:42.148 DEBUG:teuthology.orchestra.run.smithi110:> 2023-10-06T03:30:42.148 DEBUG:teuthology.orchestra.run.smithi110:> def _find_admin_socket(client_name): 2023-10-06T03:30:42.148 DEBUG:teuthology.orchestra.run.smithi110:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2023-10-06T03:30:42.148 DEBUG:teuthology.orchestra.run.smithi110:> files = glob.glob(asok_path) 2023-10-06T03:30:42.148 DEBUG:teuthology.orchestra.run.smithi110:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2023-10-06T03:30:42.148 DEBUG:teuthology.orchestra.run.smithi110:> 2023-10-06T03:30:42.148 DEBUG:teuthology.orchestra.run.smithi110:> # Given a non-glob path, it better be there 2023-10-06T03:30:42.148 DEBUG:teuthology.orchestra.run.smithi110:> if "*" not in asok_path: 2023-10-06T03:30:42.149 DEBUG:teuthology.orchestra.run.smithi110:> assert(len(files) == 1) 2023-10-06T03:30:42.149 DEBUG:teuthology.orchestra.run.smithi110:> return files[0] 2023-10-06T03:30:42.149 DEBUG:teuthology.orchestra.run.smithi110:> 2023-10-06T03:30:42.149 DEBUG:teuthology.orchestra.run.smithi110:> for f in files: 2023-10-06T03:30:42.149 DEBUG:teuthology.orchestra.run.smithi110:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2023-10-06T03:30:42.149 DEBUG:teuthology.orchestra.run.smithi110:> if os.path.exists("/proc/{0}".format(pid)): 2023-10-06T03:30:42.149 DEBUG:teuthology.orchestra.run.smithi110:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2023-10-06T03:30:42.149 DEBUG:teuthology.orchestra.run.smithi110:> contents = proc_f.read() 2023-10-06T03:30:42.149 DEBUG:teuthology.orchestra.run.smithi110:> if mountpoint in contents: 2023-10-06T03:30:42.149 DEBUG:teuthology.orchestra.run.smithi110:> return f 2023-10-06T03:30:42.149 DEBUG:teuthology.orchestra.run.smithi110:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2023-10-06T03:30:42.149 DEBUG:teuthology.orchestra.run.smithi110:> 2023-10-06T03:30:42.149 DEBUG:teuthology.orchestra.run.smithi110:> print(_find_admin_socket("client.0")) 2023-10-06T03:30:42.150 DEBUG:teuthology.orchestra.run.smithi110:> ' 2023-10-06T03:30:42.234 INFO:teuthology.orchestra.run.smithi110.stdout:/var/run/ceph/ceph-client.0.42294.asok 2023-10-06T03:30:42.417 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.42294.asok 2023-10-06T03:30:42.417 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-06T03:30:42.418 DEBUG:teuthology.orchestra.run.smithi110:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.42294.asok status 2023-10-06T03:30:42.506 INFO:teuthology.orchestra.run.smithi110.stdout:{ 2023-10-06T03:30:42.506 INFO:teuthology.orchestra.run.smithi110.stdout: "metadata": { 2023-10-06T03:30:42.506 INFO:teuthology.orchestra.run.smithi110.stdout: "ceph_sha1": "736ac5ba72cb47af7d6f6666b1e100f7e20a20e2", 2023-10-06T03:30:42.506 INFO:teuthology.orchestra.run.smithi110.stdout: "ceph_version": "ceph version 17.2.6-1296-g736ac5ba (736ac5ba72cb47af7d6f6666b1e100f7e20a20e2) quincy (stable)", 2023-10-06T03:30:42.506 INFO:teuthology.orchestra.run.smithi110.stdout: "entity_id": "0", 2023-10-06T03:30:42.507 INFO:teuthology.orchestra.run.smithi110.stdout: "hostname": "smithi110", 2023-10-06T03:30:42.507 INFO:teuthology.orchestra.run.smithi110.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2023-10-06T03:30:42.507 INFO:teuthology.orchestra.run.smithi110.stdout: "pid": "42294", 2023-10-06T03:30:42.507 INFO:teuthology.orchestra.run.smithi110.stdout: "root": "/" 2023-10-06T03:30:42.507 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2023-10-06T03:30:42.507 INFO:teuthology.orchestra.run.smithi110.stdout: "dentry_count": 0, 2023-10-06T03:30:42.507 INFO:teuthology.orchestra.run.smithi110.stdout: "dentry_pinned_count": 0, 2023-10-06T03:30:42.507 INFO:teuthology.orchestra.run.smithi110.stdout: "id": 5370, 2023-10-06T03:30:42.507 INFO:teuthology.orchestra.run.smithi110.stdout: "inst": { 2023-10-06T03:30:42.507 INFO:teuthology.orchestra.run.smithi110.stdout: "name": { 2023-10-06T03:30:42.507 INFO:teuthology.orchestra.run.smithi110.stdout: "type": "client", 2023-10-06T03:30:42.508 INFO:teuthology.orchestra.run.smithi110.stdout: "num": 5370 2023-10-06T03:30:42.508 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2023-10-06T03:30:42.508 INFO:teuthology.orchestra.run.smithi110.stdout: "addr": { 2023-10-06T03:30:42.508 INFO:teuthology.orchestra.run.smithi110.stdout: "type": "v1", 2023-10-06T03:30:42.508 INFO:teuthology.orchestra.run.smithi110.stdout: "addr": "192.168.0.1:0", 2023-10-06T03:30:42.508 INFO:teuthology.orchestra.run.smithi110.stdout: "nonce": 1298704808 2023-10-06T03:30:42.508 INFO:teuthology.orchestra.run.smithi110.stdout: } 2023-10-06T03:30:42.508 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2023-10-06T03:30:42.508 INFO:teuthology.orchestra.run.smithi110.stdout: "addr": { 2023-10-06T03:30:42.508 INFO:teuthology.orchestra.run.smithi110.stdout: "type": "v1", 2023-10-06T03:30:42.508 INFO:teuthology.orchestra.run.smithi110.stdout: "addr": "192.168.0.1:0", 2023-10-06T03:30:42.509 INFO:teuthology.orchestra.run.smithi110.stdout: "nonce": 1298704808 2023-10-06T03:30:42.509 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2023-10-06T03:30:42.509 INFO:teuthology.orchestra.run.smithi110.stdout: "inst_str": "client.5370 192.168.0.1:0/1298704808", 2023-10-06T03:30:42.509 INFO:teuthology.orchestra.run.smithi110.stdout: "addr_str": "192.168.0.1:0/1298704808", 2023-10-06T03:30:42.509 INFO:teuthology.orchestra.run.smithi110.stdout: "inode_count": 1, 2023-10-06T03:30:42.509 INFO:teuthology.orchestra.run.smithi110.stdout: "mds_epoch": 16, 2023-10-06T03:30:42.509 INFO:teuthology.orchestra.run.smithi110.stdout: "osd_epoch": 42, 2023-10-06T03:30:42.509 INFO:teuthology.orchestra.run.smithi110.stdout: "osd_epoch_barrier": 0, 2023-10-06T03:30:42.509 INFO:teuthology.orchestra.run.smithi110.stdout: "blocklisted": false, 2023-10-06T03:30:42.509 INFO:teuthology.orchestra.run.smithi110.stdout: "fs_name": "cephfs" 2023-10-06T03:30:42.509 INFO:teuthology.orchestra.run.smithi110.stdout:} 2023-10-06T03:30:42.514 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-06T03:30:42.514 DEBUG:teuthology.orchestra.run.smithi110:> stat --file-system '--printf=%T 2023-10-06T03:30:42.514 DEBUG:teuthology.orchestra.run.smithi110:> ' -- /home/ubuntu/cephtest/mnt.0 2023-10-06T03:30:42.520 INFO:teuthology.orchestra.run.smithi110.stdout:fuseblk 2023-10-06T03:30:42.520 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2023-10-06T03:30:42.520 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-06T03:30:42.520 DEBUG:teuthology.orchestra.run.smithi110:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2023-10-06T03:30:42.536 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2023-10-06T03:30:42.641 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:42.636+0000 7f715090a700 1 Processor -- start 2023-10-06T03:30:42.642 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:42.636+0000 7f715090a700 1 -- start start 2023-10-06T03:30:42.644 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:42.640+0000 7f715090a700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f714c15ad60 0x7f714c15b130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:42.644 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:42.640+0000 7f715090a700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f714c0ae4b0 con 0x7f714c0adc30 2023-10-06T03:30:42.644 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:42.640+0000 7f715090a700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f714c0ac570 con 0x7f714c0ae080 2023-10-06T03:30:42.644 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:42.640+0000 7f715090a700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f714c15aa10 con 0x7f714c15ad60 2023-10-06T03:30:42.644 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:42.640+0000 7f7142ffd700 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f714c0adc30 0x7f714c0ab750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:55790/0 (socket says 172.21.15.110:55790) 2023-10-06T03:30:42.644 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:42.640+0000 7f7142ffd700 1 -- 172.21.15.110:0/472070919 learned_addr learned my addr 172.21.15.110:0/472070919 (peer_addr_for_me v1:172.21.15.110:0/0) 2023-10-06T03:30:42.645 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:42.640+0000 7f714b7fe700 1 --2- 172.21.15.110:0/472070919 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f714c15ad60 0x7f714c15b130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:42.646 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:42.640+0000 7f714affd700 1 -- 172.21.15.110:0/472070919 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1499690252 0 0) 0x7f714c0ae4b0 con 0x7f714c0adc30 2023-10-06T03:30:42.646 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:42.640+0000 7f714affd700 1 -- 172.21.15.110:0/472070919 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7138003570 con 0x7f714c0adc30 2023-10-06T03:30:42.646 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:42.640+0000 7f714affd700 1 -- 172.21.15.110:0/472070919 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4280745562 0 0) 0x7f714c0ac570 con 0x7f714c0ae080 2023-10-06T03:30:42.646 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:42.640+0000 7f714affd700 1 -- 172.21.15.110:0/472070919 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f714c0ae4b0 con 0x7f714c0ae080 2023-10-06T03:30:42.646 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:42.640+0000 7f714affd700 1 -- 172.21.15.110:0/472070919 <== mon.0 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1266010355 0 0) 0x7f7138003570 con 0x7f714c0adc30 2023-10-06T03:30:42.646 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:42.640+0000 7f714b7fe700 1 -- 172.21.15.110:0/472070919 >> v1:172.21.15.192:6789/0 conn(0x7f714c0ae080 legacy=0x7f714c0abe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:42.646 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:42.640+0000 7f714b7fe700 1 -- 172.21.15.110:0/472070919 >> v1:172.21.15.110:6789/0 conn(0x7f714c0adc30 legacy=0x7f714c0ab750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:42.647 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:42.640+0000 7f714b7fe700 1 -- 172.21.15.110:0/472070919 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f714c06c270 con 0x7f714c15ad60 2023-10-06T03:30:42.647 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:42.640+0000 7f714b7fe700 1 --2- 172.21.15.110:0/472070919 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f714c15ad60 0x7f714c15b130 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f713c009910 tx=0x7f713c0091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e7d956ef3c54f9a server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:30:42.647 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:42.640+0000 7f714affd700 1 -- 172.21.15.110:0/472070919 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7134001470 con 0x7f714c15ad60 2023-10-06T03:30:42.647 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:42.640+0000 7f714affd700 1 -- 172.21.15.110:0/472070919 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f713c011420 con 0x7f714c15ad60 2023-10-06T03:30:42.647 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:42.640+0000 7f714affd700 1 -- 172.21.15.110:0/472070919 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f713c0117a0 con 0x7f714c15ad60 2023-10-06T03:30:42.648 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:42.640+0000 7f715090a700 1 -- 172.21.15.110:0/472070919 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f714c15ad60 msgr2=0x7f714c15b130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:42.648 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:42.640+0000 7f715090a700 1 --2- 172.21.15.110:0/472070919 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f714c15ad60 0x7f714c15b130 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f713c009910 tx=0x7f713c0091a0 comp rx=0 tx=0).stop 2023-10-06T03:30:42.648 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:42.640+0000 7f715090a700 1 -- 172.21.15.110:0/472070919 shutdown_connections 2023-10-06T03:30:42.648 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:42.640+0000 7f715090a700 1 --2- 172.21.15.110:0/472070919 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f714c15ad60 0x7f714c15b130 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:42.648 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:42.640+0000 7f715090a700 1 -- 172.21.15.110:0/472070919 >> 172.21.15.110:0/472070919 conn(0x7f714c06e760 msgr2=0x7f714c0cba30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:30:42.648 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:42.644+0000 7f715090a700 1 -- 172.21.15.110:0/472070919 shutdown_connections 2023-10-06T03:30:42.648 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:42.644+0000 7f715090a700 1 -- 172.21.15.110:0/472070919 wait complete. 2023-10-06T03:30:42.650 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:42.644+0000 7f715090a700 1 Processor -- start 2023-10-06T03:30:42.650 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:42.644+0000 7f715090a700 1 -- start start 2023-10-06T03:30:42.651 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:42.644+0000 7f715090a700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f714c0adc30 0x7f714c1671e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:42.651 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:42.644+0000 7f715090a700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f714c0ae080 0x7f714c06a240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:42.651 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:42.644+0000 7f715090a700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f714c15ad60 0x7f714c0c4900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:42.652 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:42.644+0000 7f715090a700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f714c0c2ec0 con 0x7f714c0adc30 2023-10-06T03:30:42.652 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:42.644+0000 7f715090a700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f714c0c31a0 con 0x7f714c0ae080 2023-10-06T03:30:42.652 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:42.644+0000 7f715090a700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f714c0c3480 con 0x7f714c15ad60 2023-10-06T03:30:42.652 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:42.644+0000 7f714b7fe700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f714c0adc30 0x7f714c1671e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:42.652 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:42.648+0000 7f714b7fe700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f714c0adc30 0x7f714c1671e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:47040/0 (socket says 172.21.15.110:47040) 2023-10-06T03:30:42.652 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:42.648+0000 7f714b7fe700 1 -- 172.21.15.110:0/2205532816 learned_addr learned my addr 172.21.15.110:0/2205532816 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:30:42.652 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:42.648+0000 7f7142ffd700 1 --2- 172.21.15.110:0/2205532816 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f714c0ae080 0x7f714c06a240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:42.652 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:42.648+0000 7f714bfff700 1 --2- 172.21.15.110:0/2205532816 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f714c15ad60 0x7f714c0c4900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:42.653 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:42.648+0000 7f714b7fe700 1 -- 172.21.15.110:0/2205532816 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f714c15ad60 msgr2=0x7f714c0c4900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:42.653 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:42.648+0000 7f714b7fe700 1 --2- 172.21.15.110:0/2205532816 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f714c15ad60 0x7f714c0c4900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:42.653 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:42.648+0000 7f714b7fe700 1 -- 172.21.15.110:0/2205532816 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f714c0ae080 msgr2=0x7f714c06a240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:42.653 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:42.648+0000 7f714b7fe700 1 --2- 172.21.15.110:0/2205532816 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f714c0ae080 0x7f714c06a240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:42.654 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:42.648+0000 7f714b7fe700 1 -- 172.21.15.110:0/2205532816 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f713c009420 con 0x7f714c0adc30 2023-10-06T03:30:42.654 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:42.648+0000 7f714b7fe700 1 --2- 172.21.15.110:0/2205532816 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f714c0adc30 0x7f714c1671e0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f713c009860 tx=0x7f713c016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:30:42.654 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:42.648+0000 7f7143fff700 1 -- 172.21.15.110:0/2205532816 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f713c01cb70 con 0x7f714c0adc30 2023-10-06T03:30:42.654 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:42.648+0000 7f7143fff700 1 -- 172.21.15.110:0/2205532816 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f713c01ccd0 con 0x7f714c0adc30 2023-10-06T03:30:42.654 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:42.648+0000 7f715090a700 1 -- 172.21.15.110:0/2205532816 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f714c0c3760 con 0x7f714c0adc30 2023-10-06T03:30:42.654 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:42.648+0000 7f7143fff700 1 -- 172.21.15.110:0/2205532816 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f713c01a6a0 con 0x7f714c0adc30 2023-10-06T03:30:42.654 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:42.648+0000 7f715090a700 1 -- 172.21.15.110:0/2205532816 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f714c0c2ae0 con 0x7f714c0adc30 2023-10-06T03:30:42.654 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:42.648+0000 7f715090a700 1 -- 172.21.15.110:0/2205532816 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f714c070ff0 con 0x7f714c0adc30 2023-10-06T03:30:42.656 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:42.652+0000 7f7143fff700 1 -- 172.21.15.110:0/2205532816 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7f713c01aad0 con 0x7f714c0adc30 2023-10-06T03:30:42.656 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:42.652+0000 7f7143fff700 1 --2- 172.21.15.110:0/2205532816 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f712c060550 0x7f712c062a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:42.656 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:42.652+0000 7f7143fff700 1 -- 172.21.15.110:0/2205532816 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6378+0+0 (secure 0 0 0) 0x7f713c07e600 con 0x7f714c0adc30 2023-10-06T03:30:42.657 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:42.652+0000 7f7142ffd700 1 --2- 172.21.15.110:0/2205532816 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f712c060550 0x7f712c062a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:42.657 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:42.652+0000 7f7142ffd700 1 --2- 172.21.15.110:0/2205532816 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f712c060550 0x7f712c062a00 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f714c0aae30 tx=0x7f7134000bc0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:30:42.661 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:42.656+0000 7f7143fff700 1 -- 172.21.15.110:0/2205532816 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f713c04e8f0 con 0x7f714c0adc30 2023-10-06T03:30:42.844 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:42.840+0000 7f715090a700 1 -- 172.21.15.110:0/2205532816 --> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f714c0bf5c0 con 0x7f712c060550 2023-10-06T03:30:42.963 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:42.840+0000 7f7143fff700 1 -- 172.21.15.110:0/2205532816 <== mgr.4110 v2:172.21.15.110:6832/33268 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7f714c0bf5c0 con 0x7f712c060550 2023-10-06T03:30:42.963 INFO:teuthology.orchestra.run.smithi110.stdout:[ 2023-10-06T03:30:42.963 INFO:teuthology.orchestra.run.smithi110.stdout: { 2023-10-06T03:30:42.963 INFO:teuthology.orchestra.run.smithi110.stdout: "name": "cephfs" 2023-10-06T03:30:42.963 INFO:teuthology.orchestra.run.smithi110.stdout: } 2023-10-06T03:30:42.963 INFO:teuthology.orchestra.run.smithi110.stdout:] 2023-10-06T03:30:42.965 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:42.844+0000 7f715090a700 1 -- 172.21.15.110:0/2205532816 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f712c060550 msgr2=0x7f712c062a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:42.965 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:42.844+0000 7f715090a700 1 --2- 172.21.15.110:0/2205532816 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f712c060550 0x7f712c062a00 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f714c0aae30 tx=0x7f7134000bc0 comp rx=0 tx=0).stop 2023-10-06T03:30:42.965 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:42.844+0000 7f715090a700 1 -- 172.21.15.110:0/2205532816 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f714c0adc30 msgr2=0x7f714c1671e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:42.965 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:42.844+0000 7f715090a700 1 --2- 172.21.15.110:0/2205532816 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f714c0adc30 0x7f714c1671e0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f713c009860 tx=0x7f713c016000 comp rx=0 tx=0).stop 2023-10-06T03:30:42.965 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:42.844+0000 7f715090a700 1 -- 172.21.15.110:0/2205532816 shutdown_connections 2023-10-06T03:30:42.965 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:42.844+0000 7f715090a700 1 --2- 172.21.15.110:0/2205532816 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f712c060550 0x7f712c062a00 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:42.966 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:42.844+0000 7f715090a700 1 --2- 172.21.15.110:0/2205532816 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f714c15ad60 0x7f714c0c4900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:42.966 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:42.844+0000 7f715090a700 1 --2- 172.21.15.110:0/2205532816 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f714c0ae080 0x7f714c06a240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:42.966 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:42.844+0000 7f715090a700 1 --2- 172.21.15.110:0/2205532816 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f714c0adc30 0x7f714c1671e0 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:42.966 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:42.844+0000 7f715090a700 1 -- 172.21.15.110:0/2205532816 >> 172.21.15.110:0/2205532816 conn(0x7f714c06e760 msgr2=0x7f714c0ab3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:30:42.966 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:42.844+0000 7f715090a700 1 -- 172.21.15.110:0/2205532816 shutdown_connections 2023-10-06T03:30:42.966 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:42.844+0000 7f715090a700 1 -- 172.21.15.110:0/2205532816 wait complete. 2023-10-06T03:30:42.966 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module enable snap_schedule 2023-10-06T03:30:43.089 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:43.084+0000 7f5b732e4700 1 Processor -- start 2023-10-06T03:30:43.089 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:43.084+0000 7f5b732e4700 1 -- start start 2023-10-06T03:30:43.091 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:43.084+0000 7f5b732e4700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f5b6c069090 0x7f5b6c0c9e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:43.091 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:43.084+0000 7f5b732e4700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5b6c0ce610 0x7f5b6c0c8780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:43.091 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:43.084+0000 7f5b732e4700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b6c0c8d80 con 0x7f5b6c0cc9f0 2023-10-06T03:30:43.091 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:43.084+0000 7f5b732e4700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5b6c0c9060 con 0x7f5b6c069090 2023-10-06T03:30:43.091 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:43.084+0000 7f5b732e4700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b6c0c9340 con 0x7f5b6c0ce610 2023-10-06T03:30:43.091 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:43.084+0000 7f5b71ae1700 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f5b6c0cc9f0 0x7f5b6c158b40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:55804/0 (socket says 172.21.15.110:55804) 2023-10-06T03:30:43.091 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:43.084+0000 7f5b71ae1700 1 -- 172.21.15.110:0/3296881910 learned_addr learned my addr 172.21.15.110:0/3296881910 (peer_addr_for_me v1:172.21.15.110:0/0) 2023-10-06T03:30:43.091 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:43.084+0000 7f5b722e2700 1 --2- 172.21.15.110:0/3296881910 >> v2:172.21.15.192:3300/0 conn(0x7f5b6c069090 0x7f5b6c0c9e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:43.091 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:43.084+0000 7f5b72ae3700 1 --2- 172.21.15.110:0/3296881910 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5b6c0ce610 0x7f5b6c0c8780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:43.092 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:43.084+0000 7f5b712e0700 1 -- 172.21.15.110:0/3296881910 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 127679853 0 0) 0x7f5b6c0c8d80 con 0x7f5b6c0cc9f0 2023-10-06T03:30:43.092 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:43.084+0000 7f5b712e0700 1 -- 172.21.15.110:0/3296881910 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5b58003570 con 0x7f5b6c0cc9f0 2023-10-06T03:30:43.093 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:43.084+0000 7f5b712e0700 1 -- 172.21.15.110:0/3296881910 <== mon.0 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3500385440 0 0) 0x7f5b58003570 con 0x7f5b6c0cc9f0 2023-10-06T03:30:43.093 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:43.084+0000 7f5b712e0700 1 -- 172.21.15.110:0/3296881910 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5b6c0ce610 msgr2=0x7f5b6c0c8780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:30:43.093 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:43.084+0000 7f5b712e0700 1 --2- 172.21.15.110:0/3296881910 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5b6c0ce610 0x7f5b6c0c8780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:43.093 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:43.084+0000 7f5b712e0700 1 -- 172.21.15.110:0/3296881910 >> v2:172.21.15.192:3300/0 conn(0x7f5b6c069090 msgr2=0x7f5b6c0c9e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:30:43.093 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:43.088+0000 7f5b712e0700 1 --2- 172.21.15.110:0/3296881910 >> v2:172.21.15.192:3300/0 conn(0x7f5b6c069090 0x7f5b6c0c9e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:43.094 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:43.088+0000 7f5b712e0700 1 -- 172.21.15.110:0/3296881910 --> v1:172.21.15.110:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b6c0c9620 con 0x7f5b6c0cc9f0 2023-10-06T03:30:43.094 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:43.088+0000 7f5b712e0700 1 -- 172.21.15.110:0/3296881910 <== mon.0 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 118603703 0 0) 0x7f5b5c001470 con 0x7f5b6c0cc9f0 2023-10-06T03:30:43.094 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:43.088+0000 7f5b712e0700 1 -- 172.21.15.110:0/3296881910 >> v1:172.21.15.110:6789/0 conn(0x7f5b6c0cc9f0 legacy=0x7f5b6c158b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:43.094 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:43.088+0000 7f5b712e0700 1 --2- 172.21.15.110:0/3296881910 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5b58003a60 0x7f5b58003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:43.094 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:43.088+0000 7f5b712e0700 1 --2- 172.21.15.110:0/3296881910 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5b6c0ce610 0x7f5b58004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:43.094 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:43.088+0000 7f5b712e0700 1 --2- 172.21.15.110:0/3296881910 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5b58004d50 0x7f5b580051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:43.094 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:43.088+0000 7f5b712e0700 1 -- 172.21.15.110:0/3296881910 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b6c0c9060 con 0x7f5b58003a60 2023-10-06T03:30:43.094 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:43.088+0000 7f5b712e0700 1 -- 172.21.15.110:0/3296881910 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b6c0c9340 con 0x7f5b58004d50 2023-10-06T03:30:43.094 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:43.088+0000 7f5b712e0700 1 -- 172.21.15.110:0/3296881910 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b5800ba60 con 0x7f5b6c0ce610 2023-10-06T03:30:43.094 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:43.088+0000 7f5b722e2700 1 --2- 172.21.15.110:0/3296881910 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5b58003a60 0x7f5b58003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:43.094 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:43.088+0000 7f5b71ae1700 1 --2- 172.21.15.110:0/3296881910 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5b6c0ce610 0x7f5b58004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:43.095 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:43.088+0000 7f5b72ae3700 1 --2- 172.21.15.110:0/3296881910 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5b58004d50 0x7f5b580051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:43.095 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:43.088+0000 7f5b722e2700 1 -- 172.21.15.110:0/3296881910 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5b6c0ce610 msgr2=0x7f5b58004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:30:43.095 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:43.088+0000 7f5b722e2700 1 --2- 172.21.15.110:0/3296881910 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5b6c0ce610 0x7f5b58004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:43.095 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:43.088+0000 7f5b722e2700 1 -- 172.21.15.110:0/3296881910 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5b58004d50 msgr2=0x7f5b580051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:30:43.095 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:43.088+0000 7f5b722e2700 1 --2- 172.21.15.110:0/3296881910 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5b58004d50 0x7f5b580051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:43.095 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:43.088+0000 7f5b722e2700 1 -- 172.21.15.110:0/3296881910 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b58003570 con 0x7f5b58003a60 2023-10-06T03:30:43.095 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:43.088+0000 7f5b722e2700 1 --2- 172.21.15.110:0/3296881910 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5b58003a60 0x7f5b58003e70 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f5b68009c00 tx=0x7f5b680091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4520c68cc114979e server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:30:43.095 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:43.088+0000 7f5b712e0700 1 -- 172.21.15.110:0/3296881910 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b6800e6b0 con 0x7f5b58003a60 2023-10-06T03:30:43.095 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:43.088+0000 7f5b712e0700 1 -- 172.21.15.110:0/3296881910 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b6800e810 con 0x7f5b58003a60 2023-10-06T03:30:43.095 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:43.088+0000 7f5b712e0700 1 -- 172.21.15.110:0/3296881910 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b6800ebd0 con 0x7f5b58003a60 2023-10-06T03:30:43.096 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:43.088+0000 7f5b732e4700 1 -- 172.21.15.110:0/3296881910 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5b58003a60 msgr2=0x7f5b58003e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:43.096 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:43.088+0000 7f5b732e4700 1 --2- 172.21.15.110:0/3296881910 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5b58003a60 0x7f5b58003e70 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f5b68009c00 tx=0x7f5b680091a0 comp rx=0 tx=0).stop 2023-10-06T03:30:43.096 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:43.088+0000 7f5b72ae3700 1 -- 172.21.15.110:0/3296881910 reap_dead start 2023-10-06T03:30:43.097 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:43.088+0000 7f5b732e4700 1 -- 172.21.15.110:0/3296881910 shutdown_connections 2023-10-06T03:30:43.097 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:43.088+0000 7f5b732e4700 1 -- 172.21.15.110:0/3296881910 >> 172.21.15.110:0/3296881910 conn(0x7f5b6c068230 msgr2=0x7f5b6c0cdcf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:30:43.097 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:43.088+0000 7f5b732e4700 1 -- 172.21.15.110:0/3296881910 shutdown_connections 2023-10-06T03:30:43.097 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:43.088+0000 7f5b732e4700 1 -- 172.21.15.110:0/3296881910 wait complete. 2023-10-06T03:30:43.097 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:43.092+0000 7f5b732e4700 1 Processor -- start 2023-10-06T03:30:43.097 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:43.092+0000 7f5b732e4700 1 -- start start 2023-10-06T03:30:43.098 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:43.092+0000 7f5b732e4700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5b6c153c70 0x7f5b6c16c790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:43.098 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:43.092+0000 7f5b732e4700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5b6c154040 0x7f5b6c16ccd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:43.098 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:43.092+0000 7f5b732e4700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5b6c1544e0 0x7f5b6c16d210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:43.098 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:43.092+0000 7f5b732e4700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b6c15ab70 con 0x7f5b6c1544e0 2023-10-06T03:30:43.098 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:43.092+0000 7f5b732e4700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b6c15ae50 con 0x7f5b6c154040 2023-10-06T03:30:43.099 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:43.092+0000 7f5b732e4700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b6c15b130 con 0x7f5b6c153c70 2023-10-06T03:30:43.099 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:43.092+0000 7f5b72ae3700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5b6c1544e0 0x7f5b6c16d210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:43.099 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:43.092+0000 7f5b72ae3700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5b6c1544e0 0x7f5b6c16d210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:47048/0 (socket says 172.21.15.110:47048) 2023-10-06T03:30:43.099 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:43.092+0000 7f5b72ae3700 1 -- 172.21.15.110:0/2766975387 learned_addr learned my addr 172.21.15.110:0/2766975387 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:30:43.099 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:43.092+0000 7f5b71ae1700 1 --2- 172.21.15.110:0/2766975387 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5b6c154040 0x7f5b6c16ccd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:43.099 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:43.092+0000 7f5b722e2700 1 --2- 172.21.15.110:0/2766975387 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5b6c153c70 0x7f5b6c16c790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:43.100 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:43.092+0000 7f5b72ae3700 1 -- 172.21.15.110:0/2766975387 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5b6c153c70 msgr2=0x7f5b6c16c790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:43.100 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:43.092+0000 7f5b72ae3700 1 --2- 172.21.15.110:0/2766975387 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5b6c153c70 0x7f5b6c16c790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:43.100 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:43.092+0000 7f5b72ae3700 1 -- 172.21.15.110:0/2766975387 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5b6c154040 msgr2=0x7f5b6c16ccd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:43.100 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:43.092+0000 7f5b72ae3700 1 --2- 172.21.15.110:0/2766975387 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5b6c154040 0x7f5b6c16ccd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:43.100 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:43.092+0000 7f5b72ae3700 1 -- 172.21.15.110:0/2766975387 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b6c15b410 con 0x7f5b6c1544e0 2023-10-06T03:30:43.100 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:43.092+0000 7f5b72ae3700 1 --2- 172.21.15.110:0/2766975387 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5b6c1544e0 0x7f5b6c16d210 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f5b6c0cee00 tx=0x7f5b6400b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:30:43.100 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:43.092+0000 7f5b627fc700 1 -- 172.21.15.110:0/2766975387 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b6400e040 con 0x7f5b6c1544e0 2023-10-06T03:30:43.101 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:43.092+0000 7f5b627fc700 1 -- 172.21.15.110:0/2766975387 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b6400f040 con 0x7f5b6c1544e0 2023-10-06T03:30:43.101 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:43.092+0000 7f5b732e4700 1 -- 172.21.15.110:0/2766975387 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b6c15a860 con 0x7f5b6c1544e0 2023-10-06T03:30:43.101 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:43.092+0000 7f5b627fc700 1 -- 172.21.15.110:0/2766975387 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b6400fd70 con 0x7f5b6c1544e0 2023-10-06T03:30:43.101 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:43.092+0000 7f5b732e4700 1 -- 172.21.15.110:0/2766975387 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b6c1691b0 con 0x7f5b6c1544e0 2023-10-06T03:30:43.102 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:43.096+0000 7f5b732e4700 1 -- 172.21.15.110:0/2766975387 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b6c15b600 con 0x7f5b6c1544e0 2023-10-06T03:30:43.102 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:43.096+0000 7f5b627fc700 1 -- 172.21.15.110:0/2766975387 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 79959+0+0 (secure 0 0 0) 0x7f5b64015640 con 0x7f5b6c1544e0 2023-10-06T03:30:43.102 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:43.096+0000 7f5b627fc700 1 --2- 172.21.15.110:0/2766975387 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f5b50060580 0x7f5b50062a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:43.102 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:43.096+0000 7f5b627fc700 1 -- 172.21.15.110:0/2766975387 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6378+0+0 (secure 0 0 0) 0x7f5b6407d2f0 con 0x7f5b6c1544e0 2023-10-06T03:30:43.102 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:43.096+0000 7f5b722e2700 1 --2- 172.21.15.110:0/2766975387 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f5b50060580 0x7f5b50062a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:43.102 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:43.096+0000 7f5b722e2700 1 --2- 172.21.15.110:0/2766975387 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f5b50060580 0x7f5b50062a30 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f5b6c06b210 tx=0x7f5b6801a000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:30:43.110 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:43.104+0000 7f5b627fc700 1 -- 172.21.15.110:0/2766975387 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f5b6404d640 con 0x7f5b6c1544e0 2023-10-06T03:30:43.311 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:43.304+0000 7f5b732e4700 1 -- 172.21.15.110:0/2766975387 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) v1 -- 0x7f5b6c15acb0 con 0x7f5b6c1544e0 2023-10-06T03:30:43.348 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:43.344+0000 7f5b627fc700 1 -- 172.21.15.110:0/2766975387 <== mon.0 v2:172.21.15.110:3300/0 7 ==== mgrmap(e 6) v1 ==== 79976+0+0 (secure 0 0 0) 0x7f5b6404da10 con 0x7f5b6c1544e0 2023-10-06T03:30:43.349 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:43.344+0000 7f5b627fc700 1 -- 172.21.15.110:0/2766975387 <== mon.0 v2:172.21.15.110:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v6) v1 ==== 92+0+0 (secure 0 0 0) 0x7f5b64049be0 con 0x7f5b6c1544e0 2023-10-06T03:30:43.353 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:43.348+0000 7f5b732e4700 1 -- 172.21.15.110:0/2766975387 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f5b50060580 msgr2=0x7f5b50062a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:43.353 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:43.348+0000 7f5b732e4700 1 --2- 172.21.15.110:0/2766975387 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f5b50060580 0x7f5b50062a30 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f5b6c06b210 tx=0x7f5b6801a000 comp rx=0 tx=0).stop 2023-10-06T03:30:43.353 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:43.348+0000 7f5b732e4700 1 -- 172.21.15.110:0/2766975387 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5b6c1544e0 msgr2=0x7f5b6c16d210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:43.354 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:43.348+0000 7f5b732e4700 1 --2- 172.21.15.110:0/2766975387 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5b6c1544e0 0x7f5b6c16d210 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f5b6c0cee00 tx=0x7f5b6400b040 comp rx=0 tx=0).stop 2023-10-06T03:30:43.354 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:43.348+0000 7f5b732e4700 1 -- 172.21.15.110:0/2766975387 shutdown_connections 2023-10-06T03:30:43.354 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:43.348+0000 7f5b732e4700 1 --2- 172.21.15.110:0/2766975387 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f5b50060580 0x7f5b50062a30 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:43.354 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:43.348+0000 7f5b732e4700 1 --2- 172.21.15.110:0/2766975387 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5b6c1544e0 0x7f5b6c16d210 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:43.354 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:43.348+0000 7f5b732e4700 1 --2- 172.21.15.110:0/2766975387 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5b6c154040 0x7f5b6c16ccd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:43.354 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:43.348+0000 7f5b732e4700 1 --2- 172.21.15.110:0/2766975387 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5b6c153c70 0x7f5b6c16c790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:43.354 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:43.348+0000 7f5b732e4700 1 -- 172.21.15.110:0/2766975387 >> 172.21.15.110:0/2766975387 conn(0x7f5b6c068230 msgr2=0x7f5b6c15eec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:30:43.354 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:43.348+0000 7f5b732e4700 1 -- 172.21.15.110:0/2766975387 shutdown_connections 2023-10-06T03:30:43.354 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:43.348+0000 7f5b732e4700 1 -- 172.21.15.110:0/2766975387 wait complete. 2023-10-06T03:30:43.367 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/allow_m_granularity True 2023-10-06T03:30:43.466 INFO:tasks.ceph.mgr.x.smithi192.stderr:2023-10-06T03:30:43.460+0000 7fd8f8763000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-06T03:30:43.468 INFO:tasks.ceph.mgr.y.smithi110.stderr:2023-10-06T03:30:43.464+0000 7fe58df28000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-06T03:30:43.470 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:43.464+0000 7f02ccd42700 1 Processor -- start 2023-10-06T03:30:43.471 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:43.464+0000 7f02ccd42700 1 -- start start 2023-10-06T03:30:43.473 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:43.468+0000 7f02ccd42700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f02c81559d0 0x7f02c8155da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:43.473 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:43.468+0000 7f02ccd42700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f02c806d4c0 0x7f02c80ca800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:43.473 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:43.468+0000 7f02ccd42700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f02c806c8d0 0x7f02c806ccf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:43.473 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:43.468+0000 7f02ccd42700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f02c80c8d40 con 0x7f02c806d4c0 2023-10-06T03:30:43.473 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:43.468+0000 7f02ccd42700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f02c806d230 con 0x7f02c81559d0 2023-10-06T03:30:43.473 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:43.468+0000 7f02ccd42700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02c80cb6c0 con 0x7f02c806c8d0 2023-10-06T03:30:43.473 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:43.468+0000 7f02bffff700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f02c806d4c0 0x7f02c80ca800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:43.473 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:43.468+0000 7f02bffff700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f02c806d4c0 0x7f02c80ca800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:47090/0 (socket says 172.21.15.110:47090) 2023-10-06T03:30:43.474 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:43.468+0000 7f02bffff700 1 -- 172.21.15.110:0/1433730063 learned_addr learned my addr 172.21.15.110:0/1433730063 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:30:43.474 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:43.468+0000 7f02c7fff700 1 --2- 172.21.15.110:0/1433730063 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f02c806c8d0 0x7f02c806ccf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:43.474 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:43.468+0000 7f02c77fe700 1 --2- 172.21.15.110:0/1433730063 >> v2:172.21.15.192:3300/0 conn(0x7f02c81559d0 0x7f02c8155da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:43.474 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:43.468+0000 7f02bffff700 1 -- 172.21.15.110:0/1433730063 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f02c806c8d0 msgr2=0x7f02c806ccf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:30:43.474 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:43.468+0000 7f02bffff700 1 --2- 172.21.15.110:0/1433730063 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f02c806c8d0 0x7f02c806ccf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:43.474 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:43.468+0000 7f02bffff700 1 -- 172.21.15.110:0/1433730063 >> v2:172.21.15.192:3300/0 conn(0x7f02c81559d0 msgr2=0x7f02c8155da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:30:43.474 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:43.468+0000 7f02bffff700 1 --2- 172.21.15.110:0/1433730063 >> v2:172.21.15.192:3300/0 conn(0x7f02c81559d0 0x7f02c8155da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:43.474 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:43.468+0000 7f02bffff700 1 -- 172.21.15.110:0/1433730063 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02c80cb100 con 0x7f02c806d4c0 2023-10-06T03:30:43.474 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:43.468+0000 7f02c7fff700 1 --2- 172.21.15.110:0/1433730063 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f02c806c8d0 0x7f02c806ccf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-06T03:30:43.474 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:43.468+0000 7f02bffff700 1 --2- 172.21.15.110:0/1433730063 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f02c806d4c0 0x7f02c80ca800 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f02c8069b50 tx=0x7f02b0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fbd619f0db2da71f server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:30:43.474 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:43.468+0000 7f02c6ffd700 1 -- 172.21.15.110:0/1433730063 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02b000b860 con 0x7f02c806d4c0 2023-10-06T03:30:43.475 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:43.468+0000 7f02c6ffd700 1 -- 172.21.15.110:0/1433730063 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02b000b9c0 con 0x7f02c806d4c0 2023-10-06T03:30:43.475 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:43.468+0000 7f02c6ffd700 1 -- 172.21.15.110:0/1433730063 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02b000bd60 con 0x7f02c806d4c0 2023-10-06T03:30:43.476 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:43.468+0000 7f02ccd42700 1 -- 172.21.15.110:0/1433730063 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f02c806d4c0 msgr2=0x7f02c80ca800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:43.476 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:43.468+0000 7f02ccd42700 1 --2- 172.21.15.110:0/1433730063 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f02c806d4c0 0x7f02c80ca800 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f02c8069b50 tx=0x7f02b0009250 comp rx=0 tx=0).stop 2023-10-06T03:30:43.476 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:43.468+0000 7f02ccd42700 1 -- 172.21.15.110:0/1433730063 shutdown_connections 2023-10-06T03:30:43.476 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:43.468+0000 7f02ccd42700 1 --2- 172.21.15.110:0/1433730063 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f02c806c8d0 0x7f02c806ccf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:43.476 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:43.468+0000 7f02ccd42700 1 --2- 172.21.15.110:0/1433730063 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f02c806d4c0 0x7f02c80ca800 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:43.476 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:43.468+0000 7f02ccd42700 1 --2- 172.21.15.110:0/1433730063 >> v2:172.21.15.192:3300/0 conn(0x7f02c81559d0 0x7f02c8155da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:43.476 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:43.468+0000 7f02ccd42700 1 -- 172.21.15.110:0/1433730063 >> 172.21.15.110:0/1433730063 conn(0x7f02c80670a0 msgr2=0x7f02c80674a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:30:43.476 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:43.468+0000 7f02ccd42700 1 -- 172.21.15.110:0/1433730063 shutdown_connections 2023-10-06T03:30:43.476 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:43.468+0000 7f02ccd42700 1 -- 172.21.15.110:0/1433730063 wait complete. 2023-10-06T03:30:43.476 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:43.472+0000 7f02ccd42700 1 Processor -- start 2023-10-06T03:30:43.477 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:43.472+0000 7f02ccd42700 1 -- start start 2023-10-06T03:30:43.479 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:43.472+0000 7f02ccd42700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f02c806c8d0 0x7f02c80b2620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:43.479 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:43.472+0000 7f02ccd42700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f02c806d4c0 0x7f02c80c46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:43.479 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:43.472+0000 7f02ccd42700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f02c81559d0 0x7f02c80b0b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:43.479 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:43.472+0000 7f02ccd42700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02c80c56c0 con 0x7f02c806d4c0 2023-10-06T03:30:43.479 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:43.472+0000 7f02ccd42700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02c80c4c30 con 0x7f02c81559d0 2023-10-06T03:30:43.479 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:43.472+0000 7f02ccd42700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02c8155fd0 con 0x7f02c806c8d0 2023-10-06T03:30:43.479 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:43.472+0000 7f02bffff700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f02c806d4c0 0x7f02c80c46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:43.480 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:43.472+0000 7f02c77fe700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f02c806c8d0 0x7f02c80b2620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:43.480 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:43.472+0000 7f02bffff700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f02c806d4c0 0x7f02c80c46f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:47100/0 (socket says 172.21.15.110:47100) 2023-10-06T03:30:43.481 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:43.472+0000 7f02bffff700 1 -- 172.21.15.110:0/560458066 learned_addr learned my addr 172.21.15.110:0/560458066 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:30:43.481 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:43.472+0000 7f02c77fe700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f02c806c8d0 0x7f02c80b2620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3301/0 says I am v2:172.21.15.110:48776/0 (socket says 172.21.15.110:48776) 2023-10-06T03:30:43.481 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:43.472+0000 7f02c7fff700 1 --2- 172.21.15.110:0/560458066 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f02c81559d0 0x7f02c80b0b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:43.481 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:43.472+0000 7f02bffff700 1 -- 172.21.15.110:0/560458066 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f02c806c8d0 msgr2=0x7f02c80b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:43.481 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:43.472+0000 7f02bffff700 1 --2- 172.21.15.110:0/560458066 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f02c806c8d0 0x7f02c80b2620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:43.481 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:43.472+0000 7f02bffff700 1 -- 172.21.15.110:0/560458066 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f02c81559d0 msgr2=0x7f02c80b0b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:43.481 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:43.472+0000 7f02bffff700 1 --2- 172.21.15.110:0/560458066 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f02c81559d0 0x7f02c80b0b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:43.481 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:43.472+0000 7f02bffff700 1 -- 172.21.15.110:0/560458066 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02b00094f0 con 0x7f02c806d4c0 2023-10-06T03:30:43.481 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:43.472+0000 7f02bffff700 1 --2- 172.21.15.110:0/560458066 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f02c806d4c0 0x7f02c80c46f0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f02b0009290 tx=0x7f02b0017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:30:43.482 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:43.472+0000 7f02bf7fe700 1 -- 172.21.15.110:0/560458066 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02b0011e40 con 0x7f02c806d4c0 2023-10-06T03:30:43.482 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:43.472+0000 7f02bf7fe700 1 -- 172.21.15.110:0/560458066 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02b001a450 con 0x7f02c806d4c0 2023-10-06T03:30:43.482 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:43.472+0000 7f02bf7fe700 1 -- 172.21.15.110:0/560458066 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02b001a810 con 0x7f02c806d4c0 2023-10-06T03:30:43.482 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:43.472+0000 7f02ccd42700 1 -- 172.21.15.110:0/560458066 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02c80c4e20 con 0x7f02c806d4c0 2023-10-06T03:30:43.482 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:43.472+0000 7f02ccd42700 1 -- 172.21.15.110:0/560458066 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02c80c53e0 con 0x7f02c806d4c0 2023-10-06T03:30:43.483 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:43.476+0000 7f02ccd42700 1 -- 172.21.15.110:0/560458066 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f02c8156140 con 0x7f02c806d4c0 2023-10-06T03:30:43.483 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:43.476+0000 7f02bf7fe700 1 -- 172.21.15.110:0/560458066 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 6) v1 ==== 79976+0+0 (secure 0 0 0) 0x7f02b001ab80 con 0x7f02c806d4c0 2023-10-06T03:30:43.483 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:43.476+0000 7f02bf7fe700 1 --2- 172.21.15.110:0/560458066 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f02a80605a0 0x7f02a8062a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:43.483 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:43.476+0000 7f02c77fe700 1 -- 172.21.15.110:0/560458066 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f02a80605a0 msgr2=0x7f02a8062a50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.110:6832/33268 2023-10-06T03:30:43.483 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:43.476+0000 7f02bf7fe700 1 -- 172.21.15.110:0/560458066 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6378+0+0 (secure 0 0 0) 0x7f02b007dfe0 con 0x7f02c806d4c0 2023-10-06T03:30:43.483 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:43.476+0000 7f02c77fe700 1 --2- 172.21.15.110:0/560458066 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f02a80605a0 0x7f02a8062a50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-06T03:30:43.487 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:43.484+0000 7f02bf7fe700 1 -- 172.21.15.110:0/560458066 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f02b004e330 con 0x7f02c806d4c0 2023-10-06T03:30:43.646 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:43.640+0000 7f02ccd42700 1 -- 172.21.15.110:0/560458066 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) v1 -- 0x7f02c8070ff0 con 0x7f02c806d4c0 2023-10-06T03:30:43.656 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:43.652+0000 7f02bf7fe700 1 -- 172.21.15.110:0/560458066 <== mon.0 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v11)=0 v11) v1 ==== 138+0+0 (secure 0 0 0) 0x7f02b0051950 con 0x7f02c806d4c0 2023-10-06T03:30:43.659 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:43.652+0000 7f02ccd42700 1 -- 172.21.15.110:0/560458066 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f02a80605a0 msgr2=0x7f02a8062a50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-06T03:30:43.659 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:43.652+0000 7f02ccd42700 1 --2- 172.21.15.110:0/560458066 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f02a80605a0 0x7f02a8062a50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:43.659 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:43.652+0000 7f02ccd42700 1 -- 172.21.15.110:0/560458066 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f02c806d4c0 msgr2=0x7f02c80c46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:43.659 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:43.652+0000 7f02ccd42700 1 --2- 172.21.15.110:0/560458066 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f02c806d4c0 0x7f02c80c46f0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f02b0009290 tx=0x7f02b0017000 comp rx=0 tx=0).stop 2023-10-06T03:30:43.659 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:43.652+0000 7f02ccd42700 1 -- 172.21.15.110:0/560458066 shutdown_connections 2023-10-06T03:30:43.659 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:43.652+0000 7f02ccd42700 1 --2- 172.21.15.110:0/560458066 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f02a80605a0 0x7f02a8062a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:43.659 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:43.652+0000 7f02ccd42700 1 --2- 172.21.15.110:0/560458066 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f02c81559d0 0x7f02c80b0b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:43.659 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:43.652+0000 7f02ccd42700 1 --2- 172.21.15.110:0/560458066 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f02c806d4c0 0x7f02c80c46f0 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:43.659 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:43.652+0000 7f02ccd42700 1 --2- 172.21.15.110:0/560458066 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f02c806c8d0 0x7f02c80b2620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:43.659 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:43.652+0000 7f02ccd42700 1 -- 172.21.15.110:0/560458066 >> 172.21.15.110:0/560458066 conn(0x7f02c80670a0 msgr2=0x7f02c814f970 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:30:43.660 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:43.652+0000 7f02ccd42700 1 -- 172.21.15.110:0/560458066 shutdown_connections 2023-10-06T03:30:43.660 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:43.652+0000 7f02ccd42700 1 -- 172.21.15.110:0/560458066 wait complete. 2023-10-06T03:30:43.671 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/dump_on_update True 2023-10-06T03:30:43.781 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:43.776+0000 7fd4e333c700 1 Processor -- start 2023-10-06T03:30:43.781 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:43.776+0000 7fd4e333c700 1 -- start start 2023-10-06T03:30:43.783 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:43.776+0000 7fd4e333c700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fd4dc1559d0 0x7fd4dc155da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:43.783 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:43.776+0000 7fd4e333c700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fd4dc06d4c0 0x7fd4dc0ca800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:43.783 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:43.776+0000 7fd4e333c700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd4dc06c8d0 0x7fd4dc06ccf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:43.784 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:43.776+0000 7fd4e333c700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd4dc0c8d40 con 0x7fd4dc1559d0 2023-10-06T03:30:43.784 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:43.776+0000 7fd4e333c700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd4dc06d230 con 0x7fd4dc06d4c0 2023-10-06T03:30:43.784 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:43.776+0000 7fd4e333c700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4dc0cb6c0 con 0x7fd4dc06c8d0 2023-10-06T03:30:43.784 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:43.776+0000 7fd4e233a700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fd4dc1559d0 0x7fd4dc155da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:43.784 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:43.776+0000 7fd4e233a700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fd4dc1559d0 0x7fd4dc155da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:47102/0 (socket says 172.21.15.110:47102) 2023-10-06T03:30:43.784 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:43.776+0000 7fd4e233a700 1 -- 172.21.15.110:0/28027235 learned_addr learned my addr 172.21.15.110:0/28027235 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:30:43.784 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:43.776+0000 7fd4e2b3b700 1 --2- 172.21.15.110:0/28027235 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd4dc06c8d0 0x7fd4dc06ccf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:43.784 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:43.776+0000 7fd4e1b39700 1 --2- 172.21.15.110:0/28027235 >> v2:172.21.15.192:3300/0 conn(0x7fd4dc06d4c0 0x7fd4dc0ca800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:43.784 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:43.776+0000 7fd4e233a700 1 -- 172.21.15.110:0/28027235 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd4dc06c8d0 msgr2=0x7fd4dc06ccf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:30:43.784 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:43.776+0000 7fd4e233a700 1 --2- 172.21.15.110:0/28027235 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd4dc06c8d0 0x7fd4dc06ccf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:43.784 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:43.776+0000 7fd4e233a700 1 -- 172.21.15.110:0/28027235 >> v2:172.21.15.192:3300/0 conn(0x7fd4dc06d4c0 msgr2=0x7fd4dc0ca800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:30:43.785 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:43.776+0000 7fd4e233a700 1 --2- 172.21.15.110:0/28027235 >> v2:172.21.15.192:3300/0 conn(0x7fd4dc06d4c0 0x7fd4dc0ca800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:43.785 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:43.776+0000 7fd4e233a700 1 -- 172.21.15.110:0/28027235 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4dc0cb100 con 0x7fd4dc1559d0 2023-10-06T03:30:43.785 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:43.780+0000 7fd4e233a700 1 --2- 172.21.15.110:0/28027235 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd4dc1559d0 0x7fd4dc155da0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7fd4dc0ccbe0 tx=0x7fd4d8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb735c17b77cbb00 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:30:43.785 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:43.780+0000 7fd4e1338700 1 -- 172.21.15.110:0/28027235 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4d8009e70 con 0x7fd4dc1559d0 2023-10-06T03:30:43.785 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:43.780+0000 7fd4e1338700 1 -- 172.21.15.110:0/28027235 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4d800b860 con 0x7fd4dc1559d0 2023-10-06T03:30:43.785 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:43.780+0000 7fd4e1338700 1 -- 172.21.15.110:0/28027235 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4d800bc00 con 0x7fd4dc1559d0 2023-10-06T03:30:43.786 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:43.780+0000 7fd4e333c700 1 -- 172.21.15.110:0/28027235 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd4dc1559d0 msgr2=0x7fd4dc155da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:43.786 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:43.780+0000 7fd4e333c700 1 --2- 172.21.15.110:0/28027235 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd4dc1559d0 0x7fd4dc155da0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7fd4dc0ccbe0 tx=0x7fd4d8009250 comp rx=0 tx=0).stop 2023-10-06T03:30:43.786 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:43.780+0000 7fd4e333c700 1 -- 172.21.15.110:0/28027235 shutdown_connections 2023-10-06T03:30:43.786 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:43.780+0000 7fd4e333c700 1 --2- 172.21.15.110:0/28027235 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd4dc06c8d0 0x7fd4dc06ccf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:43.786 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:43.780+0000 7fd4e333c700 1 --2- 172.21.15.110:0/28027235 >> v2:172.21.15.192:3300/0 conn(0x7fd4dc06d4c0 0x7fd4dc0ca800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:43.786 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:43.780+0000 7fd4e333c700 1 --2- 172.21.15.110:0/28027235 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd4dc1559d0 0x7fd4dc155da0 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:43.786 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:43.780+0000 7fd4e333c700 1 -- 172.21.15.110:0/28027235 >> 172.21.15.110:0/28027235 conn(0x7fd4dc0670a0 msgr2=0x7fd4dc0674a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:30:43.787 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:43.780+0000 7fd4e333c700 1 -- 172.21.15.110:0/28027235 shutdown_connections 2023-10-06T03:30:43.787 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:43.780+0000 7fd4e333c700 1 -- 172.21.15.110:0/28027235 wait complete. 2023-10-06T03:30:43.787 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:43.780+0000 7fd4e333c700 1 Processor -- start 2023-10-06T03:30:43.787 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:43.780+0000 7fd4e333c700 1 -- start start 2023-10-06T03:30:43.790 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:43.784+0000 7fd4e333c700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd4dc06c8d0 0x7fd4dc153690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:43.790 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:43.784+0000 7fd4e333c700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd4dc06d4c0 0x7fd4dc169040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:43.790 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:43.784+0000 7fd4e333c700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd4dc1559d0 0x7fd4dc0c26c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:43.790 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:43.784+0000 7fd4e333c700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4dc144fa0 con 0x7fd4dc06c8d0 2023-10-06T03:30:43.790 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:43.784+0000 7fd4e333c700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4dc165ff0 con 0x7fd4dc06d4c0 2023-10-06T03:30:43.790 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:43.784+0000 7fd4e333c700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4dc153bd0 con 0x7fd4dc1559d0 2023-10-06T03:30:43.790 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:43.784+0000 7fd4e233a700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd4dc06c8d0 0x7fd4dc153690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:43.790 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:43.784+0000 7fd4e233a700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd4dc06c8d0 0x7fd4dc153690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:47108/0 (socket says 172.21.15.110:47108) 2023-10-06T03:30:43.790 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:43.784+0000 7fd4e233a700 1 -- 172.21.15.110:0/379687122 learned_addr learned my addr 172.21.15.110:0/379687122 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:30:43.790 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:43.784+0000 7fd4e1b39700 1 --2- 172.21.15.110:0/379687122 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd4dc06d4c0 0x7fd4dc169040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:43.791 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:43.784+0000 7fd4e2b3b700 1 --2- 172.21.15.110:0/379687122 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd4dc1559d0 0x7fd4dc0c26c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:43.791 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:43.784+0000 7fd4e233a700 1 -- 172.21.15.110:0/379687122 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd4dc1559d0 msgr2=0x7fd4dc0c26c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:43.791 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:43.784+0000 7fd4e233a700 1 --2- 172.21.15.110:0/379687122 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd4dc1559d0 0x7fd4dc0c26c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:43.791 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:43.784+0000 7fd4e233a700 1 -- 172.21.15.110:0/379687122 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd4dc06d4c0 msgr2=0x7fd4dc169040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:43.791 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:43.784+0000 7fd4e233a700 1 --2- 172.21.15.110:0/379687122 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd4dc06d4c0 0x7fd4dc169040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:43.791 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:43.784+0000 7fd4e233a700 1 -- 172.21.15.110:0/379687122 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4d80094f0 con 0x7fd4dc06c8d0 2023-10-06T03:30:43.791 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:43.784+0000 7fd4e1b39700 1 --2- 172.21.15.110:0/379687122 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd4dc06d4c0 0x7fd4dc169040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-06T03:30:43.791 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:43.784+0000 7fd4e233a700 1 --2- 172.21.15.110:0/379687122 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd4dc06c8d0 0x7fd4dc153690 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7fd4dc070bb0 tx=0x7fd4d8018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:30:43.791 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:43.784+0000 7fd4d27fc700 1 -- 172.21.15.110:0/379687122 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4d800bbe0 con 0x7fd4dc06c8d0 2023-10-06T03:30:43.791 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:43.784+0000 7fd4d27fc700 1 -- 172.21.15.110:0/379687122 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4d8023450 con 0x7fd4dc06c8d0 2023-10-06T03:30:43.791 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:43.784+0000 7fd4e333c700 1 -- 172.21.15.110:0/379687122 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4dc169580 con 0x7fd4dc06c8d0 2023-10-06T03:30:43.792 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:43.784+0000 7fd4d27fc700 1 -- 172.21.15.110:0/379687122 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4d80237f0 con 0x7fd4dc06c8d0 2023-10-06T03:30:43.792 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:43.784+0000 7fd4e333c700 1 -- 172.21.15.110:0/379687122 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4dc1656c0 con 0x7fd4dc06c8d0 2023-10-06T03:30:43.792 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:43.784+0000 7fd4e333c700 1 -- 172.21.15.110:0/379687122 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd4dc0642e0 con 0x7fd4dc06c8d0 2023-10-06T03:30:43.793 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:43.788+0000 7fd4d27fc700 1 -- 172.21.15.110:0/379687122 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 6) v1 ==== 79976+0+0 (secure 0 0 0) 0x7fd4d801f070 con 0x7fd4dc06c8d0 2023-10-06T03:30:43.793 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:43.788+0000 7fd4d27fc700 1 --2- 172.21.15.110:0/379687122 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fd4c00605a0 0x7fd4c0062a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:43.793 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:43.788+0000 7fd4e1b39700 1 -- 172.21.15.110:0/379687122 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fd4c00605a0 msgr2=0x7fd4c0062a50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.110:6832/33268 2023-10-06T03:30:43.793 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:43.788+0000 7fd4d27fc700 1 -- 172.21.15.110:0/379687122 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6378+0+0 (secure 0 0 0) 0x7fd4d8082410 con 0x7fd4dc06c8d0 2023-10-06T03:30:43.793 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:43.788+0000 7fd4e1b39700 1 --2- 172.21.15.110:0/379687122 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fd4c00605a0 0x7fd4c0062a50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-06T03:30:43.798 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:43.792+0000 7fd4d27fc700 1 -- 172.21.15.110:0/379687122 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fd4dc0642e0 con 0x7fd4dc06c8d0 2023-10-06T03:30:43.885 INFO:tasks.ceph.mgr.x.smithi192.stderr:2023-10-06T03:30:43.880+0000 7fd8f8763000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-06T03:30:43.905 INFO:tasks.ceph.mgr.y.smithi110.stderr:2023-10-06T03:30:43.900+0000 7fe58df28000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-06T03:30:43.933 INFO:tasks.ceph.mgr.x.smithi192.stderr:2023-10-06T03:30:43.928+0000 7fd8f8763000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-06T03:30:43.954 INFO:tasks.ceph.mgr.y.smithi110.stderr:2023-10-06T03:30:43.948+0000 7fe58df28000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-06T03:30:43.957 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:43.952+0000 7fd4e333c700 1 -- 172.21.15.110:0/379687122 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) v1 -- 0x7fd4dc0642e0 con 0x7fd4dc06c8d0 2023-10-06T03:30:43.962 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:43.956+0000 7fd4d27fc700 1 -- 172.21.15.110:0/379687122 <== mon.0 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v12)=0 v12) v1 ==== 133+0+0 (secure 0 0 0) 0x7fd4dc0642e0 con 0x7fd4dc06c8d0 2023-10-06T03:30:43.965 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:43.960+0000 7fd4e333c700 1 -- 172.21.15.110:0/379687122 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fd4c00605a0 msgr2=0x7fd4c0062a50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-06T03:30:43.965 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:43.960+0000 7fd4e333c700 1 --2- 172.21.15.110:0/379687122 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fd4c00605a0 0x7fd4c0062a50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:43.965 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:43.960+0000 7fd4e333c700 1 -- 172.21.15.110:0/379687122 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd4dc06c8d0 msgr2=0x7fd4dc153690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:43.965 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:43.960+0000 7fd4e333c700 1 --2- 172.21.15.110:0/379687122 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd4dc06c8d0 0x7fd4dc153690 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7fd4dc070bb0 tx=0x7fd4d8018000 comp rx=0 tx=0).stop 2023-10-06T03:30:43.966 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:43.960+0000 7fd4e333c700 1 -- 172.21.15.110:0/379687122 shutdown_connections 2023-10-06T03:30:43.966 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:43.960+0000 7fd4e333c700 1 --2- 172.21.15.110:0/379687122 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fd4c00605a0 0x7fd4c0062a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:43.966 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:43.960+0000 7fd4e333c700 1 --2- 172.21.15.110:0/379687122 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd4dc1559d0 0x7fd4dc0c26c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:43.966 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:43.960+0000 7fd4e333c700 1 --2- 172.21.15.110:0/379687122 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd4dc06d4c0 0x7fd4dc169040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:43.966 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:43.960+0000 7fd4e333c700 1 --2- 172.21.15.110:0/379687122 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd4dc06c8d0 0x7fd4dc153690 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:43.966 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:43.960+0000 7fd4e333c700 1 -- 172.21.15.110:0/379687122 >> 172.21.15.110:0/379687122 conn(0x7fd4dc0670a0 msgr2=0x7fd4dc0c91d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:30:43.966 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:43.960+0000 7fd4e333c700 1 -- 172.21.15.110:0/379687122 shutdown_connections 2023-10-06T03:30:43.966 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:43.960+0000 7fd4e333c700 1 -- 172.21.15.110:0/379687122 wait complete. 2023-10-06T03:30:43.978 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-06T03:30:43.978 DEBUG:teuthology.orchestra.run.smithi110:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1) 2023-10-06T03:30:43.991 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add --fs cephfs snap_test_dir1 1M 2023-10-06T03:30:44.038 INFO:tasks.ceph.mgr.x.smithi192.stderr:2023-10-06T03:30:44.036+0000 7fd8f8763000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-06T03:30:44.062 INFO:tasks.ceph.mgr.y.smithi110.stderr:2023-10-06T03:30:44.056+0000 7fe58df28000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-06T03:30:44.148 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:44.144+0000 7f26609fa700 1 Processor -- start 2023-10-06T03:30:44.148 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:44.144+0000 7f26609fa700 1 -- start start 2023-10-06T03:30:44.151 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:44.144+0000 7f26609fa700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f265c0acaa0 0x7f265c15a410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:44.151 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:44.144+0000 7f26609fa700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f265c157430 0x7f265c06c860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:44.151 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:44.144+0000 7f26609fa700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f265c150770 con 0x7f265c0ace70 2023-10-06T03:30:44.151 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:44.144+0000 7f26609fa700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f265c06b7c0 con 0x7f265c157430 2023-10-06T03:30:44.151 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:44.144+0000 7f26609fa700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f265c06baa0 con 0x7f265c0acaa0 2023-10-06T03:30:44.151 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:44.144+0000 7f265b59e700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f265c157430 0x7f265c06c860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:44.151 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:44.144+0000 7f265b59e700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f265c157430 0x7f265c06c860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:47116/0 (socket says 172.21.15.110:47116) 2023-10-06T03:30:44.151 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:44.144+0000 7f265b59e700 1 -- 172.21.15.110:0/1761520571 learned_addr learned my addr 172.21.15.110:0/1761520571 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:30:44.152 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:44.144+0000 7f265ad9d700 1 --2- 172.21.15.110:0/1761520571 >> v2:172.21.15.192:3300/0 conn(0x7f265c0acaa0 0x7f265c15a410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:44.152 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:44.144+0000 7f265b59e700 1 -- 172.21.15.110:0/1761520571 >> v2:172.21.15.192:3300/0 conn(0x7f265c0acaa0 msgr2=0x7f265c15a410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:30:44.152 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:44.144+0000 7f265b59e700 1 --2- 172.21.15.110:0/1761520571 >> v2:172.21.15.192:3300/0 conn(0x7f265c0acaa0 0x7f265c15a410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:44.152 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:44.144+0000 7f265b59e700 1 -- 172.21.15.110:0/1761520571 >> v1:172.21.15.110:6789/0 conn(0x7f265c0ace70 legacy=0x7f265c0c8590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:44.152 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:44.144+0000 7f265b59e700 1 -- 172.21.15.110:0/1761520571 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f265c06bd80 con 0x7f265c157430 2023-10-06T03:30:44.152 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:44.144+0000 7f265b59e700 1 --2- 172.21.15.110:0/1761520571 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f265c157430 0x7f265c06c860 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f265c068910 tx=0x7f2654009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f52c2836777f37b server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:30:44.152 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:44.144+0000 7f2659d9b700 1 -- 172.21.15.110:0/1761520571 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f265400e070 con 0x7f265c157430 2023-10-06T03:30:44.152 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:44.144+0000 7f2659d9b700 1 -- 172.21.15.110:0/1761520571 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f265400ba50 con 0x7f265c157430 2023-10-06T03:30:44.153 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:44.144+0000 7f2659d9b700 1 -- 172.21.15.110:0/1761520571 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26540092e0 con 0x7f265c157430 2023-10-06T03:30:44.153 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:44.148+0000 7f26609fa700 1 -- 172.21.15.110:0/1761520571 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f265c157430 msgr2=0x7f265c06c860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:44.153 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:44.148+0000 7f26609fa700 1 --2- 172.21.15.110:0/1761520571 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f265c157430 0x7f265c06c860 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f265c068910 tx=0x7f2654009840 comp rx=0 tx=0).stop 2023-10-06T03:30:44.154 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:44.148+0000 7f26609fa700 1 -- 172.21.15.110:0/1761520571 shutdown_connections 2023-10-06T03:30:44.154 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:44.148+0000 7f26609fa700 1 --2- 172.21.15.110:0/1761520571 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f265c157430 0x7f265c06c860 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:44.154 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:44.148+0000 7f26609fa700 1 --2- 172.21.15.110:0/1761520571 >> v2:172.21.15.192:3300/0 conn(0x7f265c0acaa0 0x7f265c15a410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:44.154 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:44.148+0000 7f26609fa700 1 -- 172.21.15.110:0/1761520571 >> 172.21.15.110:0/1761520571 conn(0x7f265c06e520 msgr2=0x7f265c06e920 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:30:44.154 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:44.148+0000 7f26609fa700 1 -- 172.21.15.110:0/1761520571 shutdown_connections 2023-10-06T03:30:44.154 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:44.148+0000 7f26609fa700 1 -- 172.21.15.110:0/1761520571 wait complete. 2023-10-06T03:30:44.155 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:44.148+0000 7f26609fa700 1 Processor -- start 2023-10-06T03:30:44.155 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:44.148+0000 7f26609fa700 1 -- start start 2023-10-06T03:30:44.157 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:44.152+0000 7f26609fa700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f265c0acaa0 0x7f265c149e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:44.157 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:44.152+0000 7f26609fa700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f265c0ace70 0x7f265c14d080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:44.157 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:44.152+0000 7f26609fa700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f265c157430 0x7f265c069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:44.158 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:44.152+0000 7f26609fa700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f265c16cb30 con 0x7f265c0ace70 2023-10-06T03:30:44.158 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:44.152+0000 7f26609fa700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f265c16c200 con 0x7f265c0acaa0 2023-10-06T03:30:44.158 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:44.152+0000 7f26609fa700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f265c16c4e0 con 0x7f265c157430 2023-10-06T03:30:44.158 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:44.152+0000 7f265a59c700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f265c0ace70 0x7f265c14d080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:44.158 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:44.152+0000 7f265a59c700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f265c0ace70 0x7f265c14d080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:47118/0 (socket says 172.21.15.110:47118) 2023-10-06T03:30:44.158 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:44.152+0000 7f265a59c700 1 -- 172.21.15.110:0/198477447 learned_addr learned my addr 172.21.15.110:0/198477447 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:30:44.158 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:44.152+0000 7f265ad9d700 1 --2- 172.21.15.110:0/198477447 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f265c0acaa0 0x7f265c149e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:44.158 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:44.152+0000 7f265b59e700 1 --2- 172.21.15.110:0/198477447 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f265c157430 0x7f265c069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:44.158 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:44.152+0000 7f265a59c700 1 -- 172.21.15.110:0/198477447 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f265c157430 msgr2=0x7f265c069d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:44.158 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:44.152+0000 7f265a59c700 1 --2- 172.21.15.110:0/198477447 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f265c157430 0x7f265c069d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:44.158 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:44.152+0000 7f265a59c700 1 -- 172.21.15.110:0/198477447 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f265c0acaa0 msgr2=0x7f265c149e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:44.159 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:44.152+0000 7f265a59c700 1 --2- 172.21.15.110:0/198477447 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f265c0acaa0 0x7f265c149e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:44.159 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:44.152+0000 7f265a59c700 1 -- 172.21.15.110:0/198477447 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2654009ae0 con 0x7f265c0ace70 2023-10-06T03:30:44.159 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:44.152+0000 7f265b59e700 1 --2- 172.21.15.110:0/198477447 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f265c157430 0x7f265c069d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-06T03:30:44.159 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:44.152+0000 7f265a59c700 1 --2- 172.21.15.110:0/198477447 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f265c0ace70 0x7f265c14d080 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f265000c590 tx=0x7f2650007f50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:30:44.159 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:44.152+0000 7f2642ffd700 1 -- 172.21.15.110:0/198477447 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2650011590 con 0x7f265c0ace70 2023-10-06T03:30:44.159 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:44.152+0000 7f2642ffd700 1 -- 172.21.15.110:0/198477447 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2650011710 con 0x7f265c0ace70 2023-10-06T03:30:44.159 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:44.152+0000 7f26609fa700 1 -- 172.21.15.110:0/198477447 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f265c16c7c0 con 0x7f265c0ace70 2023-10-06T03:30:44.159 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:44.152+0000 7f2642ffd700 1 -- 172.21.15.110:0/198477447 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2650011ad0 con 0x7f265c0ace70 2023-10-06T03:30:44.159 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:44.152+0000 7f26609fa700 1 -- 172.21.15.110:0/198477447 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f265c153e80 con 0x7f265c0ace70 2023-10-06T03:30:44.160 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:44.156+0000 7f26609fa700 1 -- 172.21.15.110:0/198477447 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f265c16cce0 con 0x7f265c0ace70 2023-10-06T03:30:44.161 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:44.156+0000 7f2642ffd700 1 -- 172.21.15.110:0/198477447 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 6) v1 ==== 79976+0+0 (secure 0 0 0) 0x7f265001b3b0 con 0x7f265c0ace70 2023-10-06T03:30:44.161 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:44.156+0000 7f2642ffd700 1 --2- 172.21.15.110:0/198477447 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f26480605a0 0x7f2648062a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:44.161 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:44.156+0000 7f265ad9d700 1 -- 172.21.15.110:0/198477447 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f26480605a0 msgr2=0x7f2648062a50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.110:6832/33268 2023-10-06T03:30:44.162 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:44.156+0000 7f2642ffd700 1 -- 172.21.15.110:0/198477447 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6378+0+0 (secure 0 0 0) 0x7f265007bcd0 con 0x7f265c0ace70 2023-10-06T03:30:44.162 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:44.156+0000 7f265ad9d700 1 --2- 172.21.15.110:0/198477447 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f26480605a0 0x7f2648062a50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-06T03:30:44.169 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:44.164+0000 7f2642ffd700 1 -- 172.21.15.110:0/198477447 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f265c16cce0 con 0x7f265c0ace70 2023-10-06T03:30:44.232 INFO:tasks.ceph.mgr.x.smithi192.stderr:2023-10-06T03:30:44.228+0000 7fd8f8763000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-06T03:30:44.264 INFO:tasks.ceph.mgr.y.smithi110.stderr:2023-10-06T03:30:44.260+0000 7fe58df28000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-06T03:30:44.279 INFO:tasks.ceph.mgr.x.smithi192.stderr:2023-10-06T03:30:44.276+0000 7fd8f8763000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-06T03:30:44.313 INFO:tasks.ceph.mgr.y.smithi110.stderr:2023-10-06T03:30:44.308+0000 7fe58df28000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-06T03:30:44.346 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:44.340+0000 7f26609fa700 1 -- 172.21.15.110:0/198477447 --> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "snap_test_dir1", "snap_schedule": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7f265c150770 con 0x7f26480605a0 2023-10-06T03:30:44.363 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:44.360+0000 7f265ad9d700 1 -- 172.21.15.110:0/198477447 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f26480605a0 msgr2=0x7f2648062a50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.110:6832/33268 2023-10-06T03:30:44.364 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:44.360+0000 7f265ad9d700 1 --2- 172.21.15.110:0/198477447 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f26480605a0 0x7f2648062a50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-06T03:30:44.431 INFO:tasks.ceph.mgr.x.smithi192.stderr:2023-10-06T03:30:44.428+0000 7fd8f8763000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-06T03:30:44.463 INFO:tasks.ceph.mgr.y.smithi110.stderr:2023-10-06T03:30:44.460+0000 7fe58df28000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-06T03:30:44.486 INFO:tasks.ceph.mgr.x.smithi192.stderr:2023-10-06T03:30:44.484+0000 7fd8f8763000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-06T03:30:44.520 INFO:tasks.ceph.mgr.y.smithi110.stderr:2023-10-06T03:30:44.516+0000 7fe58df28000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-06T03:30:44.544 INFO:tasks.ceph.mgr.x.smithi192.stderr:2023-10-06T03:30:44.540+0000 7fd8f8763000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-06T03:30:44.578 INFO:tasks.ceph.mgr.y.smithi110.stderr:2023-10-06T03:30:44.572+0000 7fe58df28000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-06T03:30:44.767 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:44.764+0000 7f265ad9d700 1 -- 172.21.15.110:0/198477447 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f26480605a0 msgr2=0x7f2648062a50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.110:6832/33268 2023-10-06T03:30:44.768 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:44.764+0000 7f265ad9d700 1 --2- 172.21.15.110:0/198477447 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f26480605a0 0x7f2648062a50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-06T03:30:44.841 INFO:tasks.ceph.mgr.x.smithi192.stderr:2023-10-06T03:30:44.836+0000 7fd8f8763000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-06T03:30:44.874 INFO:tasks.ceph.mgr.y.smithi110.stderr:2023-10-06T03:30:44.868+0000 7fe58df28000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-06T03:30:44.951 INFO:tasks.ceph.mgr.x.smithi192.stderr:2023-10-06T03:30:44.948+0000 7fd8f8763000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-06T03:30:44.985 INFO:tasks.ceph.mgr.y.smithi110.stderr:2023-10-06T03:30:44.980+0000 7fe58df28000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-06T03:30:45.231 INFO:tasks.ceph.mgr.x.smithi192.stderr:2023-10-06T03:30:45.228+0000 7fd8f8763000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-06T03:30:45.265 INFO:tasks.ceph.mgr.y.smithi110.stderr:2023-10-06T03:30:45.260+0000 7fe58df28000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-06T03:30:45.390 INFO:tasks.ceph.mgr.x.smithi192.stderr:2023-10-06T03:30:45.384+0000 7fd8f8763000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-06T03:30:45.427 INFO:tasks.ceph.mgr.y.smithi110.stderr:2023-10-06T03:30:45.420+0000 7fe58df28000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-06T03:30:45.452 INFO:tasks.ceph.mgr.x.smithi192.stderr:2023-10-06T03:30:45.448+0000 7fd8f8763000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-06T03:30:45.490 INFO:tasks.ceph.mgr.y.smithi110.stderr:2023-10-06T03:30:45.484+0000 7fe58df28000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-06T03:30:45.502 INFO:tasks.ceph.mgr.x.smithi192.stderr:2023-10-06T03:30:45.500+0000 7fd8f8763000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-06T03:30:45.542 INFO:tasks.ceph.mgr.y.smithi110.stderr:2023-10-06T03:30:45.536+0000 7fe58df28000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-06T03:30:45.556 INFO:tasks.ceph.mgr.x.smithi192.stderr:2023-10-06T03:30:45.552+0000 7fd8f8763000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-06T03:30:45.571 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:45.568+0000 7f265ad9d700 1 -- 172.21.15.110:0/198477447 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f26480605a0 msgr2=0x7f2648062a50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.110:6832/33268 2023-10-06T03:30:45.572 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:45.568+0000 7f265ad9d700 1 --2- 172.21.15.110:0/198477447 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f26480605a0 0x7f2648062a50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-10-06T03:30:45.596 INFO:tasks.ceph.mgr.y.smithi110.stderr:2023-10-06T03:30:45.592+0000 7fe58df28000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-06T03:30:45.729 INFO:tasks.ceph.mgr.x.smithi192.stderr:2023-10-06T03:30:45.724+0000 7fd8f8763000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-06T03:30:45.768 INFO:tasks.ceph.mgr.y.smithi110.stderr:2023-10-06T03:30:45.764+0000 7fe58df28000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-06T03:30:45.783 INFO:tasks.ceph.mgr.x.smithi192.stderr:2023-10-06T03:30:45.776+0000 7fd8f8763000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-06T03:30:45.822 INFO:tasks.ceph.mgr.y.smithi110.stderr:2023-10-06T03:30:45.816+0000 7fe58df28000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-06T03:30:45.843 INFO:tasks.ceph.mgr.x.smithi192.stderr:2023-10-06T03:30:45.840+0000 7fd8f8763000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-06T03:30:45.884 INFO:tasks.ceph.mgr.y.smithi110.stderr:2023-10-06T03:30:45.880+0000 7fe58df28000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-06T03:30:45.988 INFO:tasks.ceph.mgr.x.smithi192.stderr:2023-10-06T03:30:45.984+0000 7fd8f8763000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-06T03:30:46.032 INFO:tasks.ceph.mgr.y.smithi110.stderr:2023-10-06T03:30:46.028+0000 7fe58df28000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-06T03:30:46.046 INFO:tasks.ceph.mgr.x.smithi192.stderr:2023-10-06T03:30:46.044+0000 7fd8f8763000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-06T03:30:46.091 INFO:tasks.ceph.mgr.y.smithi110.stderr:2023-10-06T03:30:46.084+0000 7fe58df28000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-06T03:30:46.634 INFO:tasks.ceph.mgr.x.smithi192.stderr:2023-10-06T03:30:46.628+0000 7fd8f8763000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-06T03:30:46.677 INFO:tasks.ceph.mgr.y.smithi110.stderr:2023-10-06T03:30:46.672+0000 7fe58df28000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-06T03:30:46.715 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:46.708+0000 7f2642ffd700 1 -- 172.21.15.110:0/198477447 <== mon.0 v2:172.21.15.110:3300/0 7 ==== mgrmap(e 7) v1 ==== 79687+0+0 (secure 0 0 0) 0x7f265004c3f0 con 0x7f265c0ace70 2023-10-06T03:30:46.715 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:46.708+0000 7f2642ffd700 1 -- 172.21.15.110:0/198477447 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f26480605a0 msgr2=0x7f2648062a50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-06T03:30:46.715 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:46.708+0000 7f2642ffd700 1 --2- 172.21.15.110:0/198477447 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f26480605a0 0x7f2648062a50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:46.865 INFO:tasks.ceph.mgr.y.smithi110.stderr:2023-10-06T03:30:46.860+0000 7fe54e8d3700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-06T03:30:46.865 INFO:tasks.ceph.mgr.y.smithi110.stderr:2023-10-06T03:30:46.860+0000 7fe54e8d3700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-06T03:30:46.866 INFO:tasks.ceph.mgr.y.smithi110.stderr:2023-10-06T03:30:46.860+0000 7fe54e8d3700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-06T03:30:46.866 INFO:tasks.ceph.mgr.y.smithi110.stderr:2023-10-06T03:30:46.860+0000 7fe54e8d3700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-06T03:30:46.866 INFO:tasks.ceph.mgr.y.smithi110.stderr:2023-10-06T03:30:46.860+0000 7fe54e8d3700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-06T03:30:46.866 INFO:tasks.ceph.mgr.y.smithi110.stderr:2023-10-06T03:30:46.860+0000 7fe5510d8700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-06T03:30:46.866 INFO:tasks.ceph.mgr.y.smithi110.stderr:2023-10-06T03:30:46.860+0000 7fe5510d8700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-06T03:30:46.867 INFO:tasks.ceph.mgr.y.smithi110.stderr:2023-10-06T03:30:46.860+0000 7fe5510d8700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-06T03:30:46.867 INFO:tasks.ceph.mgr.y.smithi110.stderr:2023-10-06T03:30:46.860+0000 7fe5510d8700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-06T03:30:46.867 INFO:tasks.ceph.mgr.y.smithi110.stderr:2023-10-06T03:30:46.860+0000 7fe5510d8700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-06T03:30:47.743 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:47.736+0000 7f2642ffd700 1 -- 172.21.15.110:0/198477447 <== mon.0 v2:172.21.15.110:3300/0 8 ==== mgrmap(e 8) v1 ==== 79814+0+0 (secure 0 0 0) 0x7f265004c3f0 con 0x7f265c0ace70 2023-10-06T03:30:47.743 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:47.736+0000 7f2642ffd700 1 --2- 172.21.15.110:0/198477447 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f26480605a0 0x7f2648062a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:47.743 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:47.736+0000 7f2642ffd700 1 -- 172.21.15.110:0/198477447 --> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "snap_test_dir1", "snap_schedule": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7f265c150770 con 0x7f26480605a0 2023-10-06T03:30:47.746 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:47.740+0000 7f265ad9d700 1 --2- 172.21.15.110:0/198477447 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f26480605a0 0x7f2648062a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:47.748 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:47.744+0000 7f265ad9d700 1 --2- 172.21.15.110:0/198477447 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f26480605a0 0x7f2648062a50 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f264c008010 tx=0x7f264c008040 comp rx=0 tx=0).ready entity=mgr.5329 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:30:47.874 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:47.868+0000 7f2642ffd700 1 -- 172.21.15.110:0/198477447 <== mgr.5329 v2:172.21.15.110:6832/33268 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f265c150770 con 0x7f26480605a0 2023-10-06T03:30:47.875 INFO:teuthology.orchestra.run.smithi110.stdout:Schedule set for path snap_test_dir1 2023-10-06T03:30:47.878 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:47.872+0000 7f26609fa700 1 -- 172.21.15.110:0/198477447 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f26480605a0 msgr2=0x7f2648062a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:47.878 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:47.872+0000 7f26609fa700 1 --2- 172.21.15.110:0/198477447 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f26480605a0 0x7f2648062a50 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f264c008010 tx=0x7f264c008040 comp rx=0 tx=0).stop 2023-10-06T03:30:47.878 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:47.872+0000 7f26609fa700 1 -- 172.21.15.110:0/198477447 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f265c0ace70 msgr2=0x7f265c14d080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:47.879 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:47.872+0000 7f26609fa700 1 --2- 172.21.15.110:0/198477447 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f265c0ace70 0x7f265c14d080 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f265000c590 tx=0x7f2650007f50 comp rx=0 tx=0).stop 2023-10-06T03:30:47.879 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:47.872+0000 7f26609fa700 1 -- 172.21.15.110:0/198477447 shutdown_connections 2023-10-06T03:30:47.879 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:47.872+0000 7f26609fa700 1 --2- 172.21.15.110:0/198477447 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f26480605a0 0x7f2648062a50 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:47.880 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:47.872+0000 7f26609fa700 1 --2- 172.21.15.110:0/198477447 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f265c157430 0x7f265c069d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:47.880 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:47.872+0000 7f26609fa700 1 --2- 172.21.15.110:0/198477447 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f265c0ace70 0x7f265c14d080 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:47.880 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:47.872+0000 7f26609fa700 1 --2- 172.21.15.110:0/198477447 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f265c0acaa0 0x7f265c149e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:47.880 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:47.872+0000 7f26609fa700 1 -- 172.21.15.110:0/198477447 >> 172.21.15.110:0/198477447 conn(0x7f265c06e520 msgr2=0x7f265c159c10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:30:47.880 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:47.872+0000 7f26609fa700 1 -- 172.21.15.110:0/198477447 shutdown_connections 2023-10-06T03:30:47.880 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:47.872+0000 7f26609fa700 1 -- 172.21.15.110:0/198477447 wait complete. 2023-10-06T03:30:47.897 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path snap_test_dir1 2023-10-06T03:30:47.898 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule retention add --fs cephfs snap_test_dir1 1M 2023-10-06T03:30:48.025 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:48.020+0000 7f509d81f700 1 Processor -- start 2023-10-06T03:30:48.025 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:48.020+0000 7f509d81f700 1 -- start start 2023-10-06T03:30:48.027 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:48.020+0000 7f509d81f700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f50980ce880 0x7f50980b0a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:48.027 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:48.020+0000 7f509d81f700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f509806bb90 0x7f509806c010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:48.027 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:48.020+0000 7f509d81f700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50980cfa00 con 0x7f509806b7c0 2023-10-06T03:30:48.028 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:48.020+0000 7f509d81f700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f50980cb1c0 con 0x7f509806bb90 2023-10-06T03:30:48.028 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:48.020+0000 7f509d81f700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50980cedf0 con 0x7f50980ce880 2023-10-06T03:30:48.028 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:48.020+0000 7f509d01e700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f509806bb90 0x7f509806c010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:48.028 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:48.020+0000 7f509c81d700 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f509806b7c0 0x7f50980ce170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:55830/0 (socket says 172.21.15.110:55830) 2023-10-06T03:30:48.028 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:48.020+0000 7f509d01e700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f509806bb90 0x7f509806c010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:47266/0 (socket says 172.21.15.110:47266) 2023-10-06T03:30:48.028 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:48.020+0000 7f509c81d700 1 -- 172.21.15.110:0/265332824 learned_addr learned my addr 172.21.15.110:0/265332824 (peer_addr_for_me v1:172.21.15.110:0/0) 2023-10-06T03:30:48.028 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:48.020+0000 7f5097fff700 1 --2- 172.21.15.110:0/265332824 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f50980ce880 0x7f50980b0a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:48.028 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:48.020+0000 7f509d01e700 1 -- 172.21.15.110:0/265332824 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f50980ce880 msgr2=0x7f50980b0a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:30:48.028 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:48.020+0000 7f509d01e700 1 --2- 172.21.15.110:0/265332824 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f50980ce880 0x7f50980b0a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:48.028 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:48.020+0000 7f509d01e700 1 -- 172.21.15.110:0/265332824 >> v1:172.21.15.110:6789/0 conn(0x7f509806b7c0 legacy=0x7f50980ce170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:48.028 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:48.020+0000 7f50977fe700 1 -- 172.21.15.110:0/265332824 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 55636446 0 0) 0x7f50980cfa00 con 0x7f509806b7c0 2023-10-06T03:30:48.029 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:48.020+0000 7f509d01e700 1 -- 172.21.15.110:0/265332824 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50980cf0d0 con 0x7f509806bb90 2023-10-06T03:30:48.029 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:48.020+0000 7f509d01e700 1 --2- 172.21.15.110:0/265332824 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f509806bb90 0x7f509806c010 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f5090009e60 tx=0x7f5090009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=324d7ba630ac66e1 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:30:48.029 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:48.024+0000 7f50977fe700 1 -- 172.21.15.110:0/265332824 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f509000e070 con 0x7f509806bb90 2023-10-06T03:30:48.029 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:48.024+0000 7f50977fe700 1 -- 172.21.15.110:0/265332824 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f509000bc30 con 0x7f509806bb90 2023-10-06T03:30:48.029 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:48.024+0000 7f50977fe700 1 -- 172.21.15.110:0/265332824 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5090012430 con 0x7f509806bb90 2023-10-06T03:30:48.030 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:48.024+0000 7f509d81f700 1 -- 172.21.15.110:0/265332824 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f509806bb90 msgr2=0x7f509806c010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:48.030 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:48.024+0000 7f509d81f700 1 --2- 172.21.15.110:0/265332824 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f509806bb90 0x7f509806c010 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f5090009e60 tx=0x7f5090009780 comp rx=0 tx=0).stop 2023-10-06T03:30:48.030 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:48.024+0000 7f509d81f700 1 -- 172.21.15.110:0/265332824 shutdown_connections 2023-10-06T03:30:48.030 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:48.024+0000 7f509d81f700 1 --2- 172.21.15.110:0/265332824 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f509806bb90 0x7f509806c010 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:48.030 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:48.024+0000 7f509d81f700 1 --2- 172.21.15.110:0/265332824 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f50980ce880 0x7f50980b0a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:48.030 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:48.024+0000 7f509d81f700 1 -- 172.21.15.110:0/265332824 >> 172.21.15.110:0/265332824 conn(0x7f50980cfea0 msgr2=0x7f50980d02a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:30:48.030 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:48.024+0000 7f509d81f700 1 -- 172.21.15.110:0/265332824 shutdown_connections 2023-10-06T03:30:48.030 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:48.024+0000 7f509d81f700 1 -- 172.21.15.110:0/265332824 wait complete. 2023-10-06T03:30:48.031 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:48.024+0000 7f509d81f700 1 Processor -- start 2023-10-06T03:30:48.031 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:48.024+0000 7f509d81f700 1 -- start start 2023-10-06T03:30:48.033 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:48.024+0000 7f509d81f700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f509806b7c0 0x7f5098147ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:48.033 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:48.024+0000 7f509d81f700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f509806bb90 0x7f50980c90a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:48.033 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:48.024+0000 7f509d81f700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f50980ce880 0x7f50980c95e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:48.033 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:48.028+0000 7f509d81f700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50980b4560 con 0x7f50980ce880 2023-10-06T03:30:48.033 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:48.028+0000 7f509d81f700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f509815c180 con 0x7f509806b7c0 2023-10-06T03:30:48.033 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:48.028+0000 7f509d81f700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50981697d0 con 0x7f509806bb90 2023-10-06T03:30:48.033 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:48.028+0000 7f509d01e700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f50980ce880 0x7f50980c95e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:48.033 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:48.028+0000 7f509d01e700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f50980ce880 0x7f50980c95e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:47282/0 (socket says 172.21.15.110:47282) 2023-10-06T03:30:48.033 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:48.028+0000 7f509d01e700 1 -- 172.21.15.110:0/454907318 learned_addr learned my addr 172.21.15.110:0/454907318 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:30:48.033 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:48.028+0000 7f509c81d700 1 --2- 172.21.15.110:0/454907318 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f509806b7c0 0x7f5098147ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:48.034 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:48.028+0000 7f5097fff700 1 --2- 172.21.15.110:0/454907318 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f509806bb90 0x7f50980c90a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:48.034 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:48.028+0000 7f509d01e700 1 -- 172.21.15.110:0/454907318 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f509806bb90 msgr2=0x7f50980c90a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:48.034 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:48.028+0000 7f509d01e700 1 --2- 172.21.15.110:0/454907318 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f509806bb90 0x7f50980c90a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:48.034 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:48.028+0000 7f509d01e700 1 -- 172.21.15.110:0/454907318 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f509806b7c0 msgr2=0x7f5098147ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:48.034 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:48.028+0000 7f509d01e700 1 --2- 172.21.15.110:0/454907318 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f509806b7c0 0x7f5098147ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:48.034 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:48.028+0000 7f509d01e700 1 -- 172.21.15.110:0/454907318 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5090009a20 con 0x7f50980ce880 2023-10-06T03:30:48.034 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:48.028+0000 7f509d01e700 1 --2- 172.21.15.110:0/454907318 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f50980ce880 0x7f50980c95e0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f5090013000 tx=0x7f5090019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:30:48.034 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:48.028+0000 7f5094ff9700 1 -- 172.21.15.110:0/454907318 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5090027470 con 0x7f50980ce880 2023-10-06T03:30:48.034 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:48.028+0000 7f5094ff9700 1 -- 172.21.15.110:0/454907318 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50900275f0 con 0x7f50980ce880 2023-10-06T03:30:48.034 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:48.028+0000 7f5094ff9700 1 -- 172.21.15.110:0/454907318 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50900279b0 con 0x7f50980ce880 2023-10-06T03:30:48.034 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:48.028+0000 7f509d81f700 1 -- 172.21.15.110:0/454907318 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5098169ab0 con 0x7f50980ce880 2023-10-06T03:30:48.034 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:48.028+0000 7f509d81f700 1 -- 172.21.15.110:0/454907318 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50980b41f0 con 0x7f50980ce880 2023-10-06T03:30:48.035 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:48.028+0000 7f509d81f700 1 -- 172.21.15.110:0/454907318 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f50980b4710 con 0x7f50980ce880 2023-10-06T03:30:48.035 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:48.032+0000 7f5094ff9700 1 -- 172.21.15.110:0/454907318 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 8) v1 ==== 79814+0+0 (secure 0 0 0) 0x7f5090024070 con 0x7f50980ce880 2023-10-06T03:30:48.036 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:48.032+0000 7f5094ff9700 1 --2- 172.21.15.110:0/454907318 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f50840603f0 0x7f50840628a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:48.036 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:48.032+0000 7f5094ff9700 1 -- 172.21.15.110:0/454907318 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 6636+0+0 (secure 0 0 0) 0x7f509007f2b0 con 0x7f50980ce880 2023-10-06T03:30:48.036 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:48.032+0000 7f509c81d700 1 --2- 172.21.15.110:0/454907318 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f50840603f0 0x7f50840628a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:48.037 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:48.032+0000 7f509c81d700 1 --2- 172.21.15.110:0/454907318 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f50840603f0 0x7f50840628a0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f509815b8d0 tx=0x7f508c007d30 comp rx=0 tx=0).ready entity=mgr.5329 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:30:48.043 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:48.036+0000 7f5094ff9700 1 -- 172.21.15.110:0/454907318 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f50980b4710 con 0x7f50980ce880 2023-10-06T03:30:48.225 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:48.220+0000 7f509d81f700 1 -- 172.21.15.110:0/454907318 --> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] -- mgr_command(tid 0: {"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "snap_test_dir1", "retention_spec_or_period": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7f5098070ff0 con 0x7f50840603f0 2023-10-06T03:30:48.336 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:48.332+0000 7f5094ff9700 1 -- 172.21.15.110:0/454907318 <== mgr.5329 v2:172.21.15.110:6832/33268 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+39 (secure 0 0 0) 0x7f5098070ff0 con 0x7f50840603f0 2023-10-06T03:30:48.336 INFO:teuthology.orchestra.run.smithi110.stdout:Retention added to path snap_test_dir1 2023-10-06T03:30:48.341 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:48.336+0000 7f509d81f700 1 -- 172.21.15.110:0/454907318 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f50840603f0 msgr2=0x7f50840628a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:48.341 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:48.336+0000 7f509d81f700 1 --2- 172.21.15.110:0/454907318 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f50840603f0 0x7f50840628a0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f509815b8d0 tx=0x7f508c007d30 comp rx=0 tx=0).stop 2023-10-06T03:30:48.341 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:48.336+0000 7f509d81f700 1 -- 172.21.15.110:0/454907318 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f50980ce880 msgr2=0x7f50980c95e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:48.341 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:48.336+0000 7f509d81f700 1 --2- 172.21.15.110:0/454907318 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f50980ce880 0x7f50980c95e0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f5090013000 tx=0x7f5090019000 comp rx=0 tx=0).stop 2023-10-06T03:30:48.341 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:48.336+0000 7f509d81f700 1 -- 172.21.15.110:0/454907318 shutdown_connections 2023-10-06T03:30:48.341 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:48.336+0000 7f509d81f700 1 --2- 172.21.15.110:0/454907318 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f50840603f0 0x7f50840628a0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:48.341 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:48.336+0000 7f509d81f700 1 --2- 172.21.15.110:0/454907318 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f50980ce880 0x7f50980c95e0 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:48.341 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:48.336+0000 7f509d81f700 1 --2- 172.21.15.110:0/454907318 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f509806bb90 0x7f50980c90a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:48.341 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:48.336+0000 7f509d81f700 1 --2- 172.21.15.110:0/454907318 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f509806b7c0 0x7f5098147ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:48.341 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:48.336+0000 7f509d81f700 1 -- 172.21.15.110:0/454907318 >> 172.21.15.110:0/454907318 conn(0x7f50980cfea0 msgr2=0x7f5098152a50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:30:48.342 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:48.336+0000 7f509d81f700 1 -- 172.21.15.110:0/454907318 shutdown_connections 2023-10-06T03:30:48.342 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:48.336+0000 7f509d81f700 1 -- 172.21.15.110:0/454907318 wait complete. 2023-10-06T03:30:48.357 DEBUG:tasks.cephfs.test_snap_schedules:res=Retention added to path snap_test_dir1 2023-10-06T03:30:48.361 DEBUG:teuthology.orchestra.run.smithi110:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2023-10-06T03:30:48.363 DEBUG:teuthology.orchestra.run.smithi192:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2023-10-06T03:30:48.392 INFO:teuthology.orchestra.run.smithi110.stdout:[global] 2023-10-06T03:30:48.393 INFO:teuthology.orchestra.run.smithi110.stdout: chdir = "" 2023-10-06T03:30:48.393 INFO:teuthology.orchestra.run.smithi110.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-10-06T03:30:48.393 INFO:teuthology.orchestra.run.smithi110.stdout: auth supported = cephx 2023-10-06T03:30:48.393 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-10-06T03:30:48.393 INFO:teuthology.orchestra.run.smithi110.stdout: filestore xattr use omap = true 2023-10-06T03:30:48.393 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-10-06T03:30:48.393 INFO:teuthology.orchestra.run.smithi110.stdout: mon clock drift allowed = 1.000 2023-10-06T03:30:48.393 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-10-06T03:30:48.393 INFO:teuthology.orchestra.run.smithi110.stdout: osd crush chooseleaf type = 0 2023-10-06T03:30:48.393 INFO:teuthology.orchestra.run.smithi110.stdout: auth debug = true 2023-10-06T03:30:48.394 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-10-06T03:30:48.394 INFO:teuthology.orchestra.run.smithi110.stdout: ms die on old message = true 2023-10-06T03:30:48.394 INFO:teuthology.orchestra.run.smithi110.stdout: ms die on bug = true 2023-10-06T03:30:48.394 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-10-06T03:30:48.394 INFO:teuthology.orchestra.run.smithi110.stdout: mon max pg per osd = 10000 # >= luminous 2023-10-06T03:30:48.394 INFO:teuthology.orchestra.run.smithi110.stdout: mon pg warn max object skew = 0 2023-10-06T03:30:48.394 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-10-06T03:30:48.394 INFO:teuthology.orchestra.run.smithi110.stdout: # disable pg_autoscaler by default for new pools 2023-10-06T03:30:48.394 INFO:teuthology.orchestra.run.smithi110.stdout: osd_pool_default_pg_autoscale_mode = off 2023-10-06T03:30:48.394 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-10-06T03:30:48.394 INFO:teuthology.orchestra.run.smithi110.stdout: osd pool default size = 2 2023-10-06T03:30:48.395 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-10-06T03:30:48.395 INFO:teuthology.orchestra.run.smithi110.stdout: mon osd allow primary affinity = true 2023-10-06T03:30:48.395 INFO:teuthology.orchestra.run.smithi110.stdout: mon osd allow pg remap = true 2023-10-06T03:30:48.395 INFO:teuthology.orchestra.run.smithi110.stdout: mon warn on legacy crush tunables = false 2023-10-06T03:30:48.395 INFO:teuthology.orchestra.run.smithi110.stdout: mon warn on crush straw calc version zero = false 2023-10-06T03:30:48.395 INFO:teuthology.orchestra.run.smithi110.stdout: mon warn on no sortbitwise = false 2023-10-06T03:30:48.395 INFO:teuthology.orchestra.run.smithi110.stdout: mon warn on osd down out interval zero = false 2023-10-06T03:30:48.395 INFO:teuthology.orchestra.run.smithi110.stdout: mon warn on too few osds = false 2023-10-06T03:30:48.395 INFO:teuthology.orchestra.run.smithi110.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-10-06T03:30:48.395 INFO:teuthology.orchestra.run.smithi110.stdout: mon_warn_on_pool_no_redundancy = false 2023-10-06T03:30:48.395 INFO:teuthology.orchestra.run.smithi110.stdout: mon_allow_pool_size_one = true 2023-10-06T03:30:48.395 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-10-06T03:30:48.396 INFO:teuthology.orchestra.run.smithi110.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-10-06T03:30:48.396 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-10-06T03:30:48.396 INFO:teuthology.orchestra.run.smithi110.stdout: osd default data pool replay window = 5 2023-10-06T03:30:48.396 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-10-06T03:30:48.396 INFO:teuthology.orchestra.run.smithi110.stdout: mon allow pool delete = true 2023-10-06T03:30:48.396 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-10-06T03:30:48.396 INFO:teuthology.orchestra.run.smithi110.stdout: mon cluster log file level = debug 2023-10-06T03:30:48.396 INFO:teuthology.orchestra.run.smithi110.stdout: debug asserts on shutdown = true 2023-10-06T03:30:48.396 INFO:teuthology.orchestra.run.smithi110.stdout: mon health detail to clog = false 2023-10-06T03:30:48.396 INFO:teuthology.orchestra.run.smithi110.stdout: mon host = "172.21.15.110,172.21.15.192,[v2:172.21.15.192:3301,v1:172.21.15.192:6790]" 2023-10-06T03:30:48.396 INFO:teuthology.orchestra.run.smithi110.stdout: fsid = 93b8f6ac-4c92-4b45-b8e4-b59617024aa2 2023-10-06T03:30:48.396 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-10-06T03:30:48.397 INFO:teuthology.orchestra.run.smithi110.stdout:[osd] 2023-10-06T03:30:48.397 INFO:teuthology.orchestra.run.smithi110.stdout: osd journal size = 100 2023-10-06T03:30:48.397 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-10-06T03:30:48.397 INFO:teuthology.orchestra.run.smithi110.stdout: osd scrub load threshold = 5.0 2023-10-06T03:30:48.397 INFO:teuthology.orchestra.run.smithi110.stdout: osd scrub max interval = 600 2023-10-06T03:30:48.397 INFO:teuthology.orchestra.run.smithi110.stdout: osd mclock profile = high_recovery_ops 2023-10-06T03:30:48.397 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-10-06T03:30:48.397 INFO:teuthology.orchestra.run.smithi110.stdout: osd recover clone overlap = true 2023-10-06T03:30:48.397 INFO:teuthology.orchestra.run.smithi110.stdout: osd recovery max chunk = 1048576 2023-10-06T03:30:48.397 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-10-06T03:30:48.397 INFO:teuthology.orchestra.run.smithi110.stdout: osd debug shutdown = true 2023-10-06T03:30:48.397 INFO:teuthology.orchestra.run.smithi110.stdout: osd debug op order = true 2023-10-06T03:30:48.398 INFO:teuthology.orchestra.run.smithi110.stdout: osd debug verify stray on activate = true 2023-10-06T03:30:48.398 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-10-06T03:30:48.398 INFO:teuthology.orchestra.run.smithi110.stdout: osd open classes on start = true 2023-10-06T03:30:48.398 INFO:teuthology.orchestra.run.smithi110.stdout: osd debug pg log writeout = true 2023-10-06T03:30:48.398 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-10-06T03:30:48.398 INFO:teuthology.orchestra.run.smithi110.stdout: osd deep scrub update digest min age = 30 2023-10-06T03:30:48.398 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-10-06T03:30:48.398 INFO:teuthology.orchestra.run.smithi110.stdout: osd map max advance = 10 2023-10-06T03:30:48.398 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-10-06T03:30:48.398 INFO:teuthology.orchestra.run.smithi110.stdout: journal zero on create = true 2023-10-06T03:30:48.398 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-10-06T03:30:48.399 INFO:teuthology.orchestra.run.smithi110.stdout: filestore ondisk finisher threads = 3 2023-10-06T03:30:48.399 INFO:teuthology.orchestra.run.smithi110.stdout: filestore apply finisher threads = 3 2023-10-06T03:30:48.399 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-10-06T03:30:48.399 INFO:teuthology.orchestra.run.smithi110.stdout: bdev debug aio = true 2023-10-06T03:30:48.399 INFO:teuthology.orchestra.run.smithi110.stdout: osd debug misdirected ops = true 2023-10-06T03:30:48.399 INFO:teuthology.orchestra.run.smithi110.stdout: bluestore block size = 96636764160 2023-10-06T03:30:48.399 INFO:teuthology.orchestra.run.smithi110.stdout: bluestore fsck on mount = True 2023-10-06T03:30:48.399 INFO:teuthology.orchestra.run.smithi110.stdout: debug bluefs = 20 2023-10-06T03:30:48.399 INFO:teuthology.orchestra.run.smithi110.stdout: debug bluestore = 20 2023-10-06T03:30:48.399 INFO:teuthology.orchestra.run.smithi110.stdout: debug ms = 1 2023-10-06T03:30:48.399 INFO:teuthology.orchestra.run.smithi110.stdout: debug osd = 20 2023-10-06T03:30:48.399 INFO:teuthology.orchestra.run.smithi110.stdout: debug rocksdb = 10 2023-10-06T03:30:48.400 INFO:teuthology.orchestra.run.smithi110.stdout: mon osd backfillfull_ratio = 0.85 2023-10-06T03:30:48.400 INFO:teuthology.orchestra.run.smithi110.stdout: mon osd full ratio = 0.9 2023-10-06T03:30:48.400 INFO:teuthology.orchestra.run.smithi110.stdout: mon osd nearfull ratio = 0.8 2023-10-06T03:30:48.400 INFO:teuthology.orchestra.run.smithi110.stdout: osd failsafe full ratio = 0.95 2023-10-06T03:30:48.400 INFO:teuthology.orchestra.run.smithi110.stdout: osd objectstore = bluestore 2023-10-06T03:30:48.400 INFO:teuthology.orchestra.run.smithi110.stdout: osd op complaint time = 180 2023-10-06T03:30:48.400 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-10-06T03:30:48.400 INFO:teuthology.orchestra.run.smithi110.stdout:[mgr] 2023-10-06T03:30:48.400 INFO:teuthology.orchestra.run.smithi110.stdout: debug ms = 1 2023-10-06T03:30:48.400 INFO:teuthology.orchestra.run.smithi110.stdout: debug mgr = 20 2023-10-06T03:30:48.400 INFO:teuthology.orchestra.run.smithi110.stdout: debug mon = 20 2023-10-06T03:30:48.401 INFO:teuthology.orchestra.run.smithi110.stdout: debug auth = 20 2023-10-06T03:30:48.401 INFO:teuthology.orchestra.run.smithi110.stdout: mon reweight min pgs per osd = 4 2023-10-06T03:30:48.401 INFO:teuthology.orchestra.run.smithi110.stdout: mon reweight min bytes per osd = 10 2023-10-06T03:30:48.401 INFO:teuthology.orchestra.run.smithi110.stdout: mgr/telemetry/nag = false 2023-10-06T03:30:48.401 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-10-06T03:30:48.401 INFO:teuthology.orchestra.run.smithi110.stdout:[mon] 2023-10-06T03:30:48.401 INFO:teuthology.orchestra.run.smithi110.stdout: debug ms = 1 2023-10-06T03:30:48.401 INFO:teuthology.orchestra.run.smithi110.stdout: debug mon = 20 2023-10-06T03:30:48.401 INFO:teuthology.orchestra.run.smithi110.stdout: debug paxos = 20 2023-10-06T03:30:48.401 INFO:teuthology.orchestra.run.smithi110.stdout: debug auth = 20 2023-10-06T03:30:48.401 INFO:teuthology.orchestra.run.smithi110.stdout: mon data avail warn = 5 2023-10-06T03:30:48.401 INFO:teuthology.orchestra.run.smithi110.stdout: mon mgr mkfs grace = 240 2023-10-06T03:30:48.402 INFO:teuthology.orchestra.run.smithi110.stdout: mon reweight min pgs per osd = 4 2023-10-06T03:30:48.402 INFO:teuthology.orchestra.run.smithi110.stdout: mon osd reporter subtree level = osd 2023-10-06T03:30:48.403 INFO:teuthology.orchestra.run.smithi110.stdout: mon osd prime pg temp = true 2023-10-06T03:30:48.403 INFO:teuthology.orchestra.run.smithi110.stdout: mon reweight min bytes per osd = 10 2023-10-06T03:30:48.403 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-10-06T03:30:48.403 INFO:teuthology.orchestra.run.smithi110.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-10-06T03:30:48.403 INFO:teuthology.orchestra.run.smithi110.stdout: auth mon ticket ttl = 660 # 11m 2023-10-06T03:30:48.403 INFO:teuthology.orchestra.run.smithi110.stdout: auth service ticket ttl = 240 # 4m 2023-10-06T03:30:48.403 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-10-06T03:30:48.403 INFO:teuthology.orchestra.run.smithi110.stdout: # don't complain about insecure global_id in the test suite 2023-10-06T03:30:48.403 INFO:teuthology.orchestra.run.smithi110.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-10-06T03:30:48.403 INFO:teuthology.orchestra.run.smithi110.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-10-06T03:30:48.403 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-10-06T03:30:48.404 INFO:teuthology.orchestra.run.smithi110.stdout: # 1m isn't quite enough 2023-10-06T03:30:48.404 INFO:teuthology.orchestra.run.smithi110.stdout: mon_down_mkfs_grace = 2m 2023-10-06T03:30:48.404 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-10-06T03:30:48.404 INFO:teuthology.orchestra.run.smithi110.stdout: mon_warn_on_filestore_osds = false 2023-10-06T03:30:48.404 INFO:teuthology.orchestra.run.smithi110.stdout: mon op complaint time = 120 2023-10-06T03:30:48.404 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-10-06T03:30:48.404 INFO:teuthology.orchestra.run.smithi110.stdout:[client] 2023-10-06T03:30:48.404 INFO:teuthology.orchestra.run.smithi110.stdout: rgw cache enabled = true 2023-10-06T03:30:48.404 INFO:teuthology.orchestra.run.smithi110.stdout: rgw enable ops log = true 2023-10-06T03:30:48.404 INFO:teuthology.orchestra.run.smithi110.stdout: rgw enable usage log = true 2023-10-06T03:30:48.404 INFO:teuthology.orchestra.run.smithi110.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-10-06T03:30:48.404 INFO:teuthology.orchestra.run.smithi110.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-10-06T03:30:48.404 INFO:teuthology.orchestra.run.smithi110.stdout: client mount timeout = 600 2023-10-06T03:30:48.405 INFO:teuthology.orchestra.run.smithi110.stdout: client snapdir = .snap 2023-10-06T03:30:48.405 INFO:teuthology.orchestra.run.smithi110.stdout: debug client = 20 2023-10-06T03:30:48.405 INFO:teuthology.orchestra.run.smithi110.stdout: debug ms = 1 2023-10-06T03:30:48.405 INFO:teuthology.orchestra.run.smithi110.stdout: pid file = "" 2023-10-06T03:30:48.405 INFO:teuthology.orchestra.run.smithi110.stdout: rados mon op timeout = 900 2023-10-06T03:30:48.405 INFO:teuthology.orchestra.run.smithi110.stdout: rados osd op timeout = 900 2023-10-06T03:30:48.405 INFO:teuthology.orchestra.run.smithi110.stdout:[mon.a] 2023-10-06T03:30:48.405 INFO:teuthology.orchestra.run.smithi110.stdout:[mon.b] 2023-10-06T03:30:48.405 INFO:teuthology.orchestra.run.smithi110.stdout:[mon.c] 2023-10-06T03:30:48.405 INFO:teuthology.orchestra.run.smithi110.stdout:[mds.a] 2023-10-06T03:30:48.405 INFO:teuthology.orchestra.run.smithi110.stdout:[mds.c] 2023-10-06T03:30:48.405 INFO:teuthology.orchestra.run.smithi110.stdout:[mds.b] 2023-10-06T03:30:48.406 INFO:teuthology.orchestra.run.smithi110.stdout:[mds.d] 2023-10-06T03:30:48.406 INFO:teuthology.orchestra.run.smithi110.stdout:[mds] 2023-10-06T03:30:48.406 INFO:teuthology.orchestra.run.smithi110.stdout: debug mds = 20 2023-10-06T03:30:48.406 INFO:teuthology.orchestra.run.smithi110.stdout: debug ms = 1 2023-10-06T03:30:48.406 INFO:teuthology.orchestra.run.smithi110.stdout: mds debug frag = True 2023-10-06T03:30:48.406 INFO:teuthology.orchestra.run.smithi110.stdout: mds debug scatterstat = True 2023-10-06T03:30:48.406 INFO:teuthology.orchestra.run.smithi110.stdout: mds op complaint time = 180 2023-10-06T03:30:48.406 INFO:teuthology.orchestra.run.smithi110.stdout: mds verify scatter = True 2023-10-06T03:30:48.406 INFO:teuthology.orchestra.run.smithi110.stdout: osd op complaint time = 180 2023-10-06T03:30:48.406 INFO:teuthology.orchestra.run.smithi110.stdout: rados mon op timeout = 900 2023-10-06T03:30:48.406 INFO:teuthology.orchestra.run.smithi110.stdout: rados osd op timeout = 900 2023-10-06T03:30:48.407 INFO:teuthology.orchestra.run.smithi192.stdout:[global] 2023-10-06T03:30:48.407 INFO:teuthology.orchestra.run.smithi192.stdout: chdir = "" 2023-10-06T03:30:48.407 INFO:teuthology.orchestra.run.smithi192.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-10-06T03:30:48.407 INFO:teuthology.orchestra.run.smithi192.stdout: auth supported = cephx 2023-10-06T03:30:48.408 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-10-06T03:30:48.408 INFO:teuthology.orchestra.run.smithi192.stdout: filestore xattr use omap = true 2023-10-06T03:30:48.408 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-10-06T03:30:48.408 INFO:teuthology.orchestra.run.smithi192.stdout: mon clock drift allowed = 1.000 2023-10-06T03:30:48.408 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-10-06T03:30:48.408 INFO:teuthology.orchestra.run.smithi192.stdout: osd crush chooseleaf type = 0 2023-10-06T03:30:48.408 INFO:teuthology.orchestra.run.smithi192.stdout: auth debug = true 2023-10-06T03:30:48.408 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-10-06T03:30:48.408 INFO:teuthology.orchestra.run.smithi192.stdout: ms die on old message = true 2023-10-06T03:30:48.408 INFO:teuthology.orchestra.run.smithi192.stdout: ms die on bug = true 2023-10-06T03:30:48.408 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-10-06T03:30:48.408 INFO:teuthology.orchestra.run.smithi192.stdout: mon max pg per osd = 10000 # >= luminous 2023-10-06T03:30:48.409 INFO:teuthology.orchestra.run.smithi192.stdout: mon pg warn max object skew = 0 2023-10-06T03:30:48.409 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-10-06T03:30:48.409 INFO:teuthology.orchestra.run.smithi192.stdout: # disable pg_autoscaler by default for new pools 2023-10-06T03:30:48.409 INFO:teuthology.orchestra.run.smithi192.stdout: osd_pool_default_pg_autoscale_mode = off 2023-10-06T03:30:48.409 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-10-06T03:30:48.409 INFO:teuthology.orchestra.run.smithi192.stdout: osd pool default size = 2 2023-10-06T03:30:48.409 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-10-06T03:30:48.409 INFO:teuthology.orchestra.run.smithi192.stdout: mon osd allow primary affinity = true 2023-10-06T03:30:48.409 INFO:teuthology.orchestra.run.smithi192.stdout: mon osd allow pg remap = true 2023-10-06T03:30:48.409 INFO:teuthology.orchestra.run.smithi192.stdout: mon warn on legacy crush tunables = false 2023-10-06T03:30:48.409 INFO:teuthology.orchestra.run.smithi192.stdout: mon warn on crush straw calc version zero = false 2023-10-06T03:30:48.409 INFO:teuthology.orchestra.run.smithi192.stdout: mon warn on no sortbitwise = false 2023-10-06T03:30:48.410 INFO:teuthology.orchestra.run.smithi192.stdout: mon warn on osd down out interval zero = false 2023-10-06T03:30:48.410 INFO:teuthology.orchestra.run.smithi192.stdout: mon warn on too few osds = false 2023-10-06T03:30:48.410 INFO:teuthology.orchestra.run.smithi192.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-10-06T03:30:48.410 INFO:teuthology.orchestra.run.smithi192.stdout: mon_warn_on_pool_no_redundancy = false 2023-10-06T03:30:48.410 INFO:teuthology.orchestra.run.smithi192.stdout: mon_allow_pool_size_one = true 2023-10-06T03:30:48.410 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-10-06T03:30:48.410 INFO:teuthology.orchestra.run.smithi192.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-10-06T03:30:48.410 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-10-06T03:30:48.410 INFO:teuthology.orchestra.run.smithi192.stdout: osd default data pool replay window = 5 2023-10-06T03:30:48.410 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-10-06T03:30:48.410 INFO:teuthology.orchestra.run.smithi192.stdout: mon allow pool delete = true 2023-10-06T03:30:48.410 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-10-06T03:30:48.411 INFO:teuthology.orchestra.run.smithi192.stdout: mon cluster log file level = debug 2023-10-06T03:30:48.411 INFO:teuthology.orchestra.run.smithi192.stdout: debug asserts on shutdown = true 2023-10-06T03:30:48.411 INFO:teuthology.orchestra.run.smithi192.stdout: mon health detail to clog = false 2023-10-06T03:30:48.411 INFO:teuthology.orchestra.run.smithi192.stdout: mon host = "172.21.15.110,172.21.15.192,[v2:172.21.15.192:3301,v1:172.21.15.192:6790]" 2023-10-06T03:30:48.411 INFO:teuthology.orchestra.run.smithi192.stdout: fsid = 93b8f6ac-4c92-4b45-b8e4-b59617024aa2 2023-10-06T03:30:48.411 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-10-06T03:30:48.411 INFO:teuthology.orchestra.run.smithi192.stdout:[osd] 2023-10-06T03:30:48.411 INFO:teuthology.orchestra.run.smithi192.stdout: osd journal size = 100 2023-10-06T03:30:48.411 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-10-06T03:30:48.411 INFO:teuthology.orchestra.run.smithi192.stdout: osd scrub load threshold = 5.0 2023-10-06T03:30:48.411 INFO:teuthology.orchestra.run.smithi192.stdout: osd scrub max interval = 600 2023-10-06T03:30:48.411 INFO:teuthology.orchestra.run.smithi192.stdout: osd mclock profile = high_recovery_ops 2023-10-06T03:30:48.412 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-10-06T03:30:48.412 INFO:teuthology.orchestra.run.smithi192.stdout: osd recover clone overlap = true 2023-10-06T03:30:48.412 INFO:teuthology.orchestra.run.smithi192.stdout: osd recovery max chunk = 1048576 2023-10-06T03:30:48.412 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-10-06T03:30:48.412 INFO:teuthology.orchestra.run.smithi192.stdout: osd debug shutdown = true 2023-10-06T03:30:48.412 INFO:teuthology.orchestra.run.smithi192.stdout: osd debug op order = true 2023-10-06T03:30:48.412 INFO:teuthology.orchestra.run.smithi192.stdout: osd debug verify stray on activate = true 2023-10-06T03:30:48.412 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-10-06T03:30:48.412 INFO:teuthology.orchestra.run.smithi192.stdout: osd open classes on start = true 2023-10-06T03:30:48.412 INFO:teuthology.orchestra.run.smithi192.stdout: osd debug pg log writeout = true 2023-10-06T03:30:48.412 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-10-06T03:30:48.412 INFO:teuthology.orchestra.run.smithi192.stdout: osd deep scrub update digest min age = 30 2023-10-06T03:30:48.413 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-10-06T03:30:48.413 INFO:teuthology.orchestra.run.smithi192.stdout: osd map max advance = 10 2023-10-06T03:30:48.413 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-10-06T03:30:48.413 INFO:teuthology.orchestra.run.smithi192.stdout: journal zero on create = true 2023-10-06T03:30:48.413 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-10-06T03:30:48.413 INFO:teuthology.orchestra.run.smithi192.stdout: filestore ondisk finisher threads = 3 2023-10-06T03:30:48.413 INFO:teuthology.orchestra.run.smithi192.stdout: filestore apply finisher threads = 3 2023-10-06T03:30:48.413 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-10-06T03:30:48.413 INFO:teuthology.orchestra.run.smithi192.stdout: bdev debug aio = true 2023-10-06T03:30:48.413 INFO:teuthology.orchestra.run.smithi192.stdout: osd debug misdirected ops = true 2023-10-06T03:30:48.413 INFO:teuthology.orchestra.run.smithi192.stdout: bluestore block size = 96636764160 2023-10-06T03:30:48.413 INFO:teuthology.orchestra.run.smithi192.stdout: bluestore fsck on mount = True 2023-10-06T03:30:48.413 INFO:teuthology.orchestra.run.smithi192.stdout: debug bluefs = 20 2023-10-06T03:30:48.414 INFO:teuthology.orchestra.run.smithi192.stdout: debug bluestore = 20 2023-10-06T03:30:48.414 INFO:teuthology.orchestra.run.smithi192.stdout: debug ms = 1 2023-10-06T03:30:48.414 INFO:teuthology.orchestra.run.smithi192.stdout: debug osd = 20 2023-10-06T03:30:48.414 INFO:teuthology.orchestra.run.smithi192.stdout: debug rocksdb = 10 2023-10-06T03:30:48.414 INFO:teuthology.orchestra.run.smithi192.stdout: mon osd backfillfull_ratio = 0.85 2023-10-06T03:30:48.414 INFO:teuthology.orchestra.run.smithi192.stdout: mon osd full ratio = 0.9 2023-10-06T03:30:48.414 INFO:teuthology.orchestra.run.smithi192.stdout: mon osd nearfull ratio = 0.8 2023-10-06T03:30:48.414 INFO:teuthology.orchestra.run.smithi192.stdout: osd failsafe full ratio = 0.95 2023-10-06T03:30:48.414 INFO:teuthology.orchestra.run.smithi192.stdout: osd objectstore = bluestore 2023-10-06T03:30:48.414 INFO:teuthology.orchestra.run.smithi192.stdout: osd op complaint time = 180 2023-10-06T03:30:48.414 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-10-06T03:30:48.415 INFO:teuthology.orchestra.run.smithi192.stdout:[mgr] 2023-10-06T03:30:48.415 INFO:teuthology.orchestra.run.smithi192.stdout: debug ms = 1 2023-10-06T03:30:48.415 INFO:teuthology.orchestra.run.smithi192.stdout: debug mgr = 20 2023-10-06T03:30:48.415 INFO:teuthology.orchestra.run.smithi192.stdout: debug mon = 20 2023-10-06T03:30:48.415 INFO:teuthology.orchestra.run.smithi192.stdout: debug auth = 20 2023-10-06T03:30:48.415 INFO:teuthology.orchestra.run.smithi192.stdout: mon reweight min pgs per osd = 4 2023-10-06T03:30:48.415 INFO:teuthology.orchestra.run.smithi192.stdout: mon reweight min bytes per osd = 10 2023-10-06T03:30:48.415 INFO:teuthology.orchestra.run.smithi192.stdout: mgr/telemetry/nag = false 2023-10-06T03:30:48.415 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-10-06T03:30:48.415 INFO:teuthology.orchestra.run.smithi192.stdout:[mon] 2023-10-06T03:30:48.415 INFO:teuthology.orchestra.run.smithi192.stdout: debug ms = 1 2023-10-06T03:30:48.415 INFO:teuthology.orchestra.run.smithi192.stdout: debug mon = 20 2023-10-06T03:30:48.416 INFO:teuthology.orchestra.run.smithi192.stdout: debug paxos = 20 2023-10-06T03:30:48.416 INFO:teuthology.orchestra.run.smithi192.stdout: debug auth = 20 2023-10-06T03:30:48.416 INFO:teuthology.orchestra.run.smithi192.stdout: mon data avail warn = 5 2023-10-06T03:30:48.416 INFO:teuthology.orchestra.run.smithi192.stdout: mon mgr mkfs grace = 240 2023-10-06T03:30:48.416 INFO:teuthology.orchestra.run.smithi192.stdout: mon reweight min pgs per osd = 4 2023-10-06T03:30:48.416 INFO:teuthology.orchestra.run.smithi192.stdout: mon osd reporter subtree level = osd 2023-10-06T03:30:48.416 INFO:teuthology.orchestra.run.smithi192.stdout: mon osd prime pg temp = true 2023-10-06T03:30:48.416 INFO:teuthology.orchestra.run.smithi192.stdout: mon reweight min bytes per osd = 10 2023-10-06T03:30:48.416 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-10-06T03:30:48.416 INFO:teuthology.orchestra.run.smithi192.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-10-06T03:30:48.416 INFO:teuthology.orchestra.run.smithi192.stdout: auth mon ticket ttl = 660 # 11m 2023-10-06T03:30:48.416 INFO:teuthology.orchestra.run.smithi192.stdout: auth service ticket ttl = 240 # 4m 2023-10-06T03:30:48.416 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-10-06T03:30:48.417 INFO:teuthology.orchestra.run.smithi192.stdout: # don't complain about insecure global_id in the test suite 2023-10-06T03:30:48.417 INFO:teuthology.orchestra.run.smithi192.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-10-06T03:30:48.417 INFO:teuthology.orchestra.run.smithi192.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-10-06T03:30:48.417 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-10-06T03:30:48.417 INFO:teuthology.orchestra.run.smithi192.stdout: # 1m isn't quite enough 2023-10-06T03:30:48.417 INFO:teuthology.orchestra.run.smithi192.stdout: mon_down_mkfs_grace = 2m 2023-10-06T03:30:48.417 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-10-06T03:30:48.417 INFO:teuthology.orchestra.run.smithi192.stdout: mon_warn_on_filestore_osds = false 2023-10-06T03:30:48.417 INFO:teuthology.orchestra.run.smithi192.stdout: mon op complaint time = 120 2023-10-06T03:30:48.417 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-10-06T03:30:48.417 INFO:teuthology.orchestra.run.smithi192.stdout:[client] 2023-10-06T03:30:48.417 INFO:teuthology.orchestra.run.smithi192.stdout: rgw cache enabled = true 2023-10-06T03:30:48.418 INFO:teuthology.orchestra.run.smithi192.stdout: rgw enable ops log = true 2023-10-06T03:30:48.418 INFO:teuthology.orchestra.run.smithi192.stdout: rgw enable usage log = true 2023-10-06T03:30:48.418 INFO:teuthology.orchestra.run.smithi192.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-10-06T03:30:48.418 INFO:teuthology.orchestra.run.smithi192.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-10-06T03:30:48.418 INFO:teuthology.orchestra.run.smithi192.stdout: client mount timeout = 600 2023-10-06T03:30:48.418 INFO:teuthology.orchestra.run.smithi192.stdout: client snapdir = .snap 2023-10-06T03:30:48.418 INFO:teuthology.orchestra.run.smithi192.stdout: debug client = 20 2023-10-06T03:30:48.418 INFO:teuthology.orchestra.run.smithi192.stdout: debug ms = 1 2023-10-06T03:30:48.418 INFO:teuthology.orchestra.run.smithi192.stdout: pid file = "" 2023-10-06T03:30:48.418 INFO:teuthology.orchestra.run.smithi192.stdout: rados mon op timeout = 900 2023-10-06T03:30:48.418 INFO:teuthology.orchestra.run.smithi192.stdout: rados osd op timeout = 900 2023-10-06T03:30:48.418 INFO:teuthology.orchestra.run.smithi192.stdout:[mon.a] 2023-10-06T03:30:48.419 INFO:teuthology.orchestra.run.smithi192.stdout:[mon.b] 2023-10-06T03:30:48.419 INFO:teuthology.orchestra.run.smithi192.stdout:[mon.c] 2023-10-06T03:30:48.419 INFO:teuthology.orchestra.run.smithi192.stdout:[mds.a] 2023-10-06T03:30:48.419 INFO:teuthology.orchestra.run.smithi192.stdout:[mds.c] 2023-10-06T03:30:48.419 INFO:teuthology.orchestra.run.smithi192.stdout:[mds.b] 2023-10-06T03:30:48.419 INFO:teuthology.orchestra.run.smithi192.stdout:[mds.d] 2023-10-06T03:30:48.419 INFO:teuthology.orchestra.run.smithi192.stdout:[mds] 2023-10-06T03:30:48.419 INFO:teuthology.orchestra.run.smithi192.stdout: debug mds = 20 2023-10-06T03:30:48.419 INFO:teuthology.orchestra.run.smithi192.stdout: debug ms = 1 2023-10-06T03:30:48.419 INFO:teuthology.orchestra.run.smithi192.stdout: mds debug frag = True 2023-10-06T03:30:48.419 INFO:teuthology.orchestra.run.smithi192.stdout: mds debug scatterstat = True 2023-10-06T03:30:48.419 INFO:teuthology.orchestra.run.smithi192.stdout: mds op complaint time = 180 2023-10-06T03:30:48.419 INFO:teuthology.orchestra.run.smithi192.stdout: mds verify scatter = True 2023-10-06T03:30:48.420 INFO:teuthology.orchestra.run.smithi192.stdout: osd op complaint time = 180 2023-10-06T03:30:48.420 INFO:teuthology.orchestra.run.smithi192.stdout: rados mon op timeout = 900 2023-10-06T03:30:48.420 INFO:teuthology.orchestra.run.smithi192.stdout: rados osd op timeout = 900 2023-10-06T03:30:48.421 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi110.front.sepia.ceph.com... 2023-10-06T03:30:48.421 INFO:teuthology.orchestra.run:Running command with timeout 1800 2023-10-06T03:30:48.421 DEBUG:teuthology.orchestra.run.smithi110:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2023-10-06T03:30:48.477 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi110.stderr:ceph-fuse[42294]: fuse finished with error 0 and tester_r 0 2023-10-06T03:30:48.479 INFO:teuthology.orchestra.run:waiting for 900 2023-10-06T03:30:54.580 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi110.front.sepia.ceph.com 2023-10-06T03:30:54.581 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-06T03:30:54.581 DEBUG:teuthology.orchestra.run.smithi110:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2023-10-06T03:30:54.589 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-10-06T03:30:54.748 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:54.744+0000 7f7b36d3e700 1 Processor -- start 2023-10-06T03:30:54.748 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:54.744+0000 7f7b36d3e700 1 -- start start 2023-10-06T03:30:54.750 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:54.744+0000 7f7b36d3e700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7b300cfe50 0x7f7b300cf700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:54.750 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:54.744+0000 7f7b36d3e700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f7b300d0220 0x7f7b3006acb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:54.750 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:54.744+0000 7f7b36d3e700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f7b300b2b00 0x7f7b300b2f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:54.750 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:54.744+0000 7f7b36d3e700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7b300b16e0 con 0x7f7b300d0220 2023-10-06T03:30:54.750 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:54.744+0000 7f7b36d3e700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7b300d0630 con 0x7f7b300b2b00 2023-10-06T03:30:54.750 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:54.744+0000 7f7b36d3e700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b300b10d0 con 0x7f7b300cfe50 2023-10-06T03:30:54.750 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:54.744+0000 7f7b3553b700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f7b300d0220 0x7f7b3006acb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:54.750 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:54.744+0000 7f7b3553b700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f7b300d0220 0x7f7b3006acb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:51046/0 (socket says 172.21.15.110:51046) 2023-10-06T03:30:54.750 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:54.744+0000 7f7b3553b700 1 -- 172.21.15.110:0/4268943388 learned_addr learned my addr 172.21.15.110:0/4268943388 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:30:54.750 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:54.744+0000 7f7b35d3c700 1 --2- 172.21.15.110:0/4268943388 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7b300cfe50 0x7f7b300cf700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:54.751 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:54.744+0000 7f7b3653d700 1 --2- 172.21.15.110:0/4268943388 >> v2:172.21.15.192:3300/0 conn(0x7f7b300b2b00 0x7f7b300b2f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:54.751 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:54.744+0000 7f7b3553b700 1 -- 172.21.15.110:0/4268943388 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7b300cfe50 msgr2=0x7f7b300cf700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:30:54.751 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:54.744+0000 7f7b3553b700 1 --2- 172.21.15.110:0/4268943388 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7b300cfe50 0x7f7b300cf700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:54.751 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:54.744+0000 7f7b3553b700 1 -- 172.21.15.110:0/4268943388 >> v2:172.21.15.192:3300/0 conn(0x7f7b300b2b00 msgr2=0x7f7b300b2f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:30:54.751 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:54.744+0000 7f7b3553b700 1 --2- 172.21.15.110:0/4268943388 >> v2:172.21.15.192:3300/0 conn(0x7f7b300b2b00 0x7f7b300b2f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:54.751 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:54.744+0000 7f7b3553b700 1 -- 172.21.15.110:0/4268943388 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b3006b1f0 con 0x7f7b300d0220 2023-10-06T03:30:54.752 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:54.744+0000 7f7b3553b700 1 --2- 172.21.15.110:0/4268943388 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7b300d0220 0x7f7b3006acb0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f7b300b5d00 tx=0x7f7b20009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4840e69618c2ba2 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:30:54.752 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:54.744+0000 7f7b34d3a700 1 -- 172.21.15.110:0/4268943388 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b2000e070 con 0x7f7b300d0220 2023-10-06T03:30:54.752 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:54.744+0000 7f7b34d3a700 1 -- 172.21.15.110:0/4268943388 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b2000b970 con 0x7f7b300d0220 2023-10-06T03:30:54.753 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:54.744+0000 7f7b34d3a700 1 -- 172.21.15.110:0/4268943388 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b200092e0 con 0x7f7b300d0220 2023-10-06T03:30:54.753 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:54.748+0000 7f7b36d3e700 1 -- 172.21.15.110:0/4268943388 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7b300d0220 msgr2=0x7f7b3006acb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:54.753 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:54.748+0000 7f7b36d3e700 1 --2- 172.21.15.110:0/4268943388 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7b300d0220 0x7f7b3006acb0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f7b300b5d00 tx=0x7f7b20009840 comp rx=0 tx=0).stop 2023-10-06T03:30:54.753 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:54.748+0000 7f7b36d3e700 1 -- 172.21.15.110:0/4268943388 shutdown_connections 2023-10-06T03:30:54.753 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:54.748+0000 7f7b36d3e700 1 --2- 172.21.15.110:0/4268943388 >> v2:172.21.15.192:3300/0 conn(0x7f7b300b2b00 0x7f7b300b2f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:54.753 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:54.748+0000 7f7b36d3e700 1 --2- 172.21.15.110:0/4268943388 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7b300d0220 0x7f7b3006acb0 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:54.753 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:54.748+0000 7f7b36d3e700 1 --2- 172.21.15.110:0/4268943388 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7b300cfe50 0x7f7b300cf700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:54.753 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:54.748+0000 7f7b36d3e700 1 -- 172.21.15.110:0/4268943388 >> 172.21.15.110:0/4268943388 conn(0x7f7b30068360 msgr2=0x7f7b30068760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:30:54.753 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:54.748+0000 7f7b36d3e700 1 -- 172.21.15.110:0/4268943388 shutdown_connections 2023-10-06T03:30:54.753 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:54.748+0000 7f7b36d3e700 1 -- 172.21.15.110:0/4268943388 wait complete. 2023-10-06T03:30:54.754 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:54.748+0000 7f7b36d3e700 1 Processor -- start 2023-10-06T03:30:54.754 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:54.748+0000 7f7b36d3e700 1 -- start start 2023-10-06T03:30:54.755 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:54.748+0000 7f7b36d3e700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7b300b2b00 0x7f7b300879c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:54.756 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:54.748+0000 7f7b36d3e700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7b300cfe50 0x7f7b30089fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:54.756 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:54.748+0000 7f7b36d3e700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7b300d0220 0x7f7b300b50a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:54.756 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:54.748+0000 7f7b36d3e700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b30087f00 con 0x7f7b300d0220 2023-10-06T03:30:54.756 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:54.748+0000 7f7b36d3e700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b3008a4e0 con 0x7f7b300cfe50 2023-10-06T03:30:54.756 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:54.748+0000 7f7b36d3e700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b30088140 con 0x7f7b300b2b00 2023-10-06T03:30:54.756 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:54.752+0000 7f7b3653d700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7b300d0220 0x7f7b300b50a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:54.756 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:54.752+0000 7f7b3653d700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7b300d0220 0x7f7b300b50a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:51050/0 (socket says 172.21.15.110:51050) 2023-10-06T03:30:54.756 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:54.752+0000 7f7b3653d700 1 -- 172.21.15.110:0/1692253427 learned_addr learned my addr 172.21.15.110:0/1692253427 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:30:54.756 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:54.752+0000 7f7b35d3c700 1 --2- 172.21.15.110:0/1692253427 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7b300b2b00 0x7f7b300879c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:54.756 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:54.752+0000 7f7b3553b700 1 --2- 172.21.15.110:0/1692253427 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7b300cfe50 0x7f7b30089fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:54.757 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:54.752+0000 7f7b3653d700 1 -- 172.21.15.110:0/1692253427 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7b300b2b00 msgr2=0x7f7b300879c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:54.757 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:54.752+0000 7f7b3653d700 1 --2- 172.21.15.110:0/1692253427 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7b300b2b00 0x7f7b300879c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:54.758 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:54.752+0000 7f7b3653d700 1 -- 172.21.15.110:0/1692253427 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7b300cfe50 msgr2=0x7f7b30089fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:54.758 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:54.752+0000 7f7b3653d700 1 --2- 172.21.15.110:0/1692253427 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7b300cfe50 0x7f7b30089fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:54.758 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:54.752+0000 7f7b3653d700 1 -- 172.21.15.110:0/1692253427 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b20009ae0 con 0x7f7b300d0220 2023-10-06T03:30:54.758 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:54.752+0000 7f7b3653d700 1 --2- 172.21.15.110:0/1692253427 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7b300d0220 0x7f7b300b50a0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f7b2800ea50 tx=0x7f7b280091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:30:54.758 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:54.752+0000 7f7b2dffb700 1 -- 172.21.15.110:0/1692253427 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b2800ee90 con 0x7f7b300d0220 2023-10-06T03:30:54.758 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:54.752+0000 7f7b2dffb700 1 -- 172.21.15.110:0/1692253427 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b2800c850 con 0x7f7b300d0220 2023-10-06T03:30:54.758 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:54.752+0000 7f7b36d3e700 1 -- 172.21.15.110:0/1692253427 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b30088420 con 0x7f7b300d0220 2023-10-06T03:30:54.758 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:54.752+0000 7f7b2dffb700 1 -- 172.21.15.110:0/1692253427 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b2800cc10 con 0x7f7b300d0220 2023-10-06T03:30:54.758 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:54.752+0000 7f7b36d3e700 1 -- 172.21.15.110:0/1692253427 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b300889e0 con 0x7f7b300d0220 2023-10-06T03:30:54.759 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:54.752+0000 7f7b36d3e700 1 -- 172.21.15.110:0/1692253427 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b30064770 con 0x7f7b300d0220 2023-10-06T03:30:54.760 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:54.756+0000 7f7b2dffb700 1 -- 172.21.15.110:0/1692253427 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 9) v1 ==== 80135+0+0 (secure 0 0 0) 0x7f7b28013470 con 0x7f7b300d0220 2023-10-06T03:30:54.761 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:54.756+0000 7f7b2dffb700 1 --2- 172.21.15.110:0/1692253427 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f7b1c060780 0x7f7b1c062c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:54.761 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:54.756+0000 7f7b2dffb700 1 -- 172.21.15.110:0/1692253427 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 6636+0+0 (secure 0 0 0) 0x7f7b2807d0b0 con 0x7f7b300d0220 2023-10-06T03:30:54.761 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:54.756+0000 7f7b35d3c700 1 --2- 172.21.15.110:0/1692253427 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f7b1c060780 0x7f7b1c062c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:54.761 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:54.756+0000 7f7b35d3c700 1 --2- 172.21.15.110:0/1692253427 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f7b1c060780 0x7f7b1c062c30 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f7b24009db0 tx=0x7f7b240091a0 comp rx=0 tx=0).ready entity=mgr.5329 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:30:54.768 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:54.764+0000 7f7b2dffb700 1 -- 172.21.15.110:0/1692253427 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f7b2804d080 con 0x7f7b300d0220 2023-10-06T03:30:54.966 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:54.960+0000 7f7b36d3e700 1 -- 172.21.15.110:0/1692253427 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f7b3006b4a0 con 0x7f7b300d0220 2023-10-06T03:30:54.966 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:54.960+0000 7f7b2dffb700 1 -- 172.21.15.110:0/1692253427 <== mon.0 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v16) v1 ==== 53+0+73 (secure 0 0 0) 0x7f7b280506a0 con 0x7f7b300d0220 2023-10-06T03:30:54.967 INFO:teuthology.orchestra.run.smithi110.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-10-06T03:30:54.969 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:54.964+0000 7f7b36d3e700 1 -- 172.21.15.110:0/1692253427 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f7b1c060780 msgr2=0x7f7b1c062c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:54.969 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:54.964+0000 7f7b36d3e700 1 --2- 172.21.15.110:0/1692253427 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f7b1c060780 0x7f7b1c062c30 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f7b24009db0 tx=0x7f7b240091a0 comp rx=0 tx=0).stop 2023-10-06T03:30:54.969 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:54.964+0000 7f7b36d3e700 1 -- 172.21.15.110:0/1692253427 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7b300d0220 msgr2=0x7f7b300b50a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:54.969 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:54.964+0000 7f7b36d3e700 1 --2- 172.21.15.110:0/1692253427 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7b300d0220 0x7f7b300b50a0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f7b2800ea50 tx=0x7f7b280091a0 comp rx=0 tx=0).stop 2023-10-06T03:30:54.969 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:54.964+0000 7f7b36d3e700 1 -- 172.21.15.110:0/1692253427 shutdown_connections 2023-10-06T03:30:54.970 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:54.964+0000 7f7b36d3e700 1 --2- 172.21.15.110:0/1692253427 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f7b1c060780 0x7f7b1c062c30 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:54.970 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:54.964+0000 7f7b36d3e700 1 --2- 172.21.15.110:0/1692253427 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7b300d0220 0x7f7b300b50a0 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:54.970 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:54.964+0000 7f7b36d3e700 1 --2- 172.21.15.110:0/1692253427 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7b300cfe50 0x7f7b30089fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:54.970 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:54.964+0000 7f7b36d3e700 1 --2- 172.21.15.110:0/1692253427 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7b300b2b00 0x7f7b300879c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:54.970 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:54.964+0000 7f7b36d3e700 1 -- 172.21.15.110:0/1692253427 >> 172.21.15.110:0/1692253427 conn(0x7f7b30068360 msgr2=0x7f7b301455f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:30:54.970 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:54.964+0000 7f7b36d3e700 1 -- 172.21.15.110:0/1692253427 shutdown_connections 2023-10-06T03:30:54.970 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:54.964+0000 7f7b36d3e700 1 -- 172.21.15.110:0/1692253427 wait complete. 2023-10-06T03:30:54.981 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2023-10-06T03:30:54.981 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-10-06T03:30:54.981 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi110.front.sepia.ceph.com 2023-10-06T03:30:54.981 INFO:tasks.cephfs.mount:self.client.name = client.0 2023-10-06T03:30:54.981 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-10-06T03:30:54.981 INFO:tasks.cephfs.mount:self.cephfs_name = None 2023-10-06T03:30:54.981 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2023-10-06T03:30:54.981 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-10-06T03:30:54.981 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-10-06T03:30:54.982 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-06T03:30:54.982 DEBUG:teuthology.orchestra.run.smithi110:> ip addr 2023-10-06T03:30:54.990 INFO:teuthology.orchestra.run.smithi110.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-10-06T03:30:54.990 INFO:teuthology.orchestra.run.smithi110.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-10-06T03:30:54.990 INFO:teuthology.orchestra.run.smithi110.stdout: inet 127.0.0.1/8 scope host lo 2023-10-06T03:30:54.990 INFO:teuthology.orchestra.run.smithi110.stdout: valid_lft forever preferred_lft forever 2023-10-06T03:30:54.990 INFO:teuthology.orchestra.run.smithi110.stdout: inet6 ::1/128 scope host 2023-10-06T03:30:54.991 INFO:teuthology.orchestra.run.smithi110.stdout: valid_lft forever preferred_lft forever 2023-10-06T03:30:54.991 INFO:teuthology.orchestra.run.smithi110.stdout:2: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2023-10-06T03:30:54.991 INFO:teuthology.orchestra.run.smithi110.stdout: link/ether 0c:c4:7a:d8:db:14 brd ff:ff:ff:ff:ff:ff 2023-10-06T03:30:54.991 INFO:teuthology.orchestra.run.smithi110.stdout:3: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000 2023-10-06T03:30:54.991 INFO:teuthology.orchestra.run.smithi110.stdout: link/ether 0c:c4:7a:d8:db:15 brd ff:ff:ff:ff:ff:ff 2023-10-06T03:30:54.991 INFO:teuthology.orchestra.run.smithi110.stdout:4: enp3s0f0: mtu 1500 qdisc noop state DOWN group default qlen 1000 2023-10-06T03:30:54.991 INFO:teuthology.orchestra.run.smithi110.stdout: link/ether 0c:c4:7a:8f:cc:34 brd ff:ff:ff:ff:ff:ff 2023-10-06T03:30:54.991 INFO:teuthology.orchestra.run.smithi110.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2023-10-06T03:30:54.991 INFO:teuthology.orchestra.run.smithi110.stdout: link/ether 0c:c4:7a:8f:cc:35 brd ff:ff:ff:ff:ff:ff 2023-10-06T03:30:54.991 INFO:teuthology.orchestra.run.smithi110.stdout: inet 172.21.15.110/20 brd 172.21.15.255 scope global dynamic enp3s0f1 2023-10-06T03:30:54.991 INFO:teuthology.orchestra.run.smithi110.stdout: valid_lft 42535sec preferred_lft 42535sec 2023-10-06T03:30:54.992 INFO:teuthology.orchestra.run.smithi110.stdout: inet6 fe80::ec4:7aff:fe8f:cc35/64 scope link 2023-10-06T03:30:54.992 INFO:teuthology.orchestra.run.smithi110.stdout: valid_lft forever preferred_lft forever 2023-10-06T03:30:54.992 INFO:teuthology.orchestra.run.smithi110.stdout:6: docker0: mtu 1500 qdisc noqueue state DOWN group default 2023-10-06T03:30:54.992 INFO:teuthology.orchestra.run.smithi110.stdout: link/ether 02:42:dc:de:aa:fb brd ff:ff:ff:ff:ff:ff 2023-10-06T03:30:54.992 INFO:teuthology.orchestra.run.smithi110.stdout: inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 2023-10-06T03:30:54.992 INFO:teuthology.orchestra.run.smithi110.stdout: valid_lft forever preferred_lft forever 2023-10-06T03:30:54.992 INFO:teuthology.orchestra.run.smithi110.stdout:9: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-10-06T03:30:54.992 INFO:teuthology.orchestra.run.smithi110.stdout: link/ether 5e:32:2b:b4:0f:e6 brd ff:ff:ff:ff:ff:ff 2023-10-06T03:30:54.992 INFO:teuthology.orchestra.run.smithi110.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-10-06T03:30:54.992 INFO:teuthology.orchestra.run.smithi110.stdout: valid_lft forever preferred_lft forever 2023-10-06T03:30:54.992 INFO:teuthology.orchestra.run.smithi110.stdout: inet6 fe80::34e3:85ff:feb6:e924/64 scope link 2023-10-06T03:30:54.992 INFO:teuthology.orchestra.run.smithi110.stdout: valid_lft forever preferred_lft forever 2023-10-06T03:30:54.993 INFO:teuthology.orchestra.run.smithi110.stdout:10: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-10-06T03:30:54.993 INFO:teuthology.orchestra.run.smithi110.stdout: link/ether 5e:32:2b:b4:0f:e6 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2023-10-06T03:30:54.993 INFO:teuthology.orchestra.run.smithi110.stdout: inet6 fe80::5c32:2bff:feb4:fe6/64 scope link 2023-10-06T03:30:54.993 INFO:teuthology.orchestra.run.smithi110.stdout: valid_lft forever preferred_lft forever 2023-10-06T03:30:54.993 INFO:teuthology.orchestra.run.smithi110.stdout:11: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-10-06T03:30:54.993 INFO:teuthology.orchestra.run.smithi110.stdout: link/ether 76:f9:1f:3d:03:15 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2023-10-06T03:30:54.993 INFO:teuthology.orchestra.run.smithi110.stdout: inet6 fe80::74f9:1fff:fe3d:315/64 scope link 2023-10-06T03:30:54.993 INFO:teuthology.orchestra.run.smithi110.stdout: valid_lft forever preferred_lft forever 2023-10-06T03:30:54.993 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-06T03:30:54.994 DEBUG:teuthology.orchestra.run.smithi110:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-10-06T03:30:55.008 INFO:teuthology.orchestra.run.smithi110.stdout:1 2023-10-06T03:30:55.009 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-06T03:30:55.009 DEBUG:teuthology.orchestra.run.smithi110:> route 2023-10-06T03:30:55.060 INFO:teuthology.orchestra.run.smithi110.stdout:Kernel IP routing table 2023-10-06T03:30:55.060 INFO:teuthology.orchestra.run.smithi110.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-10-06T03:30:55.060 INFO:teuthology.orchestra.run.smithi110.stdout:default _gateway 0.0.0.0 UG 0 0 0 enp3s0f1 2023-10-06T03:30:55.060 INFO:teuthology.orchestra.run.smithi110.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2023-10-06T03:30:55.060 INFO:teuthology.orchestra.run.smithi110.stdout:172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 2023-10-06T03:30:55.060 INFO:teuthology.orchestra.run.smithi110.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 0 0 0 enp3s0f1 2023-10-06T03:30:55.060 INFO:teuthology.orchestra.run.smithi110.stdout:_gateway 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2023-10-06T03:30:55.060 INFO:teuthology.orchestra.run.smithi110.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-10-06T03:30:55.061 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-06T03:30:55.061 DEBUG:teuthology.orchestra.run.smithi110:> (cd / && exec bash -c ' 2023-10-06T03:30:55.061 DEBUG:teuthology.orchestra.run.smithi110:> set -e 2023-10-06T03:30:55.061 DEBUG:teuthology.orchestra.run.smithi110:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2023-10-06T03:30:55.061 DEBUG:teuthology.orchestra.run.smithi110:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2023-10-06T03:30:55.061 DEBUG:teuthology.orchestra.run.smithi110:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2023-10-06T03:30:55.061 DEBUG:teuthology.orchestra.run.smithi110:> ') 2023-10-06T03:30:55.101 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-06T03:30:55.101 DEBUG:teuthology.orchestra.run.smithi110:> ip netns list 2023-10-06T03:30:55.107 INFO:teuthology.orchestra.run.smithi110.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2023-10-06T03:30:55.108 INFO:teuthology.orchestra.run.smithi110.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2023-10-06T03:30:55.108 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-10-06T03:30:55.108 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-06T03:30:55.108 DEBUG:teuthology.orchestra.run.smithi110:> sudo ip link set brx.0 up 2023-10-06T03:30:55.124 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2023-10-06T03:30:55.124 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-06T03:30:55.124 DEBUG:teuthology.orchestra.run.smithi110:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2023-10-06T03:30:55.130 INFO:teuthology.orchestra.run.smithi110.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2023-10-06T03:30:55.131 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-06T03:30:55.131 DEBUG:teuthology.orchestra.run.smithi110:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2023-10-06T03:30:55.135 DEBUG:teuthology.orchestra.run.smithi110:> sudo modprobe fuse 2023-10-06T03:30:55.150 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-10-06T03:30:55.150 DEBUG:teuthology.orchestra.run.smithi110:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-10-06T03:30:55.161 INFO:teuthology.orchestra.run.smithi110.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-10-06T03:30:55.162 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-10-06T03:30:55.163 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-06T03:30:55.163 DEBUG:teuthology.orchestra.run.smithi110:> ls /sys/fs/fuse/connections 2023-10-06T03:30:55.214 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2023-10-06T03:30:55.214 DEBUG:teuthology.orchestra.run.smithi110:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2023-10-06T03:30:55.216 DEBUG:teuthology.orchestra.run.smithi110:> sudo modprobe fuse 2023-10-06T03:30:55.232 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-10-06T03:30:55.232 DEBUG:teuthology.orchestra.run.smithi110:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-10-06T03:30:55.245 INFO:teuthology.orchestra.run.smithi110.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-10-06T03:30:55.246 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-10-06T03:30:55.247 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-06T03:30:55.247 DEBUG:teuthology.orchestra.run.smithi110:> ls /sys/fs/fuse/connections 2023-10-06T03:30:55.295 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi110.stderr:2023-10-06T03:30:55.288+0000 7fd92ba79380 -1 init, newargv = 0x5631ffc92110 newargc=15 2023-10-06T03:30:55.295 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi110.stderr:ceph-fuse[42767]: starting ceph client 2023-10-06T03:30:55.304 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi110.stderr:ceph-fuse[42767]: starting fuse 2023-10-06T03:30:56.353 DEBUG:teuthology.orchestra.run.smithi110:> sudo modprobe fuse 2023-10-06T03:30:56.369 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-10-06T03:30:56.369 DEBUG:teuthology.orchestra.run.smithi110:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-10-06T03:30:56.425 INFO:teuthology.orchestra.run.smithi110.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-10-06T03:30:56.426 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-10-06T03:30:56.426 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-06T03:30:56.427 DEBUG:teuthology.orchestra.run.smithi110:> ls /sys/fs/fuse/connections 2023-10-06T03:30:56.433 INFO:teuthology.orchestra.run.smithi110.stdout:52 2023-10-06T03:30:56.433 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [52] 2023-10-06T03:30:56.433 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2023-10-06T03:30:56.433 DEBUG:teuthology.orchestra.run.smithi110:> import glob 2023-10-06T03:30:56.433 DEBUG:teuthology.orchestra.run.smithi110:> import re 2023-10-06T03:30:56.433 DEBUG:teuthology.orchestra.run.smithi110:> import os 2023-10-06T03:30:56.433 DEBUG:teuthology.orchestra.run.smithi110:> import subprocess 2023-10-06T03:30:56.433 DEBUG:teuthology.orchestra.run.smithi110:> 2023-10-06T03:30:56.433 DEBUG:teuthology.orchestra.run.smithi110:> def _find_admin_socket(client_name): 2023-10-06T03:30:56.434 DEBUG:teuthology.orchestra.run.smithi110:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2023-10-06T03:30:56.434 DEBUG:teuthology.orchestra.run.smithi110:> files = glob.glob(asok_path) 2023-10-06T03:30:56.434 DEBUG:teuthology.orchestra.run.smithi110:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2023-10-06T03:30:56.434 DEBUG:teuthology.orchestra.run.smithi110:> 2023-10-06T03:30:56.434 DEBUG:teuthology.orchestra.run.smithi110:> # Given a non-glob path, it better be there 2023-10-06T03:30:56.434 DEBUG:teuthology.orchestra.run.smithi110:> if "*" not in asok_path: 2023-10-06T03:30:56.434 DEBUG:teuthology.orchestra.run.smithi110:> assert(len(files) == 1) 2023-10-06T03:30:56.434 DEBUG:teuthology.orchestra.run.smithi110:> return files[0] 2023-10-06T03:30:56.434 DEBUG:teuthology.orchestra.run.smithi110:> 2023-10-06T03:30:56.434 DEBUG:teuthology.orchestra.run.smithi110:> for f in files: 2023-10-06T03:30:56.434 DEBUG:teuthology.orchestra.run.smithi110:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2023-10-06T03:30:56.434 DEBUG:teuthology.orchestra.run.smithi110:> if os.path.exists("/proc/{0}".format(pid)): 2023-10-06T03:30:56.435 DEBUG:teuthology.orchestra.run.smithi110:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2023-10-06T03:30:56.435 DEBUG:teuthology.orchestra.run.smithi110:> contents = proc_f.read() 2023-10-06T03:30:56.435 DEBUG:teuthology.orchestra.run.smithi110:> if mountpoint in contents: 2023-10-06T03:30:56.435 DEBUG:teuthology.orchestra.run.smithi110:> return f 2023-10-06T03:30:56.435 DEBUG:teuthology.orchestra.run.smithi110:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2023-10-06T03:30:56.435 DEBUG:teuthology.orchestra.run.smithi110:> 2023-10-06T03:30:56.435 DEBUG:teuthology.orchestra.run.smithi110:> print(_find_admin_socket("client.0")) 2023-10-06T03:30:56.435 DEBUG:teuthology.orchestra.run.smithi110:> ' 2023-10-06T03:30:56.518 INFO:teuthology.orchestra.run.smithi110.stdout:/var/run/ceph/ceph-client.0.42767.asok 2023-10-06T03:30:56.703 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.42767.asok 2023-10-06T03:30:56.703 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-06T03:30:56.703 DEBUG:teuthology.orchestra.run.smithi110:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.42767.asok status 2023-10-06T03:30:56.839 INFO:teuthology.orchestra.run.smithi110.stdout:{ 2023-10-06T03:30:56.839 INFO:teuthology.orchestra.run.smithi110.stdout: "metadata": { 2023-10-06T03:30:56.839 INFO:teuthology.orchestra.run.smithi110.stdout: "ceph_sha1": "736ac5ba72cb47af7d6f6666b1e100f7e20a20e2", 2023-10-06T03:30:56.839 INFO:teuthology.orchestra.run.smithi110.stdout: "ceph_version": "ceph version 17.2.6-1296-g736ac5ba (736ac5ba72cb47af7d6f6666b1e100f7e20a20e2) quincy (stable)", 2023-10-06T03:30:56.839 INFO:teuthology.orchestra.run.smithi110.stdout: "entity_id": "0", 2023-10-06T03:30:56.839 INFO:teuthology.orchestra.run.smithi110.stdout: "hostname": "smithi110", 2023-10-06T03:30:56.839 INFO:teuthology.orchestra.run.smithi110.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2023-10-06T03:30:56.839 INFO:teuthology.orchestra.run.smithi110.stdout: "pid": "42767", 2023-10-06T03:30:56.839 INFO:teuthology.orchestra.run.smithi110.stdout: "root": "/" 2023-10-06T03:30:56.840 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2023-10-06T03:30:56.840 INFO:teuthology.orchestra.run.smithi110.stdout: "dentry_count": 0, 2023-10-06T03:30:56.840 INFO:teuthology.orchestra.run.smithi110.stdout: "dentry_pinned_count": 0, 2023-10-06T03:30:56.840 INFO:teuthology.orchestra.run.smithi110.stdout: "id": 5478, 2023-10-06T03:30:56.840 INFO:teuthology.orchestra.run.smithi110.stdout: "inst": { 2023-10-06T03:30:56.840 INFO:teuthology.orchestra.run.smithi110.stdout: "name": { 2023-10-06T03:30:56.840 INFO:teuthology.orchestra.run.smithi110.stdout: "type": "client", 2023-10-06T03:30:56.840 INFO:teuthology.orchestra.run.smithi110.stdout: "num": 5478 2023-10-06T03:30:56.840 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2023-10-06T03:30:56.840 INFO:teuthology.orchestra.run.smithi110.stdout: "addr": { 2023-10-06T03:30:56.840 INFO:teuthology.orchestra.run.smithi110.stdout: "type": "v1", 2023-10-06T03:30:56.840 INFO:teuthology.orchestra.run.smithi110.stdout: "addr": "192.168.0.1:0", 2023-10-06T03:30:56.841 INFO:teuthology.orchestra.run.smithi110.stdout: "nonce": 1041190471 2023-10-06T03:30:56.841 INFO:teuthology.orchestra.run.smithi110.stdout: } 2023-10-06T03:30:56.841 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2023-10-06T03:30:56.841 INFO:teuthology.orchestra.run.smithi110.stdout: "addr": { 2023-10-06T03:30:56.841 INFO:teuthology.orchestra.run.smithi110.stdout: "type": "v1", 2023-10-06T03:30:56.841 INFO:teuthology.orchestra.run.smithi110.stdout: "addr": "192.168.0.1:0", 2023-10-06T03:30:56.841 INFO:teuthology.orchestra.run.smithi110.stdout: "nonce": 1041190471 2023-10-06T03:30:56.841 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2023-10-06T03:30:56.841 INFO:teuthology.orchestra.run.smithi110.stdout: "inst_str": "client.5478 192.168.0.1:0/1041190471", 2023-10-06T03:30:56.841 INFO:teuthology.orchestra.run.smithi110.stdout: "addr_str": "192.168.0.1:0/1041190471", 2023-10-06T03:30:56.841 INFO:teuthology.orchestra.run.smithi110.stdout: "inode_count": 1, 2023-10-06T03:30:56.841 INFO:teuthology.orchestra.run.smithi110.stdout: "mds_epoch": 16, 2023-10-06T03:30:56.842 INFO:teuthology.orchestra.run.smithi110.stdout: "osd_epoch": 43, 2023-10-06T03:30:56.842 INFO:teuthology.orchestra.run.smithi110.stdout: "osd_epoch_barrier": 0, 2023-10-06T03:30:56.842 INFO:teuthology.orchestra.run.smithi110.stdout: "blocklisted": false, 2023-10-06T03:30:56.842 INFO:teuthology.orchestra.run.smithi110.stdout: "fs_name": "cephfs" 2023-10-06T03:30:56.842 INFO:teuthology.orchestra.run.smithi110.stdout:} 2023-10-06T03:30:56.849 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-06T03:30:56.849 DEBUG:teuthology.orchestra.run.smithi110:> stat --file-system '--printf=%T 2023-10-06T03:30:56.849 DEBUG:teuthology.orchestra.run.smithi110:> ' -- /home/ubuntu/cephtest/mnt.0 2023-10-06T03:30:56.854 INFO:teuthology.orchestra.run.smithi110.stdout:fuseblk 2023-10-06T03:30:56.854 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2023-10-06T03:30:56.855 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-06T03:30:56.855 DEBUG:teuthology.orchestra.run.smithi110:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2023-10-06T03:30:56.869 INFO:tasks.cephfs.test_snap_schedules:expecting snap snap_test_dir1/.snap/scheduled-2023-10-06-03_31_00 in ~12s... 2023-10-06T03:30:56.869 DEBUG:tasks.cephfs.test_snap_schedules:expected_schedule: ['1M'], expected_retention: [{'M': 1}] 2023-10-06T03:30:56.870 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule list --fs cephfs --format json snap_test_dir1 2023-10-06T03:30:57.038 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:57.032+0000 7f3dbd26d700 1 Processor -- start 2023-10-06T03:30:57.038 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:57.032+0000 7f3dbd26d700 1 -- start start 2023-10-06T03:30:57.040 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:57.032+0000 7f3dbd26d700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3db8158cd0 0x7f3db81590a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:57.040 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:57.032+0000 7f3dbd26d700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f3db80ce490 0x7f3db80ce8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:57.040 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:57.036+0000 7f3dbd26d700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f3db80cafc0 0x7f3db80cb440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:57.040 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:57.036+0000 7f3dbd26d700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3db80cab30 con 0x7f3db80cafc0 2023-10-06T03:30:57.040 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:57.036+0000 7f3dbd26d700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3db80cef30 con 0x7f3db80ce490 2023-10-06T03:30:57.040 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:57.036+0000 7f3dbd26d700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3db80cf210 con 0x7f3db8158cd0 2023-10-06T03:30:57.041 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:57.036+0000 7f3db77fe700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f3db80cafc0 0x7f3db80cb440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:57.041 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:57.036+0000 7f3db77fe700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f3db80cafc0 0x7f3db80cb440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:51056/0 (socket says 172.21.15.110:51056) 2023-10-06T03:30:57.041 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:57.036+0000 7f3db77fe700 1 -- 172.21.15.110:0/2898912214 learned_addr learned my addr 172.21.15.110:0/2898912214 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:30:57.041 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:57.036+0000 7f3db67fc700 1 --2- 172.21.15.110:0/2898912214 >> v2:172.21.15.192:3300/0 conn(0x7f3db80ce490 0x7f3db80ce8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:57.041 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:57.036+0000 7f3db6ffd700 1 --2- 172.21.15.110:0/2898912214 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3db8158cd0 0x7f3db81590a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:57.041 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:57.036+0000 7f3db77fe700 1 -- 172.21.15.110:0/2898912214 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3db8158cd0 msgr2=0x7f3db81590a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:30:57.041 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:57.036+0000 7f3db77fe700 1 --2- 172.21.15.110:0/2898912214 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3db8158cd0 0x7f3db81590a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:57.041 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:57.036+0000 7f3db77fe700 1 -- 172.21.15.110:0/2898912214 >> v2:172.21.15.192:3300/0 conn(0x7f3db80ce490 msgr2=0x7f3db80ce8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:30:57.041 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:57.036+0000 7f3db77fe700 1 --2- 172.21.15.110:0/2898912214 >> v2:172.21.15.192:3300/0 conn(0x7f3db80ce490 0x7f3db80ce8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:57.041 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:57.036+0000 7f3db77fe700 1 -- 172.21.15.110:0/2898912214 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3db80cf4f0 con 0x7f3db80cafc0 2023-10-06T03:30:57.041 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:57.036+0000 7f3db77fe700 1 --2- 172.21.15.110:0/2898912214 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3db80cafc0 0x7f3db80cb440 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f3db80cda10 tx=0x7f3db0009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a24a90cd91a9b790 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:30:57.042 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:57.036+0000 7f3db5ffb700 1 -- 172.21.15.110:0/2898912214 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3db000e070 con 0x7f3db80cafc0 2023-10-06T03:30:57.042 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:57.036+0000 7f3db5ffb700 1 -- 172.21.15.110:0/2898912214 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3db000ba50 con 0x7f3db80cafc0 2023-10-06T03:30:57.043 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:57.036+0000 7f3db5ffb700 1 -- 172.21.15.110:0/2898912214 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3db00092e0 con 0x7f3db80cafc0 2023-10-06T03:30:57.043 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:57.036+0000 7f3dbd26d700 1 -- 172.21.15.110:0/2898912214 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3db80cafc0 msgr2=0x7f3db80cb440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:57.043 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:57.036+0000 7f3dbd26d700 1 --2- 172.21.15.110:0/2898912214 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3db80cafc0 0x7f3db80cb440 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f3db80cda10 tx=0x7f3db0009840 comp rx=0 tx=0).stop 2023-10-06T03:30:57.043 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:57.036+0000 7f3dbd26d700 1 -- 172.21.15.110:0/2898912214 shutdown_connections 2023-10-06T03:30:57.043 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:57.036+0000 7f3dbd26d700 1 --2- 172.21.15.110:0/2898912214 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3db80cafc0 0x7f3db80cb440 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:57.043 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:57.036+0000 7f3dbd26d700 1 --2- 172.21.15.110:0/2898912214 >> v2:172.21.15.192:3300/0 conn(0x7f3db80ce490 0x7f3db80ce8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:57.043 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:57.036+0000 7f3dbd26d700 1 --2- 172.21.15.110:0/2898912214 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3db8158cd0 0x7f3db81590a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:57.043 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:57.036+0000 7f3dbd26d700 1 -- 172.21.15.110:0/2898912214 >> 172.21.15.110:0/2898912214 conn(0x7f3db8067040 msgr2=0x7f3db8067440 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:30:57.043 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:57.036+0000 7f3dbd26d700 1 -- 172.21.15.110:0/2898912214 shutdown_connections 2023-10-06T03:30:57.043 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:57.036+0000 7f3dbd26d700 1 -- 172.21.15.110:0/2898912214 wait complete. 2023-10-06T03:30:57.044 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:57.040+0000 7f3dbd26d700 1 Processor -- start 2023-10-06T03:30:57.044 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:57.040+0000 7f3dbd26d700 1 -- start start 2023-10-06T03:30:57.045 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:57.040+0000 7f3dbd26d700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3db80cafc0 0x7f3db815a070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:57.045 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:57.040+0000 7f3dbd26d700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3db80ce490 0x7f3db815f4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:57.045 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:57.040+0000 7f3dbd26d700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3db8158cd0 0x7f3db815e200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:57.045 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:57.040+0000 7f3dbd26d700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3db80c70b0 con 0x7f3db80ce490 2023-10-06T03:30:57.045 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:57.040+0000 7f3dbd26d700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3db815a700 con 0x7f3db8158cd0 2023-10-06T03:30:57.046 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:57.040+0000 7f3dbd26d700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3db815a9e0 con 0x7f3db80cafc0 2023-10-06T03:30:57.046 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:57.040+0000 7f3db67fc700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3db80ce490 0x7f3db815f4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:57.046 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:57.040+0000 7f3db67fc700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3db80ce490 0x7f3db815f4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:51064/0 (socket says 172.21.15.110:51064) 2023-10-06T03:30:57.046 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:57.040+0000 7f3db67fc700 1 -- 172.21.15.110:0/2497370085 learned_addr learned my addr 172.21.15.110:0/2497370085 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:30:57.046 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:57.040+0000 7f3db77fe700 1 --2- 172.21.15.110:0/2497370085 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3db8158cd0 0x7f3db815e200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:57.046 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:57.040+0000 7f3db6ffd700 1 --2- 172.21.15.110:0/2497370085 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3db80cafc0 0x7f3db815a070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:57.047 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:57.040+0000 7f3db67fc700 1 -- 172.21.15.110:0/2497370085 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3db80cafc0 msgr2=0x7f3db815a070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:57.047 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:57.040+0000 7f3db67fc700 1 --2- 172.21.15.110:0/2497370085 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3db80cafc0 0x7f3db815a070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:57.047 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:57.040+0000 7f3db67fc700 1 -- 172.21.15.110:0/2497370085 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3db8158cd0 msgr2=0x7f3db815e200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:57.047 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:57.040+0000 7f3db67fc700 1 --2- 172.21.15.110:0/2497370085 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3db8158cd0 0x7f3db815e200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:57.048 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:57.040+0000 7f3db67fc700 1 -- 172.21.15.110:0/2497370085 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3db0009ae0 con 0x7f3db80ce490 2023-10-06T03:30:57.048 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:57.040+0000 7f3db67fc700 1 --2- 172.21.15.110:0/2497370085 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3db80ce490 0x7f3db815f4d0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f3da800d890 tx=0x7f3da8008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:30:57.048 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:57.040+0000 7f3d9effd700 1 -- 172.21.15.110:0/2497370085 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3da80125f0 con 0x7f3db80ce490 2023-10-06T03:30:57.048 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:57.040+0000 7f3d9effd700 1 -- 172.21.15.110:0/2497370085 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3da8012750 con 0x7f3db80ce490 2023-10-06T03:30:57.048 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:57.040+0000 7f3dbd26d700 1 -- 172.21.15.110:0/2497370085 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3db815acc0 con 0x7f3db80ce490 2023-10-06T03:30:57.048 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:57.040+0000 7f3d9effd700 1 -- 172.21.15.110:0/2497370085 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3da8012af0 con 0x7f3db80ce490 2023-10-06T03:30:57.048 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:57.040+0000 7f3dbd26d700 1 -- 172.21.15.110:0/2497370085 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3db815afa0 con 0x7f3db80ce490 2023-10-06T03:30:57.048 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:57.044+0000 7f3dbd26d700 1 -- 172.21.15.110:0/2497370085 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3db80c7220 con 0x7f3db80ce490 2023-10-06T03:30:57.049 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:57.044+0000 7f3d9effd700 1 -- 172.21.15.110:0/2497370085 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 9) v1 ==== 80135+0+0 (secure 0 0 0) 0x7f3da801e460 con 0x7f3db80ce490 2023-10-06T03:30:57.050 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:57.044+0000 7f3d9effd700 1 --2- 172.21.15.110:0/2497370085 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f3da4060780 0x7f3da4062c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:57.050 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:57.044+0000 7f3d9effd700 1 -- 172.21.15.110:0/2497370085 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 6636+0+0 (secure 0 0 0) 0x7f3da807d420 con 0x7f3db80ce490 2023-10-06T03:30:57.050 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:57.044+0000 7f3db6ffd700 1 --2- 172.21.15.110:0/2497370085 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f3da4060780 0x7f3da4062c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:57.051 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:57.044+0000 7f3db6ffd700 1 --2- 172.21.15.110:0/2497370085 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f3da4060780 0x7f3da4062c30 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f3dac0061f0 tx=0x7f3dac008040 comp rx=0 tx=0).ready entity=mgr.5329 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:30:57.057 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:57.052+0000 7f3d9effd700 1 -- 172.21.15.110:0/2497370085 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f3db80c7220 con 0x7f3db80ce490 2023-10-06T03:30:57.240 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:57.236+0000 7f3dbd26d700 1 -- 172.21.15.110:0/2497370085 --> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "snap_test_dir1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f3db80c71f0 con 0x7f3da4060780 2023-10-06T03:30:57.252 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:57.248+0000 7f3d9effd700 1 -- 172.21.15.110:0/2497370085 <== mgr.5329 v2:172.21.15.110:6832/33268 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+72 (secure 0 0 0) 0x7f3db80c71f0 con 0x7f3da4060780 2023-10-06T03:30:57.252 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-10-06T03:30:57.252 INFO:teuthology.orchestra.run.smithi110.stdout:{"path": "snap_test_dir1", "schedule": ["1M"], "retention": [{"M": 1}]} 2023-10-06T03:30:57.254 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:57.248+0000 7f3dbd26d700 1 -- 172.21.15.110:0/2497370085 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f3da4060780 msgr2=0x7f3da4062c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:57.254 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:57.248+0000 7f3dbd26d700 1 --2- 172.21.15.110:0/2497370085 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f3da4060780 0x7f3da4062c30 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f3dac0061f0 tx=0x7f3dac008040 comp rx=0 tx=0).stop 2023-10-06T03:30:57.255 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:57.248+0000 7f3dbd26d700 1 -- 172.21.15.110:0/2497370085 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3db80ce490 msgr2=0x7f3db815f4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:57.255 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:57.248+0000 7f3dbd26d700 1 --2- 172.21.15.110:0/2497370085 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3db80ce490 0x7f3db815f4d0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f3da800d890 tx=0x7f3da8008040 comp rx=0 tx=0).stop 2023-10-06T03:30:57.255 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:57.248+0000 7f3dbd26d700 1 -- 172.21.15.110:0/2497370085 shutdown_connections 2023-10-06T03:30:57.255 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:57.248+0000 7f3dbd26d700 1 --2- 172.21.15.110:0/2497370085 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f3da4060780 0x7f3da4062c30 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:57.255 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:57.248+0000 7f3dbd26d700 1 --2- 172.21.15.110:0/2497370085 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3db8158cd0 0x7f3db815e200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:57.255 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:57.248+0000 7f3dbd26d700 1 --2- 172.21.15.110:0/2497370085 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3db80ce490 0x7f3db815f4d0 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:57.255 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:57.248+0000 7f3dbd26d700 1 --2- 172.21.15.110:0/2497370085 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3db80cafc0 0x7f3db815a070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:57.255 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:57.248+0000 7f3dbd26d700 1 -- 172.21.15.110:0/2497370085 >> 172.21.15.110:0/2497370085 conn(0x7f3db8067040 msgr2=0x7f3db806b810 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:30:57.255 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:57.248+0000 7f3dbd26d700 1 -- 172.21.15.110:0/2497370085 shutdown_connections 2023-10-06T03:30:57.255 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:57.248+0000 7f3dbd26d700 1 -- 172.21.15.110:0/2497370085 wait complete. 2023-10-06T03:30:57.269 DEBUG:tasks.cephfs.test_snap_schedules:res= {"path": "snap_test_dir1", "schedule": ["1M"], "retention": [{"M": 1}]} 2023-10-06T03:30:57.269 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'path': 'snap_test_dir1', 'schedule': ['1M'], 'retention': [{'M': 1}]} 2023-10-06T03:30:57.270 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove --fs cephfs snap_test_dir1 2023-10-06T03:30:57.387 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:57.380+0000 7f12bd403700 1 Processor -- start 2023-10-06T03:30:57.387 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:57.380+0000 7f12bd403700 1 -- start start 2023-10-06T03:30:57.389 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:57.384+0000 7f12bd403700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f12b81535a0 0x7f12b80cfa30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:57.389 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:57.384+0000 7f12bd403700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f12b80cff70 0x7f12b80b1950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:57.389 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:57.384+0000 7f12bd403700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12b815d3b0 con 0x7f12b81531d0 2023-10-06T03:30:57.389 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:57.384+0000 7f12bd403700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f12b806b800 con 0x7f12b81535a0 2023-10-06T03:30:57.389 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:57.384+0000 7f12bd403700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12b815ca80 con 0x7f12b80cff70 2023-10-06T03:30:57.389 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:57.384+0000 7f12bcc02700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f12b80cff70 0x7f12b80b1950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:57.389 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:57.384+0000 7f12bcc02700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f12b80cff70 0x7f12b80b1950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3301/0 says I am v2:172.21.15.110:36008/0 (socket says 172.21.15.110:36008) 2023-10-06T03:30:57.389 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:57.384+0000 7f12bcc02700 1 -- 172.21.15.110:0/3989444272 learned_addr learned my addr 172.21.15.110:0/3989444272 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:30:57.390 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:57.384+0000 7f12b77fe700 1 --2- 172.21.15.110:0/3989444272 >> v2:172.21.15.192:3300/0 conn(0x7f12b81535a0 0x7f12b80cfa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:57.391 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:57.384+0000 7f12bcc02700 1 -- 172.21.15.110:0/3989444272 >> v2:172.21.15.192:3300/0 conn(0x7f12b81535a0 msgr2=0x7f12b80cfa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:30:57.391 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:57.384+0000 7f12bcc02700 1 --2- 172.21.15.110:0/3989444272 >> v2:172.21.15.192:3300/0 conn(0x7f12b81535a0 0x7f12b80cfa30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:57.391 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:57.384+0000 7f12bcc02700 1 -- 172.21.15.110:0/3989444272 >> v1:172.21.15.192:6789/0 conn(0x7f12b81531d0 legacy=0x7f12b80ce160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:57.391 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:57.384+0000 7f12bcc02700 1 -- 172.21.15.110:0/3989444272 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12b815cd60 con 0x7f12b80cff70 2023-10-06T03:30:57.391 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:57.384+0000 7f12b77fe700 1 --2- 172.21.15.110:0/3989444272 >> v2:172.21.15.192:3300/0 conn(0x7f12b81535a0 0x7f12b80cfa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-06T03:30:57.391 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:57.384+0000 7f12bcc02700 1 --2- 172.21.15.110:0/3989444272 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f12b80cff70 0x7f12b80b1950 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f12b0009f20 tx=0x7f12b0009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=12edf07fbad95a12 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:30:57.391 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:57.384+0000 7f12b6ffd700 1 -- 172.21.15.110:0/3989444272 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12b000e070 con 0x7f12b80cff70 2023-10-06T03:30:57.391 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:57.384+0000 7f12b6ffd700 1 -- 172.21.15.110:0/3989444272 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12b000b9d0 con 0x7f12b80cff70 2023-10-06T03:30:57.391 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:57.384+0000 7f12b6ffd700 1 -- 172.21.15.110:0/3989444272 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12b00092e0 con 0x7f12b80cff70 2023-10-06T03:30:57.392 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:57.388+0000 7f12bd403700 1 -- 172.21.15.110:0/3989444272 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f12b80cff70 msgr2=0x7f12b80b1950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:57.392 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:57.388+0000 7f12bd403700 1 --2- 172.21.15.110:0/3989444272 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f12b80cff70 0x7f12b80b1950 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f12b0009f20 tx=0x7f12b0009840 comp rx=0 tx=0).stop 2023-10-06T03:30:57.392 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:57.388+0000 7f12bd403700 1 -- 172.21.15.110:0/3989444272 shutdown_connections 2023-10-06T03:30:57.392 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:57.388+0000 7f12bd403700 1 --2- 172.21.15.110:0/3989444272 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f12b80cff70 0x7f12b80b1950 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:57.393 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:57.388+0000 7f12bd403700 1 --2- 172.21.15.110:0/3989444272 >> v2:172.21.15.192:3300/0 conn(0x7f12b81535a0 0x7f12b80cfa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:57.393 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:57.388+0000 7f12bd403700 1 -- 172.21.15.110:0/3989444272 >> 172.21.15.110:0/3989444272 conn(0x7f12b8068360 msgr2=0x7f12b8068760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:30:57.393 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:57.388+0000 7f12bd403700 1 -- 172.21.15.110:0/3989444272 shutdown_connections 2023-10-06T03:30:57.393 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:57.388+0000 7f12bd403700 1 -- 172.21.15.110:0/3989444272 wait complete. 2023-10-06T03:30:57.394 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:57.388+0000 7f12bd403700 1 Processor -- start 2023-10-06T03:30:57.394 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:57.388+0000 7f12bd403700 1 -- start start 2023-10-06T03:30:57.396 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:57.392+0000 7f12bd403700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f12b80cff70 0x7f12b81511b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:57.396 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:57.392+0000 7f12bd403700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f12b81531d0 0x7f12b80c77e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:57.396 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:57.392+0000 7f12bd403700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f12b81535a0 0x7f12b80c9290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:57.397 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:57.392+0000 7f12bd403700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12b80b3800 con 0x7f12b81535a0 2023-10-06T03:30:57.397 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:57.392+0000 7f12bd403700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12b81516f0 con 0x7f12b81531d0 2023-10-06T03:30:57.397 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:57.392+0000 7f12bd403700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12b80c59e0 con 0x7f12b80cff70 2023-10-06T03:30:57.397 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:57.392+0000 7f12bcc02700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f12b81535a0 0x7f12b80c9290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:57.397 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:57.392+0000 7f12bcc02700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f12b81535a0 0x7f12b80c9290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:51080/0 (socket says 172.21.15.110:51080) 2023-10-06T03:30:57.397 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:57.392+0000 7f12bcc02700 1 -- 172.21.15.110:0/407184255 learned_addr learned my addr 172.21.15.110:0/407184255 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:30:57.397 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:57.392+0000 7f12b77fe700 1 --2- 172.21.15.110:0/407184255 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f12b81531d0 0x7f12b80c77e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:57.397 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:57.392+0000 7f12b7fff700 1 --2- 172.21.15.110:0/407184255 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f12b80cff70 0x7f12b81511b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:57.398 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:57.392+0000 7f12bcc02700 1 -- 172.21.15.110:0/407184255 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f12b80cff70 msgr2=0x7f12b81511b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:57.398 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:57.392+0000 7f12bcc02700 1 --2- 172.21.15.110:0/407184255 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f12b80cff70 0x7f12b81511b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:57.398 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:57.392+0000 7f12bcc02700 1 -- 172.21.15.110:0/407184255 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f12b81531d0 msgr2=0x7f12b80c77e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:57.399 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:57.392+0000 7f12bcc02700 1 --2- 172.21.15.110:0/407184255 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f12b81531d0 0x7f12b80c77e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:57.399 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:57.392+0000 7f12bcc02700 1 -- 172.21.15.110:0/407184255 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12a8009420 con 0x7f12b81535a0 2023-10-06T03:30:57.399 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:57.392+0000 7f12bcc02700 1 --2- 172.21.15.110:0/407184255 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f12b81535a0 0x7f12b80c9290 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f12b000b610 tx=0x7f12b0018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:30:57.399 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:57.392+0000 7f12affff700 1 -- 172.21.15.110:0/407184255 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12b0008ad0 con 0x7f12b81535a0 2023-10-06T03:30:57.399 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:57.392+0000 7f12affff700 1 -- 172.21.15.110:0/407184255 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12b0008c30 con 0x7f12b81535a0 2023-10-06T03:30:57.399 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:57.392+0000 7f12bd403700 1 -- 172.21.15.110:0/407184255 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12b0009ae0 con 0x7f12b81535a0 2023-10-06T03:30:57.399 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:57.392+0000 7f12affff700 1 -- 172.21.15.110:0/407184255 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12b0026460 con 0x7f12b81535a0 2023-10-06T03:30:57.399 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:57.392+0000 7f12bd403700 1 -- 172.21.15.110:0/407184255 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12b80c5fa0 con 0x7f12b81535a0 2023-10-06T03:30:57.400 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:57.396+0000 7f12bd403700 1 -- 172.21.15.110:0/407184255 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f12b8064770 con 0x7f12b81535a0 2023-10-06T03:30:57.400 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:57.396+0000 7f12affff700 1 -- 172.21.15.110:0/407184255 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 9) v1 ==== 80135+0+0 (secure 0 0 0) 0x7f12b0003040 con 0x7f12b81535a0 2023-10-06T03:30:57.401 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:57.396+0000 7f12affff700 1 --2- 172.21.15.110:0/407184255 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f1298060780 0x7f1298062c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:57.401 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:57.396+0000 7f12affff700 1 -- 172.21.15.110:0/407184255 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 6636+0+0 (secure 0 0 0) 0x7f12b007ed90 con 0x7f12b81535a0 2023-10-06T03:30:57.401 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:57.396+0000 7f12b7fff700 1 --2- 172.21.15.110:0/407184255 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f1298060780 0x7f1298062c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:57.401 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:57.396+0000 7f12b7fff700 1 --2- 172.21.15.110:0/407184255 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f1298060780 0x7f1298062c30 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f12a0003a80 tx=0x7f12a0007ec0 comp rx=0 tx=0).ready entity=mgr.5329 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:30:57.411 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:57.408+0000 7f12affff700 1 -- 172.21.15.110:0/407184255 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f12b004ec50 con 0x7f12b81535a0 2023-10-06T03:30:57.593 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:57.588+0000 7f12bd403700 1 -- 172.21.15.110:0/407184255 --> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "fs": "cephfs", "path": "snap_test_dir1", "target": ["mon-mgr", ""]}) v1 -- 0x7f12b815d040 con 0x7f1298060780 2023-10-06T03:30:57.724 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:57.720+0000 7f12affff700 1 -- 172.21.15.110:0/407184255 <== mgr.5329 v2:172.21.15.110:6832/33268 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7f12b815d040 con 0x7f1298060780 2023-10-06T03:30:57.724 INFO:teuthology.orchestra.run.smithi110.stdout:Schedule removed for path snap_test_dir1 2023-10-06T03:30:57.728 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:57.724+0000 7f12bd403700 1 -- 172.21.15.110:0/407184255 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f1298060780 msgr2=0x7f1298062c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:57.728 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:57.724+0000 7f12bd403700 1 --2- 172.21.15.110:0/407184255 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f1298060780 0x7f1298062c30 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f12a0003a80 tx=0x7f12a0007ec0 comp rx=0 tx=0).stop 2023-10-06T03:30:57.729 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:57.724+0000 7f12bd403700 1 -- 172.21.15.110:0/407184255 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f12b81535a0 msgr2=0x7f12b80c9290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:57.729 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:57.724+0000 7f12bd403700 1 --2- 172.21.15.110:0/407184255 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f12b81535a0 0x7f12b80c9290 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f12b000b610 tx=0x7f12b0018000 comp rx=0 tx=0).stop 2023-10-06T03:30:57.729 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:57.724+0000 7f12bd403700 1 -- 172.21.15.110:0/407184255 shutdown_connections 2023-10-06T03:30:57.729 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:57.724+0000 7f12bd403700 1 --2- 172.21.15.110:0/407184255 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f1298060780 0x7f1298062c30 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:57.729 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:57.724+0000 7f12bd403700 1 --2- 172.21.15.110:0/407184255 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f12b81535a0 0x7f12b80c9290 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:57.729 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:57.724+0000 7f12bd403700 1 --2- 172.21.15.110:0/407184255 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f12b81531d0 0x7f12b80c77e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:57.729 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:57.724+0000 7f12bd403700 1 --2- 172.21.15.110:0/407184255 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f12b80cff70 0x7f12b81511b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:57.729 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:57.724+0000 7f12bd403700 1 -- 172.21.15.110:0/407184255 >> 172.21.15.110:0/407184255 conn(0x7f12b8068360 msgr2=0x7f12b80cd8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:30:57.729 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:57.724+0000 7f12bd403700 1 -- 172.21.15.110:0/407184255 shutdown_connections 2023-10-06T03:30:57.729 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:57.724+0000 7f12bd403700 1 -- 172.21.15.110:0/407184255 wait complete. 2023-10-06T03:30:57.744 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path snap_test_dir1 2023-10-06T03:30:57.744 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-06T03:30:57.745 DEBUG:teuthology.orchestra.run.smithi110:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-06T03:30:57.762 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-06T03:30:57.762 DEBUG:teuthology.orchestra.run.smithi110:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1) 2023-10-06T03:30:57.817 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module disable snap_schedule 2023-10-06T03:30:57.937 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:57.932+0000 7ffb1dabd700 1 Processor -- start 2023-10-06T03:30:57.938 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:57.932+0000 7ffb1dabd700 1 -- start start 2023-10-06T03:30:57.939 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:57.932+0000 7ffb1dabd700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7ffb180adc20 0x7ffb180b13f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:57.940 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:57.932+0000 7ffb1dabd700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ffb180cb910 0x7ffb180b0a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:57.940 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:57.932+0000 7ffb1dabd700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7ffb180b0f70 0x7ffb18155dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:57.940 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:57.932+0000 7ffb1dabd700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffb180cc4a0 con 0x7ffb180b0f70 2023-10-06T03:30:57.940 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:57.932+0000 7ffb1dabd700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffb18158e40 con 0x7ffb180adc20 2023-10-06T03:30:57.940 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:57.932+0000 7ffb1dabd700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb181596e0 con 0x7ffb180cb910 2023-10-06T03:30:57.940 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:57.932+0000 7ffb1d2bc700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7ffb180b0f70 0x7ffb18155dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:57.940 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:57.932+0000 7ffb1d2bc700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7ffb180b0f70 0x7ffb18155dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:51096/0 (socket says 172.21.15.110:51096) 2023-10-06T03:30:57.940 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:57.932+0000 7ffb1d2bc700 1 -- 172.21.15.110:0/3655772872 learned_addr learned my addr 172.21.15.110:0/3655772872 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:30:57.940 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:57.932+0000 7ffb0ffff700 1 --2- 172.21.15.110:0/3655772872 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ffb180cb910 0x7ffb180b0a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:57.940 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:57.932+0000 7ffb1cabb700 1 --2- 172.21.15.110:0/3655772872 >> v2:172.21.15.192:3300/0 conn(0x7ffb180adc20 0x7ffb180b13f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:57.940 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:57.932+0000 7ffb1d2bc700 1 -- 172.21.15.110:0/3655772872 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ffb180cb910 msgr2=0x7ffb180b0a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:30:57.941 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:57.932+0000 7ffb1d2bc700 1 --2- 172.21.15.110:0/3655772872 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ffb180cb910 0x7ffb180b0a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:57.941 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:57.932+0000 7ffb1d2bc700 1 -- 172.21.15.110:0/3655772872 >> v2:172.21.15.192:3300/0 conn(0x7ffb180adc20 msgr2=0x7ffb180b13f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:30:57.941 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:57.932+0000 7ffb1d2bc700 1 --2- 172.21.15.110:0/3655772872 >> v2:172.21.15.192:3300/0 conn(0x7ffb180adc20 0x7ffb180b13f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:57.941 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:57.932+0000 7ffb1d2bc700 1 -- 172.21.15.110:0/3655772872 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb180ae170 con 0x7ffb180b0f70 2023-10-06T03:30:57.941 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:57.932+0000 7ffb0ffff700 1 --2- 172.21.15.110:0/3655772872 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ffb180cb910 0x7ffb180b0a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-06T03:30:57.941 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:57.936+0000 7ffb1d2bc700 1 --2- 172.21.15.110:0/3655772872 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ffb180b0f70 0x7ffb18155dd0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7ffb18069a80 tx=0x7ffb10009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e6f12abfaa9a18e1 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:30:57.941 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:57.936+0000 7ffb0f7fe700 1 -- 172.21.15.110:0/3655772872 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb1000e070 con 0x7ffb180b0f70 2023-10-06T03:30:57.941 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:57.936+0000 7ffb0f7fe700 1 -- 172.21.15.110:0/3655772872 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb1000ba50 con 0x7ffb180b0f70 2023-10-06T03:30:57.941 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:57.936+0000 7ffb0f7fe700 1 -- 172.21.15.110:0/3655772872 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb100092e0 con 0x7ffb180b0f70 2023-10-06T03:30:57.942 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:57.936+0000 7ffb1dabd700 1 -- 172.21.15.110:0/3655772872 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ffb180b0f70 msgr2=0x7ffb18155dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:57.942 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:57.936+0000 7ffb1dabd700 1 --2- 172.21.15.110:0/3655772872 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ffb180b0f70 0x7ffb18155dd0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7ffb18069a80 tx=0x7ffb10009840 comp rx=0 tx=0).stop 2023-10-06T03:30:57.942 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:57.936+0000 7ffb1dabd700 1 -- 172.21.15.110:0/3655772872 shutdown_connections 2023-10-06T03:30:57.942 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:57.936+0000 7ffb1dabd700 1 --2- 172.21.15.110:0/3655772872 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ffb180b0f70 0x7ffb18155dd0 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:57.943 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:57.936+0000 7ffb1dabd700 1 --2- 172.21.15.110:0/3655772872 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ffb180cb910 0x7ffb180b0a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:57.943 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:57.936+0000 7ffb1dabd700 1 --2- 172.21.15.110:0/3655772872 >> v2:172.21.15.192:3300/0 conn(0x7ffb180adc20 0x7ffb180b13f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:57.943 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:57.936+0000 7ffb1dabd700 1 -- 172.21.15.110:0/3655772872 >> 172.21.15.110:0/3655772872 conn(0x7ffb1806c760 msgr2=0x7ffb1806cb60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:30:57.943 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:57.936+0000 7ffb1dabd700 1 -- 172.21.15.110:0/3655772872 shutdown_connections 2023-10-06T03:30:57.943 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:57.936+0000 7ffb1dabd700 1 -- 172.21.15.110:0/3655772872 wait complete. 2023-10-06T03:30:57.944 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:57.940+0000 7ffb1dabd700 1 Processor -- start 2023-10-06T03:30:57.944 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:57.940+0000 7ffb1dabd700 1 -- start start 2023-10-06T03:30:57.947 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:57.940+0000 7ffb1dabd700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ffb180adc20 0x7ffb180c1880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:57.947 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:57.940+0000 7ffb1dabd700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ffb180b0f70 0x7ffb180c1dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:57.947 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:57.940+0000 7ffb1dabd700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ffb180cb910 0x7ffb180c2de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:57.947 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:57.940+0000 7ffb1dabd700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb18152270 con 0x7ffb180b0f70 2023-10-06T03:30:57.947 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:57.940+0000 7ffb1dabd700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb180c13f0 con 0x7ffb180cb910 2023-10-06T03:30:57.947 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:57.940+0000 7ffb0ffff700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ffb180b0f70 0x7ffb180c1dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:57.948 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:57.940+0000 7ffb0ffff700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ffb180b0f70 0x7ffb180c1dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:51104/0 (socket says 172.21.15.110:51104) 2023-10-06T03:30:57.948 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:57.940+0000 7ffb0ffff700 1 -- 172.21.15.110:0/3201302781 learned_addr learned my addr 172.21.15.110:0/3201302781 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:30:57.948 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:57.940+0000 7ffb1dabd700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb180c07e0 con 0x7ffb180adc20 2023-10-06T03:30:57.948 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:57.940+0000 7ffb1cabb700 1 --2- 172.21.15.110:0/3201302781 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ffb180adc20 0x7ffb180c1880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:57.948 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:57.940+0000 7ffb1d2bc700 1 --2- 172.21.15.110:0/3201302781 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ffb180cb910 0x7ffb180c2de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:57.948 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:57.940+0000 7ffb0ffff700 1 -- 172.21.15.110:0/3201302781 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ffb180adc20 msgr2=0x7ffb180c1880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:57.948 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:57.940+0000 7ffb0ffff700 1 --2- 172.21.15.110:0/3201302781 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ffb180adc20 0x7ffb180c1880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:57.948 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:57.940+0000 7ffb0ffff700 1 -- 172.21.15.110:0/3201302781 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ffb180cb910 msgr2=0x7ffb180c2de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:57.948 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:57.940+0000 7ffb0ffff700 1 --2- 172.21.15.110:0/3201302781 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ffb180cb910 0x7ffb180c2de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:57.948 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:57.940+0000 7ffb0ffff700 1 -- 172.21.15.110:0/3201302781 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb10009ae0 con 0x7ffb180b0f70 2023-10-06T03:30:57.948 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:57.940+0000 7ffb1cabb700 1 --2- 172.21.15.110:0/3201302781 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ffb180adc20 0x7ffb180c1880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-06T03:30:57.948 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:57.940+0000 7ffb1d2bc700 1 --2- 172.21.15.110:0/3201302781 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ffb180cb910 0x7ffb180c2de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-06T03:30:57.949 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:57.940+0000 7ffb0ffff700 1 --2- 172.21.15.110:0/3201302781 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ffb180b0f70 0x7ffb180c1dc0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7ffb0800d860 tx=0x7ffb08008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:30:57.949 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:57.944+0000 7ffb0cff9700 1 -- 172.21.15.110:0/3201302781 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb080125f0 con 0x7ffb180b0f70 2023-10-06T03:30:57.949 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:57.944+0000 7ffb0cff9700 1 -- 172.21.15.110:0/3201302781 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb08012750 con 0x7ffb180b0f70 2023-10-06T03:30:57.950 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:57.944+0000 7ffb0cff9700 1 -- 172.21.15.110:0/3201302781 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb08012ad0 con 0x7ffb180b0f70 2023-10-06T03:30:57.950 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:57.944+0000 7ffb1dabd700 1 -- 172.21.15.110:0/3201302781 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb180c0ac0 con 0x7ffb180b0f70 2023-10-06T03:30:57.950 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:57.944+0000 7ffb1dabd700 1 -- 172.21.15.110:0/3201302781 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb180c1080 con 0x7ffb180b0f70 2023-10-06T03:30:57.950 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:57.944+0000 7ffb1dabd700 1 -- 172.21.15.110:0/3201302781 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb180c1580 con 0x7ffb180b0f70 2023-10-06T03:30:57.950 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:57.944+0000 7ffb0cff9700 1 -- 172.21.15.110:0/3201302781 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 9) v1 ==== 80135+0+0 (secure 0 0 0) 0x7ffb08014430 con 0x7ffb180b0f70 2023-10-06T03:30:57.952 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:57.948+0000 7ffb0cff9700 1 --2- 172.21.15.110:0/3201302781 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7ffb04060750 0x7ffb04062c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:57.952 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:57.948+0000 7ffb0cff9700 1 -- 172.21.15.110:0/3201302781 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 6636+0+0 (secure 0 0 0) 0x7ffb0807d3b0 con 0x7ffb180b0f70 2023-10-06T03:30:57.952 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:57.948+0000 7ffb1cabb700 1 --2- 172.21.15.110:0/3201302781 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7ffb04060750 0x7ffb04062c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:57.952 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:57.948+0000 7ffb1cabb700 1 --2- 172.21.15.110:0/3201302781 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7ffb04060750 0x7ffb04062c00 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7ffb14006fd0 tx=0x7ffb14008040 comp rx=0 tx=0).ready entity=mgr.5329 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:30:57.959 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:57.952+0000 7ffb0cff9700 1 -- 172.21.15.110:0/3201302781 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7ffb08050e10 con 0x7ffb180b0f70 2023-10-06T03:30:58.163 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:58.160+0000 7ffb1dabd700 1 -- 172.21.15.110:0/3201302781 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "mgr module disable", "module": "snap_schedule"} v 0) v1 -- 0x7ffb18071020 con 0x7ffb180b0f70 2023-10-06T03:30:58.373 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:58.368+0000 7ffb0cff9700 1 -- 172.21.15.110:0/3201302781 <== mon.0 v2:172.21.15.110:3300/0 7 ==== mgrmap(e 10) v1 ==== 80118+0+0 (secure 0 0 0) 0x7ffb0804d750 con 0x7ffb180b0f70 2023-10-06T03:30:58.374 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:58.368+0000 7ffb0cff9700 1 -- 172.21.15.110:0/3201302781 <== mon.0 v2:172.21.15.110:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module disable", "module": "snap_schedule"}]=0 v10) v1 ==== 93+0+0 (secure 0 0 0) 0x7ffb08049b20 con 0x7ffb180b0f70 2023-10-06T03:30:58.377 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:58.372+0000 7ffb1dabd700 1 -- 172.21.15.110:0/3201302781 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7ffb04060750 msgr2=0x7ffb04062c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:58.377 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:58.372+0000 7ffb1dabd700 1 --2- 172.21.15.110:0/3201302781 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7ffb04060750 0x7ffb04062c00 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7ffb14006fd0 tx=0x7ffb14008040 comp rx=0 tx=0).stop 2023-10-06T03:30:58.377 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:58.372+0000 7ffb1dabd700 1 -- 172.21.15.110:0/3201302781 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ffb180b0f70 msgr2=0x7ffb180c1dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:58.377 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:58.372+0000 7ffb1dabd700 1 --2- 172.21.15.110:0/3201302781 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ffb180b0f70 0x7ffb180c1dc0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7ffb0800d860 tx=0x7ffb08008040 comp rx=0 tx=0).stop 2023-10-06T03:30:58.377 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:58.372+0000 7ffb1dabd700 1 -- 172.21.15.110:0/3201302781 shutdown_connections 2023-10-06T03:30:58.377 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:58.372+0000 7ffb1dabd700 1 --2- 172.21.15.110:0/3201302781 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7ffb04060750 0x7ffb04062c00 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:58.377 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:58.372+0000 7ffb1dabd700 1 --2- 172.21.15.110:0/3201302781 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ffb180cb910 0x7ffb180c2de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:58.377 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:58.372+0000 7ffb1dabd700 1 --2- 172.21.15.110:0/3201302781 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ffb180b0f70 0x7ffb180c1dc0 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:58.377 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:58.372+0000 7ffb1dabd700 1 --2- 172.21.15.110:0/3201302781 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ffb180adc20 0x7ffb180c1880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:58.377 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:58.372+0000 7ffb1dabd700 1 -- 172.21.15.110:0/3201302781 >> 172.21.15.110:0/3201302781 conn(0x7ffb1806c760 msgr2=0x7ffb18156b50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:30:58.378 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:58.372+0000 7ffb1dabd700 1 -- 172.21.15.110:0/3201302781 shutdown_connections 2023-10-06T03:30:58.378 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:58.372+0000 7ffb1dabd700 1 -- 172.21.15.110:0/3201302781 wait complete. 2023-10-06T03:30:58.390 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-10-06T03:30:58.390 DEBUG:teuthology.orchestra.run.smithi110:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-06T03:30:58.403 DEBUG:teuthology.orchestra.run.smithi192:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-06T03:30:58.421 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-10-06T03:30:58.421 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi110.front.sepia.ceph.com... 2023-10-06T03:30:58.421 INFO:teuthology.orchestra.run:Running command with timeout 1800 2023-10-06T03:30:58.421 DEBUG:teuthology.orchestra.run.smithi110:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2023-10-06T03:30:58.433 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi110.stderr:ceph-fuse[42767]: fuse finished with error 0 and tester_r 0 2023-10-06T03:30:58.434 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi110.front.sepia.ceph.com 2023-10-06T03:30:58.434 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-06T03:30:58.434 DEBUG:teuthology.orchestra.run.smithi110:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2023-10-06T03:30:58.486 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi192.front.sepia.ceph.com 2023-10-06T03:30:58.487 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-06T03:30:58.487 DEBUG:teuthology.orchestra.run.smithi192:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2023-10-06T03:30:58.489 INFO:tasks.ceph.mgr.x.smithi192.stderr:2023-10-06T03:30:58.484+0000 7f76034c8000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-06T03:30:58.493 INFO:teuthology.orchestra.run.smithi192.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.1': No such file or directory 2023-10-06T03:30:58.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-06T03:30:58.494 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi110.front.sepia.ceph.com 2023-10-06T03:30:58.494 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-06T03:30:58.494 DEBUG:teuthology.orchestra.run.smithi110:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2023-10-06T03:30:58.495 INFO:tasks.ceph.mgr.y.smithi110.stderr:2023-10-06T03:30:58.488+0000 7ff496339000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-06T03:30:58.500 INFO:teuthology.orchestra.run.smithi110.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2023-10-06T03:30:58.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-06T03:30:58.502 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi192.front.sepia.ceph.com 2023-10-06T03:30:58.502 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-06T03:30:58.502 DEBUG:teuthology.orchestra.run.smithi192:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2023-10-06T03:30:58.508 INFO:teuthology.orchestra.run.smithi192.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.3': No such file or directory 2023-10-06T03:30:58.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-06T03:30:58.512 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-06T03:30:58.620 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:58.616+0000 7f521a630700 1 Processor -- start 2023-10-06T03:30:58.620 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:58.616+0000 7f521a630700 1 -- start start 2023-10-06T03:30:58.622 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:58.616+0000 7f521a630700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f52141568b0 0x7f5214156c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:58.622 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:58.616+0000 7f521a630700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5214067ef0 con 0x7f52140ca850 2023-10-06T03:30:58.622 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:58.616+0000 7f521a630700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52140cac80 con 0x7f52140b12b0 2023-10-06T03:30:58.622 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:58.616+0000 7f521a630700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5214067040 con 0x7f52141568b0 2023-10-06T03:30:58.622 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:58.616+0000 7f5218e2d700 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f52140ca850 0x7f52140ae300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:40050/0 (socket says 172.21.15.110:40050) 2023-10-06T03:30:58.622 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:58.616+0000 7f5218e2d700 1 -- 172.21.15.110:0/2800673336 learned_addr learned my addr 172.21.15.110:0/2800673336 (peer_addr_for_me v1:172.21.15.110:0/0) 2023-10-06T03:30:58.622 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:58.616+0000 7f521962e700 1 --2- 172.21.15.110:0/2800673336 >> v2:172.21.15.192:3300/0 conn(0x7f52141568b0 0x7f5214156c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:58.623 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:58.616+0000 7f520bfff700 1 -- 172.21.15.110:0/2800673336 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3381637136 0 0) 0x7f5214067ef0 con 0x7f52140ca850 2023-10-06T03:30:58.623 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:58.616+0000 7f520bfff700 1 -- 172.21.15.110:0/2800673336 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5200003570 con 0x7f52140ca850 2023-10-06T03:30:58.623 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:58.616+0000 7f520bfff700 1 -- 172.21.15.110:0/2800673336 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 425520184 0 0) 0x7f52140cac80 con 0x7f52140b12b0 2023-10-06T03:30:58.623 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:58.616+0000 7f520bfff700 1 -- 172.21.15.110:0/2800673336 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5214067ef0 con 0x7f52140b12b0 2023-10-06T03:30:58.623 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:58.616+0000 7f521962e700 1 -- 172.21.15.110:0/2800673336 >> v1:172.21.15.192:6789/0 conn(0x7f52140b12b0 legacy=0x7f521406c9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:58.623 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:58.616+0000 7f521962e700 1 -- 172.21.15.110:0/2800673336 >> v1:172.21.15.110:6789/0 conn(0x7f52140ca850 legacy=0x7f52140ae300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:58.623 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:58.616+0000 7f520bfff700 1 -- 172.21.15.110:0/2800673336 <== mon.0 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2396260907 0 0) 0x7f5200003570 con 0x7f52140ca850 2023-10-06T03:30:58.623 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:58.616+0000 7f521962e700 1 -- 172.21.15.110:0/2800673336 --> v2:172.21.15.192:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52140672e0 con 0x7f52141568b0 2023-10-06T03:30:58.624 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:58.616+0000 7f521962e700 1 --2- 172.21.15.110:0/2800673336 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f52141568b0 0x7f5214156c80 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f52100098e0 tx=0x7f52100091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2e6d95657f20d2c3 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:30:58.625 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:58.616+0000 7f520bfff700 1 -- 172.21.15.110:0/2800673336 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5210011630 con 0x7f52141568b0 2023-10-06T03:30:58.625 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:58.616+0000 7f520bfff700 1 -- 172.21.15.110:0/2800673336 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5210011790 con 0x7f52141568b0 2023-10-06T03:30:58.625 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:58.616+0000 7f520bfff700 1 -- 172.21.15.110:0/2800673336 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5210011b30 con 0x7f52141568b0 2023-10-06T03:30:58.625 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:58.620+0000 7f521a630700 1 -- 172.21.15.110:0/2800673336 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f52141568b0 msgr2=0x7f5214156c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:58.625 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:58.620+0000 7f521a630700 1 --2- 172.21.15.110:0/2800673336 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f52141568b0 0x7f5214156c80 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f52100098e0 tx=0x7f52100091a0 comp rx=0 tx=0).stop 2023-10-06T03:30:58.625 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:58.620+0000 7f521a630700 1 -- 172.21.15.110:0/2800673336 shutdown_connections 2023-10-06T03:30:58.625 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:58.620+0000 7f521a630700 1 --2- 172.21.15.110:0/2800673336 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f52141568b0 0x7f5214156c80 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:58.625 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:58.620+0000 7f521a630700 1 -- 172.21.15.110:0/2800673336 >> 172.21.15.110:0/2800673336 conn(0x7f52140cccb0 msgr2=0x7f52140adbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:30:58.625 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:58.620+0000 7f521a630700 1 -- 172.21.15.110:0/2800673336 shutdown_connections 2023-10-06T03:30:58.626 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:58.620+0000 7f521a630700 1 -- 172.21.15.110:0/2800673336 wait complete. 2023-10-06T03:30:58.626 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:58.620+0000 7f521a630700 1 Processor -- start 2023-10-06T03:30:58.626 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:58.620+0000 7f521a630700 1 -- start start 2023-10-06T03:30:58.628 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:58.620+0000 7f521a630700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f52140b12b0 0x7f52140c7e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:58.628 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:58.620+0000 7f521a630700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f52140ca850 0x7f5214153150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:58.628 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:58.620+0000 7f521a630700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f52141568b0 0x7f5214153690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:58.628 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:58.620+0000 7f521a630700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52141673b0 con 0x7f52140b12b0 2023-10-06T03:30:58.629 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:58.624+0000 7f521a630700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f521413ed00 con 0x7f52141568b0 2023-10-06T03:30:58.629 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:58.624+0000 7f521a630700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f521413e9d0 con 0x7f52140ca850 2023-10-06T03:30:58.629 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:58.624+0000 7f521962e700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f52140b12b0 0x7f52140c7e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:58.629 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:58.624+0000 7f521962e700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f52140b12b0 0x7f52140c7e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:51142/0 (socket says 172.21.15.110:51142) 2023-10-06T03:30:58.629 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:58.624+0000 7f521962e700 1 -- 172.21.15.110:0/2313352545 learned_addr learned my addr 172.21.15.110:0/2313352545 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:30:58.629 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:58.624+0000 7f5218e2d700 1 --2- 172.21.15.110:0/2313352545 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f52140ca850 0x7f5214153150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:58.629 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:58.624+0000 7f5219e2f700 1 --2- 172.21.15.110:0/2313352545 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f52141568b0 0x7f5214153690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:58.630 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:58.624+0000 7f521962e700 1 -- 172.21.15.110:0/2313352545 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f52140ca850 msgr2=0x7f5214153150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:58.630 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:58.624+0000 7f521962e700 1 --2- 172.21.15.110:0/2313352545 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f52140ca850 0x7f5214153150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:58.630 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:58.624+0000 7f521962e700 1 -- 172.21.15.110:0/2313352545 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f52141568b0 msgr2=0x7f5214153690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:58.630 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:58.624+0000 7f521962e700 1 --2- 172.21.15.110:0/2313352545 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f52141568b0 0x7f5214153690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:58.630 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:58.624+0000 7f521962e700 1 -- 172.21.15.110:0/2313352545 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5210009420 con 0x7f52140b12b0 2023-10-06T03:30:58.630 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:58.624+0000 7f521962e700 1 --2- 172.21.15.110:0/2313352545 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f52140b12b0 0x7f52140c7e80 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f5210009ed0 tx=0x7f5210017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:30:58.632 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:58.624+0000 7f52097fa700 1 -- 172.21.15.110:0/2313352545 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f521001f070 con 0x7f52140b12b0 2023-10-06T03:30:58.632 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:58.624+0000 7f52097fa700 1 -- 172.21.15.110:0/2313352545 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5210000bc0 con 0x7f52140b12b0 2023-10-06T03:30:58.632 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:58.624+0000 7f521a630700 1 -- 172.21.15.110:0/2313352545 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52140c8720 con 0x7f52140b12b0 2023-10-06T03:30:58.632 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:58.624+0000 7f52097fa700 1 -- 172.21.15.110:0/2313352545 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5210025420 con 0x7f52140b12b0 2023-10-06T03:30:58.632 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:58.624+0000 7f521a630700 1 -- 172.21.15.110:0/2313352545 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52140c8ce0 con 0x7f52140b12b0 2023-10-06T03:30:58.632 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:58.624+0000 7f521a630700 1 -- 172.21.15.110:0/2313352545 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5214069890 con 0x7f52140b12b0 2023-10-06T03:30:58.632 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:58.624+0000 7f52097fa700 1 -- 172.21.15.110:0/2313352545 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 10) v1 ==== 80118+0+0 (secure 0 0 0) 0x7f5210023070 con 0x7f52140b12b0 2023-10-06T03:30:58.633 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:58.628+0000 7f52097fa700 1 --2- 172.21.15.110:0/2313352545 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f51f8060700 0x7f51f8062bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:58.633 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:58.628+0000 7f5218e2d700 1 -- 172.21.15.110:0/2313352545 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f51f8060700 msgr2=0x7f51f8062bb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.110:6832/33268 2023-10-06T03:30:58.633 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:58.628+0000 7f52097fa700 1 -- 172.21.15.110:0/2313352545 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 6636+0+0 (secure 0 0 0) 0x7f521007e040 con 0x7f52140b12b0 2023-10-06T03:30:58.633 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:58.628+0000 7f5218e2d700 1 --2- 172.21.15.110:0/2313352545 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f51f8060700 0x7f51f8062bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-06T03:30:58.638 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:58.632+0000 7f52097fa700 1 -- 172.21.15.110:0/2313352545 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f521004e010 con 0x7f52140b12b0 2023-10-06T03:30:58.835 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:58.832+0000 7f5218e2d700 1 -- 172.21.15.110:0/2313352545 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f51f8060700 msgr2=0x7f51f8062bb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.110:6832/33268 2023-10-06T03:30:58.836 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:58.832+0000 7f5218e2d700 1 --2- 172.21.15.110:0/2313352545 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f51f8060700 0x7f51f8062bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-06T03:30:58.852 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:58.848+0000 7f521a630700 1 -- 172.21.15.110:0/2313352545 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f52140675c0 con 0x7f52140b12b0 2023-10-06T03:30:58.853 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:58.848+0000 7f52097fa700 1 -- 172.21.15.110:0/2313352545 <== mon.0 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) v1 ==== 94+0+4603 (secure 0 0 0) 0x7f5210015090 con 0x7f52140b12b0 2023-10-06T03:30:58.853 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-10-06T03:30:58.853 INFO:teuthology.orchestra.run.smithi110.stdout:{"epoch":16,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5107,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6834/884349347","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6833","nonce":884349347},{"type":"v1","addr":"172.21.15.192:6834","nonce":884349347}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":5110,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6835/4077204152","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6832","nonce":4077204152},{"type":"v1","addr":"172.21.15.192:6835","nonce":4077204152}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":5160,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/504583843","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6834","nonce":504583843},{"type":"v1","addr":"172.21.15.110:6835","nonce":504583843}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-06T03:30:34.014644+0000","modified":"2023-10-06T03:30:37.662700+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5163},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5163":{"gid":5163,"name":"a","rank":0,"incarnation":13,"state":"up:active","state_seq":4,"addr":"172.21.15.110:6837/1379510962","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6836","nonce":1379510962},{"type":"v1","addr":"172.21.15.110:6837","nonce":1379510962}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":2}]} 2023-10-06T03:30:58.855 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:58.848+0000 7f521a630700 1 -- 172.21.15.110:0/2313352545 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f51f8060700 msgr2=0x7f51f8062bb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-06T03:30:58.855 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:58.848+0000 7f521a630700 1 --2- 172.21.15.110:0/2313352545 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f51f8060700 0x7f51f8062bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:58.856 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:58.848+0000 7f521a630700 1 -- 172.21.15.110:0/2313352545 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f52140b12b0 msgr2=0x7f52140c7e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:58.856 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:58.848+0000 7f521a630700 1 --2- 172.21.15.110:0/2313352545 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f52140b12b0 0x7f52140c7e80 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f5210009ed0 tx=0x7f5210017000 comp rx=0 tx=0).stop 2023-10-06T03:30:58.856 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:58.848+0000 7f521a630700 1 -- 172.21.15.110:0/2313352545 shutdown_connections 2023-10-06T03:30:58.856 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:58.848+0000 7f521a630700 1 --2- 172.21.15.110:0/2313352545 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f51f8060700 0x7f51f8062bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:58.856 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:58.848+0000 7f521a630700 1 --2- 172.21.15.110:0/2313352545 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f52141568b0 0x7f5214153690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:58.856 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:58.848+0000 7f521a630700 1 --2- 172.21.15.110:0/2313352545 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f52140ca850 0x7f5214153150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:58.856 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:58.848+0000 7f521a630700 1 --2- 172.21.15.110:0/2313352545 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f52140b12b0 0x7f52140c7e80 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:58.856 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:58.848+0000 7f521a630700 1 -- 172.21.15.110:0/2313352545 >> 172.21.15.110:0/2313352545 conn(0x7f52140cccb0 msgr2=0x7f5214155940 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:30:58.856 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:58.852+0000 7f521a630700 1 -- 172.21.15.110:0/2313352545 shutdown_connections 2023-10-06T03:30:58.856 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:58.852+0000 7f521a630700 1 -- 172.21.15.110:0/2313352545 wait complete. 2023-10-06T03:30:58.857 INFO:teuthology.orchestra.run.smithi110.stderr:dumped fsmap epoch 16 2023-10-06T03:30:58.867 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-06T03:30:58.917 INFO:tasks.ceph.mgr.x.smithi192.stderr:2023-10-06T03:30:58.912+0000 7f76034c8000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-06T03:30:58.965 INFO:tasks.ceph.mgr.x.smithi192.stderr:2023-10-06T03:30:58.960+0000 7f76034c8000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-06T03:30:58.968 INFO:tasks.ceph.mgr.y.smithi110.stderr:2023-10-06T03:30:58.964+0000 7ff496339000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-06T03:30:58.971 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:58.964+0000 7f122359e700 1 Processor -- start 2023-10-06T03:30:58.971 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:58.964+0000 7f122359e700 1 -- start start 2023-10-06T03:30:58.972 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:58.968+0000 7f122359e700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f1224069610 0x7f1224155b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:58.972 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:58.968+0000 7f122359e700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1224067500 0x7f1224067920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:58.972 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:58.968+0000 7f122359e700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1224067e60 con 0x7f1224155750 2023-10-06T03:30:58.973 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:58.968+0000 7f122359e700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f12240c96e0 con 0x7f1224069610 2023-10-06T03:30:58.973 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:58.968+0000 7f122359e700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12240c99c0 con 0x7f1224067500 2023-10-06T03:30:58.973 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:58.968+0000 7f1221d9b700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f1224069610 0x7f1224155b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:58.973 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:58.968+0000 7f1222d9d700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1224067500 0x7f1224067920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:58.973 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:58.968+0000 7f1221d9b700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f1224069610 0x7f1224155b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:51150/0 (socket says 172.21.15.110:51150) 2023-10-06T03:30:58.973 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:58.968+0000 7f1221d9b700 1 -- 172.21.15.110:0/743310953 learned_addr learned my addr 172.21.15.110:0/743310953 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:30:58.973 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:58.968+0000 7f122259c700 1 --1- 172.21.15.110:0/743310953 >> v1:172.21.15.192:6789/0 conn(0x7f1224155750 0x7f12240c8ea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.110:42934/0 (socket says 172.21.15.110:42934) 2023-10-06T03:30:58.975 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:58.968+0000 7f1221d9b700 1 -- 172.21.15.110:0/743310953 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1224067500 msgr2=0x7f1224067920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:30:58.975 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:58.968+0000 7f1221d9b700 1 --2- 172.21.15.110:0/743310953 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1224067500 0x7f1224067920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:58.975 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:58.968+0000 7f1221d9b700 1 -- 172.21.15.110:0/743310953 >> v1:172.21.15.192:6789/0 conn(0x7f1224155750 legacy=0x7f12240c8ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:58.975 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:58.968+0000 7f1221d9b700 1 -- 172.21.15.110:0/743310953 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12240c9ca0 con 0x7f1224069610 2023-10-06T03:30:58.975 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:58.968+0000 7f1221d9b700 1 --2- 172.21.15.110:0/743310953 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1224069610 0x7f1224155b20 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f12180099b0 tx=0x7f1218009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=78fcefe767e293c4 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:30:58.976 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:58.968+0000 7f122159a700 1 -- 172.21.15.110:0/743310953 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1218009ea0 con 0x7f1224069610 2023-10-06T03:30:58.976 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:58.968+0000 7f122159a700 1 -- 172.21.15.110:0/743310953 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f121800b860 con 0x7f1224069610 2023-10-06T03:30:58.976 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:58.968+0000 7f122159a700 1 -- 172.21.15.110:0/743310953 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f121800bc00 con 0x7f1224069610 2023-10-06T03:30:58.976 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:58.968+0000 7f122359e700 1 -- 172.21.15.110:0/743310953 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1224069610 msgr2=0x7f1224155b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:58.976 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:58.968+0000 7f122359e700 1 --2- 172.21.15.110:0/743310953 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1224069610 0x7f1224155b20 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f12180099b0 tx=0x7f1218009250 comp rx=0 tx=0).stop 2023-10-06T03:30:58.976 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:58.968+0000 7f122359e700 1 -- 172.21.15.110:0/743310953 shutdown_connections 2023-10-06T03:30:58.976 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:58.968+0000 7f122359e700 1 --2- 172.21.15.110:0/743310953 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1224067500 0x7f1224067920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:58.976 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:58.968+0000 7f122359e700 1 --2- 172.21.15.110:0/743310953 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1224069610 0x7f1224155b20 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:58.976 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:58.968+0000 7f122359e700 1 -- 172.21.15.110:0/743310953 >> 172.21.15.110:0/743310953 conn(0x7f122406b4c0 msgr2=0x7f122406b8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:30:58.976 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:58.968+0000 7f122359e700 1 -- 172.21.15.110:0/743310953 shutdown_connections 2023-10-06T03:30:58.976 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:58.968+0000 7f122359e700 1 -- 172.21.15.110:0/743310953 wait complete. 2023-10-06T03:30:58.977 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:58.972+0000 7f122359e700 1 Processor -- start 2023-10-06T03:30:58.977 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:58.972+0000 7f122359e700 1 -- start start 2023-10-06T03:30:58.979 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:58.972+0000 7f122359e700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1224067500 0x7f122413e800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:58.979 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:58.972+0000 7f122359e700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1224069610 0x7f122406b890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:58.979 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:58.972+0000 7f122359e700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1224155750 0x7f122406bdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:58.979 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:58.972+0000 7f122359e700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f122413f110 con 0x7f1224155750 2023-10-06T03:30:58.979 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:58.972+0000 7f122359e700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12240c6870 con 0x7f1224067500 2023-10-06T03:30:58.979 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:58.972+0000 7f122359e700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12240c6b50 con 0x7f1224069610 2023-10-06T03:30:58.979 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:58.972+0000 7f1222d9d700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1224155750 0x7f122406bdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:58.979 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:58.972+0000 7f122259c700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1224067500 0x7f122413e800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:58.979 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:58.972+0000 7f1222d9d700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1224155750 0x7f122406bdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:51166/0 (socket says 172.21.15.110:51166) 2023-10-06T03:30:58.980 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:58.972+0000 7f1222d9d700 1 -- 172.21.15.110:0/3979897755 learned_addr learned my addr 172.21.15.110:0/3979897755 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:30:58.980 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:58.972+0000 7f1221d9b700 1 --2- 172.21.15.110:0/3979897755 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1224069610 0x7f122406b890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:58.980 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:58.972+0000 7f1222d9d700 1 -- 172.21.15.110:0/3979897755 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1224069610 msgr2=0x7f122406b890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:58.980 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:58.972+0000 7f1222d9d700 1 --2- 172.21.15.110:0/3979897755 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1224069610 0x7f122406b890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:58.980 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:58.972+0000 7f1222d9d700 1 -- 172.21.15.110:0/3979897755 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1224067500 msgr2=0x7f122413e800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:58.980 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:58.972+0000 7f1222d9d700 1 --2- 172.21.15.110:0/3979897755 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1224067500 0x7f122413e800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:58.980 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:58.972+0000 7f1222d9d700 1 -- 172.21.15.110:0/3979897755 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12180094f0 con 0x7f1224155750 2023-10-06T03:30:58.980 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:58.972+0000 7f1222d9d700 1 --2- 172.21.15.110:0/3979897755 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1224155750 0x7f122406bdd0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f121c009e90 tx=0x7f121c00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:30:58.981 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:58.972+0000 7f12127fc700 1 -- 172.21.15.110:0/3979897755 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f121c00cb70 con 0x7f1224155750 2023-10-06T03:30:58.981 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:58.976+0000 7f12127fc700 1 -- 172.21.15.110:0/3979897755 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f121c00f040 con 0x7f1224155750 2023-10-06T03:30:58.981 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:58.976+0000 7f122359e700 1 -- 172.21.15.110:0/3979897755 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f122413f360 con 0x7f1224155750 2023-10-06T03:30:58.981 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:58.976+0000 7f12127fc700 1 -- 172.21.15.110:0/3979897755 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f121c014480 con 0x7f1224155750 2023-10-06T03:30:58.981 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:58.976+0000 7f122359e700 1 -- 172.21.15.110:0/3979897755 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f122413ed40 con 0x7f1224155750 2023-10-06T03:30:58.982 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:58.976+0000 7f12127fc700 1 -- 172.21.15.110:0/3979897755 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 10) v1 ==== 80118+0+0 (secure 0 0 0) 0x7f121c014830 con 0x7f1224155750 2023-10-06T03:30:58.982 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:58.976+0000 7f122359e700 1 -- 172.21.15.110:0/3979897755 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f122406a9e0 con 0x7f1224155750 2023-10-06T03:30:58.983 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:58.976+0000 7f12127fc700 1 --2- 172.21.15.110:0/3979897755 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f1204060700 0x7f1204062bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:58.983 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:58.976+0000 7f122259c700 1 -- 172.21.15.110:0/3979897755 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f1204060700 msgr2=0x7f1204062bb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.110:6832/33268 2023-10-06T03:30:58.983 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:58.976+0000 7f12127fc700 1 -- 172.21.15.110:0/3979897755 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 6636+0+0 (secure 0 0 0) 0x7f121c07d8f0 con 0x7f1224155750 2023-10-06T03:30:58.983 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:58.976+0000 7f122259c700 1 --2- 172.21.15.110:0/3979897755 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f1204060700 0x7f1204062bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-06T03:30:58.991 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:58.984+0000 7f12127fc700 1 -- 172.21.15.110:0/3979897755 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f121c04d7b0 con 0x7f1224155750 2023-10-06T03:30:59.027 INFO:tasks.ceph.mgr.y.smithi110.stderr:2023-10-06T03:30:59.024+0000 7ff496339000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-06T03:30:59.070 INFO:tasks.ceph.mgr.x.smithi192.stderr:2023-10-06T03:30:59.068+0000 7f76034c8000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-06T03:30:59.135 INFO:tasks.ceph.mgr.y.smithi110.stderr:2023-10-06T03:30:59.132+0000 7ff496339000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-06T03:30:59.152 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:59.148+0000 7f122359e700 1 -- 172.21.15.110:0/3979897755 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f12240ca790 con 0x7f1224155750 2023-10-06T03:30:59.154 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:59.148+0000 7f12127fc700 1 -- 172.21.15.110:0/3979897755 <== mon.0 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v43) v1 ==== 74+0+17491 (secure 0 0 0) 0x7f121c050dd0 con 0x7f1224155750 2023-10-06T03:30:59.154 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-10-06T03:30:59.154 INFO:teuthology.orchestra.run.smithi110.stdout:{"epoch":43,"fsid":"93b8f6ac-4c92-4b45-b8e4-b59617024aa2","created":"2023-10-06T03:29:10.519583+0000","modified":"2023-10-06T03:30:46.697260+0000","last_up_change":"2023-10-06T03:29:21.960511+0000","last_in_change":"2023-10-06T03:29:12.560436+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-06T03:29:20.703743+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2023-10-06T03:29:28.307281+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2023-10-06T03:30:31.807897+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"41","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":7,"pool_name":"cephfs_data","create_time":"2023-10-06T03:30:32.991334+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"42","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"5866eb6c-2adc-46e4-be7a-e5bcda0f4065","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6824","nonce":33394},{"type":"v1","addr":"172.21.15.110:6825","nonce":33394}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6826","nonce":33394},{"type":"v1","addr":"172.21.15.110:6827","nonce":33394}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6830","nonce":33394},{"type":"v1","addr":"172.21.15.110:6831","nonce":33394}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6828","nonce":33394},{"type":"v1","addr":"172.21.15.110:6829","nonce":33394}]},"public_addr":"172.21.15.110:6825/33394","cluster_addr":"172.21.15.110:6827/33394","heartbeat_back_addr":"172.21.15.110:6831/33394","heartbeat_front_addr":"172.21.15.110:6829/33394","state":["exists","up"]},{"osd":1,"uuid":"a38de118-062b-4a05-9782-2ceddf6ff4df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6800","nonce":33396},{"type":"v1","addr":"172.21.15.110:6801","nonce":33396}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6802","nonce":33396},{"type":"v1","addr":"172.21.15.110:6803","nonce":33396}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6806","nonce":33396},{"type":"v1","addr":"172.21.15.110:6807","nonce":33396}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6804","nonce":33396},{"type":"v1","addr":"172.21.15.110:6805","nonce":33396}]},"public_addr":"172.21.15.110:6801/33396","cluster_addr":"172.21.15.110:6803/33396","heartbeat_back_addr":"172.21.15.110:6807/33396","heartbeat_front_addr":"172.21.15.110:6805/33396","state":["exists","up"]},{"osd":2,"uuid":"56e39be2-3407-4b69-877e-cf16617b06d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6816","nonce":33397},{"type":"v1","addr":"172.21.15.110:6817","nonce":33397}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6818","nonce":33397},{"type":"v1","addr":"172.21.15.110:6819","nonce":33397}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6822","nonce":33397},{"type":"v1","addr":"172.21.15.110:6823","nonce":33397}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6820","nonce":33397},{"type":"v1","addr":"172.21.15.110:6821","nonce":33397}]},"public_addr":"172.21.15.110:6817/33397","cluster_addr":"172.21.15.110:6819/33397","heartbeat_back_addr":"172.21.15.110:6823/33397","heartbeat_front_addr":"172.21.15.110:6821/33397","state":["exists","up"]},{"osd":3,"uuid":"01c82d13-c1d4-4bde-b55e-3e0089b40115","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6808","nonce":33395},{"type":"v1","addr":"172.21.15.110:6809","nonce":33395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6810","nonce":33395},{"type":"v1","addr":"172.21.15.110:6811","nonce":33395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6814","nonce":33395},{"type":"v1","addr":"172.21.15.110:6815","nonce":33395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6812","nonce":33395},{"type":"v1","addr":"172.21.15.110:6813","nonce":33395}]},"public_addr":"172.21.15.110:6809/33395","cluster_addr":"172.21.15.110:6811/33395","heartbeat_back_addr":"172.21.15.110:6815/33395","heartbeat_front_addr":"172.21.15.110:6813/33395","state":["exists","up"]},{"osd":4,"uuid":"86983430-5fa3-4d17-998f-03e2cd843237","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":33683},{"type":"v1","addr":"172.21.15.192:6817","nonce":33683}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":33683},{"type":"v1","addr":"172.21.15.192:6819","nonce":33683}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":33683},{"type":"v1","addr":"172.21.15.192:6823","nonce":33683}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6820","nonce":33683},{"type":"v1","addr":"172.21.15.192:6821","nonce":33683}]},"public_addr":"172.21.15.192:6817/33683","cluster_addr":"172.21.15.192:6819/33683","heartbeat_back_addr":"172.21.15.192:6823/33683","heartbeat_front_addr":"172.21.15.192:6821/33683","state":["exists","up"]},{"osd":5,"uuid":"6945257f-821d-4619-a63f-ebe857eba6f4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":33684},{"type":"v1","addr":"172.21.15.192:6801","nonce":33684}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":33684},{"type":"v1","addr":"172.21.15.192:6803","nonce":33684}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":33684},{"type":"v1","addr":"172.21.15.192:6807","nonce":33684}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6804","nonce":33684},{"type":"v1","addr":"172.21.15.192:6805","nonce":33684}]},"public_addr":"172.21.15.192:6801/33684","cluster_addr":"172.21.15.192:6803/33684","heartbeat_back_addr":"172.21.15.192:6807/33684","heartbeat_front_addr":"172.21.15.192:6805/33684","state":["exists","up"]},{"osd":6,"uuid":"7854d1ae-7060-4780-8f97-73cea5e7a12f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6824","nonce":33686},{"type":"v1","addr":"172.21.15.192:6825","nonce":33686}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":33686},{"type":"v1","addr":"172.21.15.192:6827","nonce":33686}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":33686},{"type":"v1","addr":"172.21.15.192:6831","nonce":33686}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6828","nonce":33686},{"type":"v1","addr":"172.21.15.192:6829","nonce":33686}]},"public_addr":"172.21.15.192:6825/33686","cluster_addr":"172.21.15.192:6827/33686","heartbeat_back_addr":"172.21.15.192:6831/33686","heartbeat_front_addr":"172.21.15.192:6829/33686","state":["exists","up"]},{"osd":7,"uuid":"f1ebdfd5-b859-482b-9f97-c18c4a26f445","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6808","nonce":33685},{"type":"v1","addr":"172.21.15.192:6809","nonce":33685}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":33685},{"type":"v1","addr":"172.21.15.192:6811","nonce":33685}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":33685},{"type":"v1","addr":"172.21.15.192:6815","nonce":33685}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6812","nonce":33685},{"type":"v1","addr":"172.21.15.192:6813","nonce":33685}]},"public_addr":"172.21.15.192:6809/33685","cluster_addr":"172.21.15.192:6811/33685","heartbeat_back_addr":"172.21.15.192:6815/33685","heartbeat_front_addr":"172.21.15.192:6813/33685","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T03:29:18.002048+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T03:29:17.478576+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T03:29:18.034944+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T03:29:17.628513+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T03:29:18.047338+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T03:29:17.772349+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T03:29:17.883958+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T03:29:18.151838+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.110:6832/33268":"2023-10-07T03:30:46.697197+0000","172.21.15.110:0/463196587":"2023-10-07T03:30:46.697197+0000","172.21.15.110:6833/33268":"2023-10-07T03:30:46.697197+0000","172.21.15.110:0/1139019390":"2023-10-07T03:30:46.697197+0000","172.21.15.110:0/1318624357":"2023-10-07T03:30:46.697197+0000","172.21.15.110:0/1857995858":"2023-10-07T03:30:46.697197+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-06T03:30:59.157 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:59.152+0000 7f122359e700 1 -- 172.21.15.110:0/3979897755 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f1204060700 msgr2=0x7f1204062bb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-06T03:30:59.157 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:59.152+0000 7f122359e700 1 --2- 172.21.15.110:0/3979897755 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f1204060700 0x7f1204062bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:59.157 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:59.152+0000 7f122359e700 1 -- 172.21.15.110:0/3979897755 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1224155750 msgr2=0x7f122406bdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:59.157 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:59.152+0000 7f122359e700 1 --2- 172.21.15.110:0/3979897755 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1224155750 0x7f122406bdd0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f121c009e90 tx=0x7f121c00b040 comp rx=0 tx=0).stop 2023-10-06T03:30:59.157 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:59.152+0000 7f122359e700 1 -- 172.21.15.110:0/3979897755 shutdown_connections 2023-10-06T03:30:59.157 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:59.152+0000 7f122359e700 1 --2- 172.21.15.110:0/3979897755 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f1204060700 0x7f1204062bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:59.157 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:59.152+0000 7f122359e700 1 --2- 172.21.15.110:0/3979897755 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1224155750 0x7f122406bdd0 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:59.157 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:59.152+0000 7f122359e700 1 --2- 172.21.15.110:0/3979897755 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1224069610 0x7f122406b890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:59.158 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:59.152+0000 7f122359e700 1 --2- 172.21.15.110:0/3979897755 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1224067500 0x7f122413e800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:59.158 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:59.152+0000 7f122359e700 1 -- 172.21.15.110:0/3979897755 >> 172.21.15.110:0/3979897755 conn(0x7f122406b4c0 msgr2=0x7f12240c2fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:30:59.158 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:59.152+0000 7f122359e700 1 -- 172.21.15.110:0/3979897755 shutdown_connections 2023-10-06T03:30:59.158 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:59.152+0000 7f122359e700 1 -- 172.21.15.110:0/3979897755 wait complete. 2023-10-06T03:30:59.169 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-06T03:29:20.703743+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-10-06T03:29:28.307281+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 6, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-06T03:30:31.807897+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '41', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 7, 'pool_name': 'cephfs_data', 'create_time': '2023-10-06T03:30:32.991334+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '42', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-10-06T03:30:59.169 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-06T03:30:59.265 INFO:tasks.ceph.mgr.x.smithi192.stderr:2023-10-06T03:30:59.260+0000 7f76034c8000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-06T03:30:59.273 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:59.268+0000 7f35e8108700 1 Processor -- start 2023-10-06T03:30:59.273 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:59.268+0000 7f35e8108700 1 -- start start 2023-10-06T03:30:59.274 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:59.268+0000 7f35e8108700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f35e00cbae0 0x7f35e00ae860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:59.274 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:59.268+0000 7f35e8108700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35e00aee40 con 0x7f35e00cbeb0 2023-10-06T03:30:59.274 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:59.268+0000 7f35e8108700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35e00af120 con 0x7f35e0158bf0 2023-10-06T03:30:59.274 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:59.268+0000 7f35e8108700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35e00af400 con 0x7f35e00cbae0 2023-10-06T03:30:59.274 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:59.268+0000 7f35e5ea4700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f35e00cbae0 0x7f35e00ae860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:59.275 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:59.268+0000 7f35e56a3700 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f35e00cbeb0 0x7f35e00ad9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:40060/0 (socket says 172.21.15.110:40060) 2023-10-06T03:30:59.275 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:59.268+0000 7f35e56a3700 1 -- 172.21.15.110:0/381209216 learned_addr learned my addr 172.21.15.110:0/381209216 (peer_addr_for_me v1:172.21.15.110:0/0) 2023-10-06T03:30:59.275 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:59.268+0000 7f35e4ea2700 1 -- 172.21.15.110:0/381209216 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 217824831 0 0) 0x7f35e00aee40 con 0x7f35e00cbeb0 2023-10-06T03:30:59.275 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:59.268+0000 7f35e4ea2700 1 -- 172.21.15.110:0/381209216 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f35cc003570 con 0x7f35e00cbeb0 2023-10-06T03:30:59.277 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:59.268+0000 7f35e4ea2700 1 -- 172.21.15.110:0/381209216 <== mon.0 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2157157815 0 0) 0x7f35cc003570 con 0x7f35e00cbeb0 2023-10-06T03:30:59.277 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:59.268+0000 7f35e4ea2700 1 -- 172.21.15.110:0/381209216 >> v2:172.21.15.110:3300/0 conn(0x7f35e00cbae0 msgr2=0x7f35e00ae860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:30:59.277 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:59.268+0000 7f35e4ea2700 1 --2- 172.21.15.110:0/381209216 >> v2:172.21.15.110:3300/0 conn(0x7f35e00cbae0 0x7f35e00ae860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:59.277 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:59.268+0000 7f35e4ea2700 1 -- 172.21.15.110:0/381209216 >> v1:172.21.15.192:6789/0 conn(0x7f35e0158bf0 legacy=0x7f35e0159030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:59.277 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:59.268+0000 7f35e4ea2700 1 -- 172.21.15.110:0/381209216 --> v1:172.21.15.110:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35e00af6e0 con 0x7f35e00cbeb0 2023-10-06T03:30:59.277 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:59.268+0000 7f35e4ea2700 1 -- 172.21.15.110:0/381209216 <== mon.0 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 118603703 0 0) 0x7f35d0001470 con 0x7f35e00cbeb0 2023-10-06T03:30:59.278 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:59.268+0000 7f35e4ea2700 1 -- 172.21.15.110:0/381209216 >> v1:172.21.15.110:6789/0 conn(0x7f35e00cbeb0 legacy=0x7f35e00ad9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:59.278 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:59.268+0000 7f35e4ea2700 1 --2- 172.21.15.110:0/381209216 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f35cc003a60 0x7f35cc003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:59.278 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:59.268+0000 7f35e4ea2700 1 --2- 172.21.15.110:0/381209216 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f35cc004810 0x7f35cc004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:59.278 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:59.268+0000 7f35e4ea2700 1 --2- 172.21.15.110:0/381209216 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f35cc0051c0 0x7f35cc00ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:59.278 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:59.268+0000 7f35e4ea2700 1 -- 172.21.15.110:0/381209216 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35e00af400 con 0x7f35cc004810 2023-10-06T03:30:59.278 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:59.268+0000 7f35e4ea2700 1 -- 172.21.15.110:0/381209216 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35cc00c0d0 con 0x7f35cc003a60 2023-10-06T03:30:59.278 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:59.268+0000 7f35e4ea2700 1 -- 172.21.15.110:0/381209216 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35cc00c290 con 0x7f35cc0051c0 2023-10-06T03:30:59.278 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:59.268+0000 7f35e56a3700 1 --2- 172.21.15.110:0/381209216 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f35cc004810 0x7f35cc004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:59.278 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:59.268+0000 7f35e5ea4700 1 --2- 172.21.15.110:0/381209216 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f35cc003a60 0x7f35cc003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:59.278 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:59.268+0000 7f35e66a5700 1 --2- 172.21.15.110:0/381209216 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f35cc0051c0 0x7f35cc00ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:59.278 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:59.272+0000 7f35e56a3700 1 -- 172.21.15.110:0/381209216 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f35cc0051c0 msgr2=0x7f35cc00ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:30:59.279 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:59.272+0000 7f35e56a3700 1 --2- 172.21.15.110:0/381209216 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f35cc0051c0 0x7f35cc00ba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:59.279 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:59.272+0000 7f35e56a3700 1 -- 172.21.15.110:0/381209216 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f35cc003a60 msgr2=0x7f35cc003e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:30:59.279 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:59.272+0000 7f35e56a3700 1 --2- 172.21.15.110:0/381209216 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f35cc003a60 0x7f35cc003e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:59.279 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:59.272+0000 7f35e66a5700 1 -- 172.21.15.110:0/381209216 reap_dead start 2023-10-06T03:30:59.279 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:59.272+0000 7f35e56a3700 1 -- 172.21.15.110:0/381209216 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35cc003570 con 0x7f35cc004810 2023-10-06T03:30:59.279 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:59.272+0000 7f35e56a3700 1 --2- 172.21.15.110:0/381209216 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f35cc004810 0x7f35cc004c80 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f35d000c3c0 tx=0x7f35d0001e50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=58502fe40fb0e517 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:30:59.279 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:59.272+0000 7f35e4ea2700 1 -- 172.21.15.110:0/381209216 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35d000cc50 con 0x7f35cc004810 2023-10-06T03:30:59.279 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:59.272+0000 7f35e4ea2700 1 -- 172.21.15.110:0/381209216 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35d000cdb0 con 0x7f35cc004810 2023-10-06T03:30:59.279 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:59.272+0000 7f35e4ea2700 1 -- 172.21.15.110:0/381209216 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35d0005b50 con 0x7f35cc004810 2023-10-06T03:30:59.279 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:59.272+0000 7f35e8108700 1 -- 172.21.15.110:0/381209216 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f35cc004810 msgr2=0x7f35cc004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:59.279 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:59.272+0000 7f35e8108700 1 --2- 172.21.15.110:0/381209216 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f35cc004810 0x7f35cc004c80 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f35d000c3c0 tx=0x7f35d0001e50 comp rx=0 tx=0).stop 2023-10-06T03:30:59.280 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:59.272+0000 7f35e8108700 1 -- 172.21.15.110:0/381209216 shutdown_connections 2023-10-06T03:30:59.280 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:59.272+0000 7f35e8108700 1 --2- 172.21.15.110:0/381209216 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f35cc004810 0x7f35cc004c80 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:59.280 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:59.272+0000 7f35e8108700 1 -- 172.21.15.110:0/381209216 >> 172.21.15.110:0/381209216 conn(0x7f35e0062ba0 msgr2=0x7f35e0062fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:30:59.280 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:59.272+0000 7f35e8108700 1 -- 172.21.15.110:0/381209216 shutdown_connections 2023-10-06T03:30:59.280 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:59.272+0000 7f35e8108700 1 -- 172.21.15.110:0/381209216 wait complete. 2023-10-06T03:30:59.282 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:59.272+0000 7f35e8108700 1 Processor -- start 2023-10-06T03:30:59.282 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:59.272+0000 7f35e8108700 1 -- start start 2023-10-06T03:30:59.283 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:59.276+0000 7f35e8108700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f35cc004810 0x7f35e0146fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:59.283 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:59.276+0000 7f35e8108700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f35e00c5a00 0x7f35e00ad7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:59.283 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:59.276+0000 7f35e8108700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f35e00add10 0x7f35e014aeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:59.283 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:59.276+0000 7f35e8108700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35e00c4960 con 0x7f35cc004810 2023-10-06T03:30:59.283 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:59.276+0000 7f35e8108700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35e00c4c40 con 0x7f35e00c5a00 2023-10-06T03:30:59.283 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:59.276+0000 7f35e8108700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35e00c4f20 con 0x7f35e00add10 2023-10-06T03:30:59.283 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:59.276+0000 7f35e5ea4700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f35cc004810 0x7f35e0146fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:59.283 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:59.276+0000 7f35e5ea4700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f35cc004810 0x7f35e0146fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:51194/0 (socket says 172.21.15.110:51194) 2023-10-06T03:30:59.283 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:59.276+0000 7f35e5ea4700 1 -- 172.21.15.110:0/1458029987 learned_addr learned my addr 172.21.15.110:0/1458029987 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:30:59.283 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:59.276+0000 7f35e66a5700 1 --2- 172.21.15.110:0/1458029987 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f35e00add10 0x7f35e014aeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:59.283 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:59.276+0000 7f35e5ea4700 1 -- 172.21.15.110:0/1458029987 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f35e00add10 msgr2=0x7f35e014aeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:59.284 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:59.276+0000 7f35e56a3700 1 --2- 172.21.15.110:0/1458029987 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f35e00c5a00 0x7f35e00ad7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:59.284 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:59.276+0000 7f35e5ea4700 1 --2- 172.21.15.110:0/1458029987 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f35e00add10 0x7f35e014aeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:59.284 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:59.276+0000 7f35e5ea4700 1 -- 172.21.15.110:0/1458029987 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f35e00c5a00 msgr2=0x7f35e00ad7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:59.284 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:59.276+0000 7f35e5ea4700 1 --2- 172.21.15.110:0/1458029987 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f35e00c5a00 0x7f35e00ad7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:59.284 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:59.276+0000 7f35e5ea4700 1 -- 172.21.15.110:0/1458029987 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35e00c5200 con 0x7f35cc004810 2023-10-06T03:30:59.284 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:59.276+0000 7f35e56a3700 1 --2- 172.21.15.110:0/1458029987 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f35e00c5a00 0x7f35e00ad7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-06T03:30:59.284 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:59.276+0000 7f35e66a5700 1 --2- 172.21.15.110:0/1458029987 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f35e00add10 0x7f35e014aeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-06T03:30:59.284 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:59.276+0000 7f35e5ea4700 1 --2- 172.21.15.110:0/1458029987 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f35cc004810 0x7f35e0146fa0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f35dc009e20 tx=0x7f35dc00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:30:59.284 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:59.276+0000 7f35d5ffb700 1 -- 172.21.15.110:0/1458029987 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35dc00e040 con 0x7f35cc004810 2023-10-06T03:30:59.284 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:59.276+0000 7f35e8108700 1 -- 172.21.15.110:0/1458029987 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35e00c54e0 con 0x7f35cc004810 2023-10-06T03:30:59.284 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:59.276+0000 7f35d5ffb700 1 -- 172.21.15.110:0/1458029987 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35dc00f040 con 0x7f35cc004810 2023-10-06T03:30:59.284 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:59.276+0000 7f35d5ffb700 1 -- 172.21.15.110:0/1458029987 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35dc015430 con 0x7f35cc004810 2023-10-06T03:30:59.285 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:59.276+0000 7f35e8108700 1 -- 172.21.15.110:0/1458029987 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35e00c2a80 con 0x7f35cc004810 2023-10-06T03:30:59.285 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:59.276+0000 7f35e8108700 1 -- 172.21.15.110:0/1458029987 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35e0064770 con 0x7f35cc004810 2023-10-06T03:30:59.285 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:59.280+0000 7f35d5ffb700 1 -- 172.21.15.110:0/1458029987 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 10) v1 ==== 80118+0+0 (secure 0 0 0) 0x7f35dc015820 con 0x7f35cc004810 2023-10-06T03:30:59.285 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:59.280+0000 7f35d5ffb700 1 --2- 172.21.15.110:0/1458029987 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f35c4060700 0x7f35c4062bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:59.286 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:59.280+0000 7f35e56a3700 1 -- 172.21.15.110:0/1458029987 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f35c4060700 msgr2=0x7f35c4062bb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.110:6832/33268 2023-10-06T03:30:59.286 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:59.280+0000 7f35d5ffb700 1 -- 172.21.15.110:0/1458029987 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 6636+0+0 (secure 0 0 0) 0x7f35dc07e1a0 con 0x7f35cc004810 2023-10-06T03:30:59.286 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:59.280+0000 7f35e56a3700 1 --2- 172.21.15.110:0/1458029987 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f35c4060700 0x7f35c4062bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-06T03:30:59.289 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:59.284+0000 7f35d5ffb700 1 -- 172.21.15.110:0/1458029987 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f35dc04e0e0 con 0x7f35cc004810 2023-10-06T03:30:59.314 INFO:tasks.ceph.mgr.x.smithi192.stderr:2023-10-06T03:30:59.312+0000 7f76034c8000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-06T03:30:59.336 INFO:tasks.ceph.mgr.y.smithi110.stderr:2023-10-06T03:30:59.332+0000 7ff496339000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-06T03:30:59.386 INFO:tasks.ceph.mgr.y.smithi110.stderr:2023-10-06T03:30:59.380+0000 7ff496339000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-06T03:30:59.447 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:59.444+0000 7f35e8108700 1 -- 172.21.15.110:0/1458029987 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f35e00c92d0 con 0x7f35cc004810 2023-10-06T03:30:59.448 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:59.444+0000 7f35d5ffb700 1 -- 172.21.15.110:0/1458029987 <== mon.0 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v43) v1 ==== 93+0+10 (secure 0 0 0) 0x7f35dc051700 con 0x7f35cc004810 2023-10-06T03:30:59.448 INFO:teuthology.orchestra.run.smithi110.stdout:pg_num: 1 2023-10-06T03:30:59.451 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:59.444+0000 7f35e8108700 1 -- 172.21.15.110:0/1458029987 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f35c4060700 msgr2=0x7f35c4062bb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-06T03:30:59.451 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:59.444+0000 7f35e8108700 1 --2- 172.21.15.110:0/1458029987 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f35c4060700 0x7f35c4062bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:59.451 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:59.444+0000 7f35e8108700 1 -- 172.21.15.110:0/1458029987 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f35cc004810 msgr2=0x7f35e0146fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:59.451 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:59.444+0000 7f35e8108700 1 --2- 172.21.15.110:0/1458029987 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f35cc004810 0x7f35e0146fa0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f35dc009e20 tx=0x7f35dc00b040 comp rx=0 tx=0).stop 2023-10-06T03:30:59.451 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:59.444+0000 7f35e8108700 1 -- 172.21.15.110:0/1458029987 shutdown_connections 2023-10-06T03:30:59.451 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:59.444+0000 7f35e8108700 1 --2- 172.21.15.110:0/1458029987 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f35c4060700 0x7f35c4062bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:59.451 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:59.444+0000 7f35e8108700 1 --2- 172.21.15.110:0/1458029987 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f35e00add10 0x7f35e014aeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:59.451 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:59.444+0000 7f35e8108700 1 --2- 172.21.15.110:0/1458029987 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f35e00c5a00 0x7f35e00ad7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:59.451 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:59.444+0000 7f35e8108700 1 --2- 172.21.15.110:0/1458029987 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f35cc004810 0x7f35e0146fa0 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:59.451 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:59.444+0000 7f35e8108700 1 -- 172.21.15.110:0/1458029987 >> 172.21.15.110:0/1458029987 conn(0x7f35e0062ba0 msgr2=0x7f35e00b0a90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:30:59.452 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:59.444+0000 7f35e8108700 1 -- 172.21.15.110:0/1458029987 shutdown_connections 2023-10-06T03:30:59.452 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:59.444+0000 7f35e8108700 1 -- 172.21.15.110:0/1458029987 wait complete. 2023-10-06T03:30:59.462 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-10-06T03:30:59.467 INFO:tasks.ceph.mgr.x.smithi192.stderr:2023-10-06T03:30:59.464+0000 7f76034c8000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-06T03:30:59.523 INFO:tasks.ceph.mgr.x.smithi192.stderr:2023-10-06T03:30:59.520+0000 7f76034c8000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-06T03:30:59.556 INFO:tasks.ceph.mgr.y.smithi110.stderr:2023-10-06T03:30:59.552+0000 7ff496339000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-06T03:30:59.574 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:59.568+0000 7f9d5bfff700 1 Processor -- start 2023-10-06T03:30:59.574 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:59.568+0000 7f9d5bfff700 1 -- start start 2023-10-06T03:30:59.578 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:59.572+0000 7f9d5bfff700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9d5c1559f0 0x7f9d5c0c8540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:59.578 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:59.572+0000 7f9d5bfff700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d5c0c9220 con 0x7f9d5c0c8a80 2023-10-06T03:30:59.579 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:59.572+0000 7f9d5bfff700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d5c159df0 con 0x7f9d5c06bb50 2023-10-06T03:30:59.579 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:59.572+0000 7f9d5bfff700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d5c0c8f40 con 0x7f9d5c1559f0 2023-10-06T03:30:59.579 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:59.572+0000 7f9d53fff700 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f9d5c0c8a80 0x7f9d5c06b440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:40064/0 (socket says 172.21.15.110:40064) 2023-10-06T03:30:59.579 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:59.572+0000 7f9d53fff700 1 -- 172.21.15.110:0/1820300333 learned_addr learned my addr 172.21.15.110:0/1820300333 (peer_addr_for_me v1:172.21.15.110:0/0) 2023-10-06T03:30:59.579 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:59.572+0000 7f9d5affd700 1 --2- 172.21.15.110:0/1820300333 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9d5c1559f0 0x7f9d5c0c8540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:59.579 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:59.572+0000 7f9d5a7fc700 1 -- 172.21.15.110:0/1820300333 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4243498672 0 0) 0x7f9d5c0c9220 con 0x7f9d5c0c8a80 2023-10-06T03:30:59.579 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:59.572+0000 7f9d5a7fc700 1 -- 172.21.15.110:0/1820300333 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9d48003570 con 0x7f9d5c0c8a80 2023-10-06T03:30:59.579 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:59.572+0000 7f9d5a7fc700 1 -- 172.21.15.110:0/1820300333 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2851598147 0 0) 0x7f9d5c159df0 con 0x7f9d5c06bb50 2023-10-06T03:30:59.579 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:59.572+0000 7f9d5a7fc700 1 -- 172.21.15.110:0/1820300333 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9d5c0c9220 con 0x7f9d5c06bb50 2023-10-06T03:30:59.579 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:59.572+0000 7f9d5a7fc700 1 -- 172.21.15.110:0/1820300333 <== mon.0 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4027443676 0 0) 0x7f9d48003570 con 0x7f9d5c0c8a80 2023-10-06T03:30:59.580 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:59.572+0000 7f9d5a7fc700 1 -- 172.21.15.110:0/1820300333 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9d5c1559f0 msgr2=0x7f9d5c0c8540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:30:59.580 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:59.572+0000 7f9d5a7fc700 1 --2- 172.21.15.110:0/1820300333 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9d5c1559f0 0x7f9d5c0c8540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:59.580 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:59.572+0000 7f9d5a7fc700 1 -- 172.21.15.110:0/1820300333 >> v1:172.21.15.192:6789/0 conn(0x7f9d5c06bb50 legacy=0x7f9d5c0ca600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:59.580 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:59.572+0000 7f9d5a7fc700 1 -- 172.21.15.110:0/1820300333 --> v1:172.21.15.110:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d5c1597a0 con 0x7f9d5c0c8a80 2023-10-06T03:30:59.580 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:59.572+0000 7f9d5a7fc700 1 -- 172.21.15.110:0/1820300333 <== mon.0 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 118603703 0 0) 0x7f9d44001470 con 0x7f9d5c0c8a80 2023-10-06T03:30:59.580 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:59.572+0000 7f9d5a7fc700 1 -- 172.21.15.110:0/1820300333 >> v1:172.21.15.110:6789/0 conn(0x7f9d5c0c8a80 legacy=0x7f9d5c06b440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:59.580 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:59.572+0000 7f9d5a7fc700 1 --2- 172.21.15.110:0/1820300333 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9d48003c20 0x7f9d48004010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:59.580 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:59.572+0000 7f9d5a7fc700 1 --2- 172.21.15.110:0/1820300333 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9d5c1559f0 0x7f9d48004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:59.580 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:59.572+0000 7f9d5a7fc700 1 --2- 172.21.15.110:0/1820300333 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9d48004d50 0x7f9d480051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:59.580 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:59.572+0000 7f9d5a7fc700 1 -- 172.21.15.110:0/1820300333 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d5c0c8f40 con 0x7f9d48003c20 2023-10-06T03:30:59.580 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:59.572+0000 7f9d5a7fc700 1 -- 172.21.15.110:0/1820300333 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d4800bbf0 con 0x7f9d48004d50 2023-10-06T03:30:59.580 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:59.572+0000 7f9d5a7fc700 1 -- 172.21.15.110:0/1820300333 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d4800bd60 con 0x7f9d5c1559f0 2023-10-06T03:30:59.581 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:59.572+0000 7f9d5affd700 1 --2- 172.21.15.110:0/1820300333 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9d48003c20 0x7f9d48004010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:59.581 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:59.572+0000 7f9d53fff700 1 --2- 172.21.15.110:0/1820300333 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9d5c1559f0 0x7f9d48004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:59.581 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:59.572+0000 7f9d5b7fe700 1 --2- 172.21.15.110:0/1820300333 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9d48004d50 0x7f9d480051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:59.581 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:59.572+0000 7f9d5affd700 1 -- 172.21.15.110:0/1820300333 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9d5c1559f0 msgr2=0x7f9d48004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:30:59.581 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:59.572+0000 7f9d5affd700 1 --2- 172.21.15.110:0/1820300333 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9d5c1559f0 0x7f9d48004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:59.581 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:59.572+0000 7f9d5affd700 1 -- 172.21.15.110:0/1820300333 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9d48004d50 msgr2=0x7f9d480051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:30:59.581 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:59.572+0000 7f9d5affd700 1 --2- 172.21.15.110:0/1820300333 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9d48004d50 0x7f9d480051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:59.581 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:59.572+0000 7f9d5affd700 1 -- 172.21.15.110:0/1820300333 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d48003570 con 0x7f9d48003c20 2023-10-06T03:30:59.581 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:59.572+0000 7f9d5b7fe700 1 --2- 172.21.15.110:0/1820300333 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9d48004d50 0x7f9d480051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-06T03:30:59.581 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:59.572+0000 7f9d5affd700 1 --2- 172.21.15.110:0/1820300333 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9d48003c20 0x7f9d48004010 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f9d4c006f00 tx=0x7f9d4c0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6639e1bd2b40a8bc server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:30:59.581 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:59.572+0000 7f9d5a7fc700 1 -- 172.21.15.110:0/1820300333 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d4c00e710 con 0x7f9d48003c20 2023-10-06T03:30:59.582 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:59.572+0000 7f9d5a7fc700 1 -- 172.21.15.110:0/1820300333 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d4c00e870 con 0x7f9d48003c20 2023-10-06T03:30:59.582 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:59.572+0000 7f9d5a7fc700 1 -- 172.21.15.110:0/1820300333 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d4c00ec30 con 0x7f9d48003c20 2023-10-06T03:30:59.582 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:59.572+0000 7f9d5bfff700 1 -- 172.21.15.110:0/1820300333 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9d48003c20 msgr2=0x7f9d48004010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:59.582 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:59.572+0000 7f9d5bfff700 1 --2- 172.21.15.110:0/1820300333 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9d48003c20 0x7f9d48004010 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f9d4c006f00 tx=0x7f9d4c0091a0 comp rx=0 tx=0).stop 2023-10-06T03:30:59.582 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:59.572+0000 7f9d5b7fe700 1 -- 172.21.15.110:0/1820300333 reap_dead start 2023-10-06T03:30:59.582 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:59.572+0000 7f9d5bfff700 1 -- 172.21.15.110:0/1820300333 shutdown_connections 2023-10-06T03:30:59.582 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:59.572+0000 7f9d5bfff700 1 -- 172.21.15.110:0/1820300333 >> 172.21.15.110:0/1820300333 conn(0x7f9d5c067f20 msgr2=0x7f9d5c0cca30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:30:59.582 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:59.572+0000 7f9d5bfff700 1 -- 172.21.15.110:0/1820300333 shutdown_connections 2023-10-06T03:30:59.582 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:59.572+0000 7f9d5bfff700 1 -- 172.21.15.110:0/1820300333 wait complete. 2023-10-06T03:30:59.583 INFO:tasks.ceph.mgr.x.smithi192.stderr:2023-10-06T03:30:59.580+0000 7f76034c8000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-06T03:30:59.585 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:59.576+0000 7f9d5bfff700 1 Processor -- start 2023-10-06T03:30:59.585 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:59.576+0000 7f9d5bfff700 1 -- start start 2023-10-06T03:30:59.585 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:59.576+0000 7f9d5bfff700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9d48003c20 0x7f9d5c1485f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:59.585 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:59.576+0000 7f9d5bfff700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9d5c153e10 0x7f9d5c154280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:59.585 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:59.576+0000 7f9d5bfff700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9d5c0b0a40 0x7f9d5c0b0ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:59.585 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:59.576+0000 7f9d5bfff700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d5c16a3b0 con 0x7f9d5c0b0a40 2023-10-06T03:30:59.586 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:59.576+0000 7f9d5bfff700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d5c1693d0 con 0x7f9d5c153e10 2023-10-06T03:30:59.586 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:59.576+0000 7f9d5bfff700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d5c1547c0 con 0x7f9d48003c20 2023-10-06T03:30:59.586 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:59.576+0000 7f9d5b7fe700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9d5c0b0a40 0x7f9d5c0b0ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:59.586 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:59.576+0000 7f9d5b7fe700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9d5c0b0a40 0x7f9d5c0b0ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:51220/0 (socket says 172.21.15.110:51220) 2023-10-06T03:30:59.586 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:59.576+0000 7f9d5b7fe700 1 -- 172.21.15.110:0/684948555 learned_addr learned my addr 172.21.15.110:0/684948555 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:30:59.586 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:59.576+0000 7f9d5affd700 1 --2- 172.21.15.110:0/684948555 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9d48003c20 0x7f9d5c1485f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:59.586 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:59.576+0000 7f9d53fff700 1 --2- 172.21.15.110:0/684948555 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9d5c153e10 0x7f9d5c154280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:59.586 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:59.576+0000 7f9d5b7fe700 1 -- 172.21.15.110:0/684948555 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9d48003c20 msgr2=0x7f9d5c1485f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:59.586 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:59.576+0000 7f9d5b7fe700 1 --2- 172.21.15.110:0/684948555 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9d48003c20 0x7f9d5c1485f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:59.586 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:59.576+0000 7f9d5b7fe700 1 -- 172.21.15.110:0/684948555 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9d5c153e10 msgr2=0x7f9d5c154280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:59.586 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:59.576+0000 7f9d5b7fe700 1 --2- 172.21.15.110:0/684948555 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9d5c153e10 0x7f9d5c154280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:59.587 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:59.576+0000 7f9d5b7fe700 1 -- 172.21.15.110:0/684948555 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d5c169f20 con 0x7f9d5c0b0a40 2023-10-06T03:30:59.587 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:59.576+0000 7f9d5b7fe700 1 --2- 172.21.15.110:0/684948555 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9d5c0b0a40 0x7f9d5c0b0ec0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f9d5c063690 tx=0x7f9d54009040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:30:59.587 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:59.576+0000 7f9d52ffd700 1 -- 172.21.15.110:0/684948555 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d540032e0 con 0x7f9d5c0b0a40 2023-10-06T03:30:59.587 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:59.576+0000 7f9d52ffd700 1 -- 172.21.15.110:0/684948555 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d54003440 con 0x7f9d5c0b0a40 2023-10-06T03:30:59.587 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:59.576+0000 7f9d52ffd700 1 -- 172.21.15.110:0/684948555 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d54006df0 con 0x7f9d5c0b0a40 2023-10-06T03:30:59.587 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:59.576+0000 7f9d5bfff700 1 -- 172.21.15.110:0/684948555 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d5c1695f0 con 0x7f9d5c0b0a40 2023-10-06T03:30:59.587 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:59.576+0000 7f9d5bfff700 1 -- 172.21.15.110:0/684948555 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d5c169bb0 con 0x7f9d5c0b0a40 2023-10-06T03:30:59.587 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:59.576+0000 7f9d5bfff700 1 -- 172.21.15.110:0/684948555 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d5c16a0b0 con 0x7f9d5c0b0a40 2023-10-06T03:30:59.587 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:59.576+0000 7f9d52ffd700 1 -- 172.21.15.110:0/684948555 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 10) v1 ==== 80118+0+0 (secure 0 0 0) 0x7f9d540018f0 con 0x7f9d5c0b0a40 2023-10-06T03:30:59.587 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:59.580+0000 7f9d52ffd700 1 --2- 172.21.15.110:0/684948555 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f9d3c060720 0x7f9d3c062bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:59.587 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:59.580+0000 7f9d5affd700 1 -- 172.21.15.110:0/684948555 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f9d3c060720 msgr2=0x7f9d3c062bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.110:6832/33268 2023-10-06T03:30:59.588 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:59.580+0000 7f9d52ffd700 1 -- 172.21.15.110:0/684948555 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 6636+0+0 (secure 0 0 0) 0x7f9d5407c2e0 con 0x7f9d5c0b0a40 2023-10-06T03:30:59.588 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:59.580+0000 7f9d5affd700 1 --2- 172.21.15.110:0/684948555 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f9d3c060720 0x7f9d3c062bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-06T03:30:59.589 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:59.584+0000 7f9d52ffd700 1 -- 172.21.15.110:0/684948555 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f9d5404c2b0 con 0x7f9d5c0b0a40 2023-10-06T03:30:59.614 INFO:tasks.ceph.mgr.y.smithi110.stderr:2023-10-06T03:30:59.608+0000 7ff496339000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-06T03:30:59.674 INFO:tasks.ceph.mgr.y.smithi110.stderr:2023-10-06T03:30:59.668+0000 7ff496339000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-06T03:30:59.741 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:59.736+0000 7f9d5bfff700 1 -- 172.21.15.110:0/684948555 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f9d5c071020 con 0x7f9d5c0b0a40 2023-10-06T03:30:59.742 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:59.736+0000 7f9d52ffd700 1 -- 172.21.15.110:0/684948555 <== mon.0 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v43) v1 ==== 92+0+10 (secure 0 0 0) 0x7f9d5404f8d0 con 0x7f9d5c0b0a40 2023-10-06T03:30:59.742 INFO:teuthology.orchestra.run.smithi110.stdout:pg_num: 8 2023-10-06T03:30:59.744 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:59.740+0000 7f9d5bfff700 1 -- 172.21.15.110:0/684948555 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f9d3c060720 msgr2=0x7f9d3c062bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-06T03:30:59.744 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:59.740+0000 7f9d5bfff700 1 --2- 172.21.15.110:0/684948555 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f9d3c060720 0x7f9d3c062bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:59.745 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:59.740+0000 7f9d5bfff700 1 -- 172.21.15.110:0/684948555 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9d5c0b0a40 msgr2=0x7f9d5c0b0ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:59.745 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:59.740+0000 7f9d5bfff700 1 --2- 172.21.15.110:0/684948555 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9d5c0b0a40 0x7f9d5c0b0ec0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f9d5c063690 tx=0x7f9d54009040 comp rx=0 tx=0).stop 2023-10-06T03:30:59.745 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:59.740+0000 7f9d5bfff700 1 -- 172.21.15.110:0/684948555 shutdown_connections 2023-10-06T03:30:59.745 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:59.740+0000 7f9d5bfff700 1 --2- 172.21.15.110:0/684948555 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f9d3c060720 0x7f9d3c062bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:59.745 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:59.740+0000 7f9d5bfff700 1 --2- 172.21.15.110:0/684948555 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9d5c0b0a40 0x7f9d5c0b0ec0 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:59.745 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:59.740+0000 7f9d5bfff700 1 --2- 172.21.15.110:0/684948555 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9d5c153e10 0x7f9d5c154280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:59.745 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:59.740+0000 7f9d5bfff700 1 --2- 172.21.15.110:0/684948555 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9d48003c20 0x7f9d5c1485f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:59.745 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:59.740+0000 7f9d5bfff700 1 -- 172.21.15.110:0/684948555 >> 172.21.15.110:0/684948555 conn(0x7f9d5c067f20 msgr2=0x7f9d5c0cca30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:30:59.745 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:59.740+0000 7f9d5bfff700 1 -- 172.21.15.110:0/684948555 shutdown_connections 2023-10-06T03:30:59.745 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:59.740+0000 7f9d5bfff700 1 -- 172.21.15.110:0/684948555 wait complete. 2023-10-06T03:30:59.755 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-06T03:30:59.870 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:59.864+0000 7f2410971700 1 Processor -- start 2023-10-06T03:30:59.870 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:59.864+0000 7f2410971700 1 -- start start 2023-10-06T03:30:59.871 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:59.864+0000 7f2410971700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f240c155d80 0x7f240c0c97b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:59.871 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:59.864+0000 7f2410971700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f240c0ae620 0x7f240c06d770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:59.872 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:59.864+0000 7f2410971700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f240c0cab10 con 0x7f240c1559b0 2023-10-06T03:30:59.872 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:59.864+0000 7f2410971700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f240c0c9cf0 con 0x7f240c0ae620 2023-10-06T03:30:59.872 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:59.864+0000 7f2410971700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f240c0c9f00 con 0x7f240c155d80 2023-10-06T03:30:59.872 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:59.864+0000 7f240b7fe700 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f240c1559b0 0x7f240c0aef80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:40068/0 (socket says 172.21.15.110:40068) 2023-10-06T03:30:59.872 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:59.864+0000 7f240b7fe700 1 -- 172.21.15.110:0/2920634006 learned_addr learned my addr 172.21.15.110:0/2920634006 (peer_addr_for_me v1:172.21.15.110:0/0) 2023-10-06T03:30:59.872 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:59.868+0000 7f240affd700 1 --2- 172.21.15.110:0/2920634006 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f240c155d80 0x7f240c0c97b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:59.872 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:59.868+0000 7f240bfff700 1 --2- 172.21.15.110:0/2920634006 >> v2:172.21.15.192:3300/0 conn(0x7f240c0ae620 0x7f240c06d770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:59.872 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:59.868+0000 7f240a7fc700 1 -- 172.21.15.110:0/2920634006 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3352252674 0 0) 0x7f240c0cab10 con 0x7f240c1559b0 2023-10-06T03:30:59.872 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:59.868+0000 7f240a7fc700 1 -- 172.21.15.110:0/2920634006 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f23f4003570 con 0x7f240c1559b0 2023-10-06T03:30:59.874 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:59.868+0000 7f240a7fc700 1 -- 172.21.15.110:0/2920634006 <== mon.0 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3659068894 0 0) 0x7f23f4003570 con 0x7f240c1559b0 2023-10-06T03:30:59.874 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:59.868+0000 7f240a7fc700 1 -- 172.21.15.110:0/2920634006 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f240c155d80 msgr2=0x7f240c0c97b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:30:59.875 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:59.868+0000 7f240a7fc700 1 --2- 172.21.15.110:0/2920634006 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f240c155d80 0x7f240c0c97b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:59.875 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:59.868+0000 7f240a7fc700 1 -- 172.21.15.110:0/2920634006 >> v2:172.21.15.192:3300/0 conn(0x7f240c0ae620 msgr2=0x7f240c06d770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:30:59.875 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:59.868+0000 7f240a7fc700 1 --2- 172.21.15.110:0/2920634006 >> v2:172.21.15.192:3300/0 conn(0x7f240c0ae620 0x7f240c06d770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:59.875 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:59.868+0000 7f240a7fc700 1 -- 172.21.15.110:0/2920634006 --> v1:172.21.15.110:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f240c0ca1e0 con 0x7f240c1559b0 2023-10-06T03:30:59.875 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:59.868+0000 7f240a7fc700 1 -- 172.21.15.110:0/2920634006 <== mon.0 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 118603703 0 0) 0x7f23fc001470 con 0x7f240c1559b0 2023-10-06T03:30:59.875 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:59.868+0000 7f240bfff700 1 --2- 172.21.15.110:0/2920634006 >> v2:172.21.15.192:3300/0 conn(0x7f240c0ae620 0x7f240c06d770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-06T03:30:59.875 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:59.868+0000 7f240a7fc700 1 -- 172.21.15.110:0/2920634006 >> v1:172.21.15.110:6789/0 conn(0x7f240c1559b0 legacy=0x7f240c0aef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:59.875 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:59.868+0000 7f240a7fc700 1 --2- 172.21.15.110:0/2920634006 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f23f4003a60 0x7f23f4003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:59.875 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:59.868+0000 7f240a7fc700 1 --2- 172.21.15.110:0/2920634006 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f23f4004810 0x7f23f4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:59.875 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:59.868+0000 7f240a7fc700 1 --2- 172.21.15.110:0/2920634006 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f240c155d80 0x7f23f40051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:59.876 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:59.868+0000 7f240a7fc700 1 -- 172.21.15.110:0/2920634006 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f240c0c9cf0 con 0x7f23f4003a60 2023-10-06T03:30:59.876 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:59.868+0000 7f240a7fc700 1 -- 172.21.15.110:0/2920634006 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f240c0c9f00 con 0x7f23f4004810 2023-10-06T03:30:59.876 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:59.868+0000 7f240a7fc700 1 -- 172.21.15.110:0/2920634006 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23f4009a50 con 0x7f240c155d80 2023-10-06T03:30:59.876 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:59.868+0000 7f240b7fe700 1 --2- 172.21.15.110:0/2920634006 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f23f4003a60 0x7f23f4003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:59.876 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:59.868+0000 7f240affd700 1 --2- 172.21.15.110:0/2920634006 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f23f4004810 0x7f23f4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:59.876 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:59.868+0000 7f240bfff700 1 --2- 172.21.15.110:0/2920634006 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f240c155d80 0x7f23f40051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:59.876 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:59.868+0000 7f240b7fe700 1 -- 172.21.15.110:0/2920634006 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f240c155d80 msgr2=0x7f23f40051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:30:59.876 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:59.868+0000 7f240b7fe700 1 --2- 172.21.15.110:0/2920634006 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f240c155d80 0x7f23f40051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:59.876 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:59.868+0000 7f240b7fe700 1 -- 172.21.15.110:0/2920634006 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f23f4004810 msgr2=0x7f23f4004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:30:59.876 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:59.868+0000 7f240b7fe700 1 --2- 172.21.15.110:0/2920634006 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f23f4004810 0x7f23f4004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:59.876 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:59.868+0000 7f240b7fe700 1 -- 172.21.15.110:0/2920634006 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23f4003570 con 0x7f23f4003a60 2023-10-06T03:30:59.877 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:59.868+0000 7f240bfff700 1 --2- 172.21.15.110:0/2920634006 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f240c155d80 0x7f23f40051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-06T03:30:59.877 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:59.868+0000 7f240b7fe700 1 --2- 172.21.15.110:0/2920634006 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f23f4003a60 0x7f23f4003e70 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f23fc00c3c0 tx=0x7f23fc001e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=56170ae52d3df5bf server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:30:59.877 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:59.868+0000 7f240a7fc700 1 -- 172.21.15.110:0/2920634006 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23fc00cc90 con 0x7f23f4003a60 2023-10-06T03:30:59.877 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:59.868+0000 7f240a7fc700 1 -- 172.21.15.110:0/2920634006 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23fc00ce10 con 0x7f23f4003a60 2023-10-06T03:30:59.877 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:59.868+0000 7f240a7fc700 1 -- 172.21.15.110:0/2920634006 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23fc005b70 con 0x7f23f4003a60 2023-10-06T03:30:59.877 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:59.868+0000 7f2410971700 1 -- 172.21.15.110:0/2920634006 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f23f4003a60 msgr2=0x7f23f4003e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:59.877 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:59.868+0000 7f2410971700 1 --2- 172.21.15.110:0/2920634006 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f23f4003a60 0x7f23f4003e70 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f23fc00c3c0 tx=0x7f23fc001e90 comp rx=0 tx=0).stop 2023-10-06T03:30:59.877 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:59.868+0000 7f240bfff700 1 -- 172.21.15.110:0/2920634006 reap_dead start 2023-10-06T03:30:59.877 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:59.868+0000 7f2410971700 1 -- 172.21.15.110:0/2920634006 shutdown_connections 2023-10-06T03:30:59.877 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:59.868+0000 7f2410971700 1 -- 172.21.15.110:0/2920634006 >> 172.21.15.110:0/2920634006 conn(0x7f240c06b6c0 msgr2=0x7f240c0adbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:30:59.877 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:59.868+0000 7f2410971700 1 -- 172.21.15.110:0/2920634006 shutdown_connections 2023-10-06T03:30:59.877 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:59.868+0000 7f2410971700 1 -- 172.21.15.110:0/2920634006 wait complete. 2023-10-06T03:30:59.879 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:59.872+0000 7f2410971700 1 Processor -- start 2023-10-06T03:30:59.880 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:59.872+0000 7f2410971700 1 -- start start 2023-10-06T03:30:59.880 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:59.872+0000 7f2410971700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f23f4003a60 0x7f240c14fea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:59.880 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:59.872+0000 7f2410971700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f240c158110 0x7f240c067040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:59.880 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:59.872+0000 7f2410971700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f240c067580 0x7f240c164fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:59.880 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:59.872+0000 7f2410971700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f240c159400 con 0x7f23f4003a60 2023-10-06T03:30:59.880 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:59.872+0000 7f2410971700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f240c0679f0 con 0x7f240c158110 2023-10-06T03:30:59.880 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:59.872+0000 7f2410971700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f240c158580 con 0x7f240c067580 2023-10-06T03:30:59.880 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:59.872+0000 7f240b7fe700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f23f4003a60 0x7f240c14fea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:59.880 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:59.872+0000 7f240b7fe700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f23f4003a60 0x7f240c14fea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:51238/0 (socket says 172.21.15.110:51238) 2023-10-06T03:30:59.880 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:59.872+0000 7f240b7fe700 1 -- 172.21.15.110:0/3731441635 learned_addr learned my addr 172.21.15.110:0/3731441635 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:30:59.880 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:59.872+0000 7f240affd700 1 --2- 172.21.15.110:0/3731441635 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f240c158110 0x7f240c067040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:59.881 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:59.872+0000 7f240bfff700 1 --2- 172.21.15.110:0/3731441635 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f240c067580 0x7f240c164fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:30:59.881 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:59.872+0000 7f240b7fe700 1 -- 172.21.15.110:0/3731441635 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f240c067580 msgr2=0x7f240c164fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:59.881 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:59.872+0000 7f240b7fe700 1 --2- 172.21.15.110:0/3731441635 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f240c067580 0x7f240c164fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:59.881 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:59.872+0000 7f240b7fe700 1 -- 172.21.15.110:0/3731441635 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f240c158110 msgr2=0x7f240c067040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:30:59.881 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:59.872+0000 7f240b7fe700 1 --2- 172.21.15.110:0/3731441635 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f240c158110 0x7f240c067040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:30:59.881 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:59.872+0000 7f240b7fe700 1 -- 172.21.15.110:0/3731441635 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f240c1587f0 con 0x7f23f4003a60 2023-10-06T03:30:59.881 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:59.872+0000 7f240b7fe700 1 --2- 172.21.15.110:0/3731441635 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f23f4003a60 0x7f240c14fea0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f23fc007380 tx=0x7f23fc015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:30:59.881 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:59.872+0000 7f23f37fe700 1 -- 172.21.15.110:0/3731441635 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23fc00cc90 con 0x7f23f4003a60 2023-10-06T03:30:59.881 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:59.872+0000 7f23f37fe700 1 -- 172.21.15.110:0/3731441635 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23fc00c9e0 con 0x7f23f4003a60 2023-10-06T03:30:59.881 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:59.872+0000 7f2410971700 1 -- 172.21.15.110:0/3731441635 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f240c158ad0 con 0x7f23f4003a60 2023-10-06T03:30:59.881 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:59.872+0000 7f23f37fe700 1 -- 172.21.15.110:0/3731441635 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23fc01e650 con 0x7f23f4003a60 2023-10-06T03:30:59.881 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:59.872+0000 7f2410971700 1 -- 172.21.15.110:0/3731441635 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f240c159090 con 0x7f23f4003a60 2023-10-06T03:30:59.882 INFO:tasks.ceph.mgr.x.smithi192.stderr:2023-10-06T03:30:59.876+0000 7f76034c8000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-06T03:30:59.883 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:59.876+0000 7f2410971700 1 -- 172.21.15.110:0/3731441635 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f240c164c40 con 0x7f23f4003a60 2023-10-06T03:30:59.883 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:59.876+0000 7f23f37fe700 1 -- 172.21.15.110:0/3731441635 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 10) v1 ==== 80118+0+0 (secure 0 0 0) 0x7f23fc019030 con 0x7f23f4003a60 2023-10-06T03:30:59.883 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:59.876+0000 7f23f37fe700 1 --2- 172.21.15.110:0/3731441635 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f23f8060730 0x7f23f8062be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:30:59.883 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:59.876+0000 7f23f37fe700 1 -- 172.21.15.110:0/3731441635 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 6636+0+0 (secure 0 0 0) 0x7f23fc07e110 con 0x7f23f4003a60 2023-10-06T03:30:59.883 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:59.876+0000 7f240affd700 1 -- 172.21.15.110:0/3731441635 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f23f8060730 msgr2=0x7f23f8062be0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.110:6832/33268 2023-10-06T03:30:59.883 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:59.876+0000 7f240affd700 1 --2- 172.21.15.110:0/3731441635 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f23f8060730 0x7f23f8062be0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-06T03:30:59.889 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:30:59.884+0000 7f23f37fe700 1 -- 172.21.15.110:0/3731441635 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f23fc04e0e0 con 0x7f23f4003a60 2023-10-06T03:30:59.990 INFO:tasks.ceph.mgr.x.smithi192.stderr:2023-10-06T03:30:59.984+0000 7f76034c8000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-06T03:31:00.011 INFO:tasks.ceph.mgr.y.smithi110.stderr:2023-10-06T03:31:00.008+0000 7ff496339000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-06T03:31:00.044 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:00.040+0000 7f2410971700 1 -- 172.21.15.110:0/3731441635 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f240c071020 con 0x7f23f4003a60 2023-10-06T03:31:00.044 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:00.040+0000 7f23f37fe700 1 -- 172.21.15.110:0/3731441635 <== mon.0 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v43) v1 ==== 104+0+11 (secure 0 0 0) 0x7f23fc001bb0 con 0x7f23f4003a60 2023-10-06T03:31:00.045 INFO:teuthology.orchestra.run.smithi110.stdout:pg_num: 32 2023-10-06T03:31:00.047 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:00.040+0000 7f2410971700 1 -- 172.21.15.110:0/3731441635 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f23f8060730 msgr2=0x7f23f8062be0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-06T03:31:00.047 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:00.040+0000 7f2410971700 1 --2- 172.21.15.110:0/3731441635 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f23f8060730 0x7f23f8062be0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:00.047 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:00.040+0000 7f2410971700 1 -- 172.21.15.110:0/3731441635 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f23f4003a60 msgr2=0x7f240c14fea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:31:00.047 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:00.040+0000 7f2410971700 1 --2- 172.21.15.110:0/3731441635 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f23f4003a60 0x7f240c14fea0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f23fc007380 tx=0x7f23fc015000 comp rx=0 tx=0).stop 2023-10-06T03:31:00.047 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:00.040+0000 7f2410971700 1 -- 172.21.15.110:0/3731441635 shutdown_connections 2023-10-06T03:31:00.048 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:00.040+0000 7f2410971700 1 --2- 172.21.15.110:0/3731441635 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f23f8060730 0x7f23f8062be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:00.048 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:00.040+0000 7f2410971700 1 --2- 172.21.15.110:0/3731441635 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f240c067580 0x7f240c164fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:00.048 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:00.040+0000 7f2410971700 1 --2- 172.21.15.110:0/3731441635 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f240c158110 0x7f240c067040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:00.048 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:00.040+0000 7f2410971700 1 --2- 172.21.15.110:0/3731441635 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f23f4003a60 0x7f240c14fea0 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:00.048 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:00.040+0000 7f2410971700 1 -- 172.21.15.110:0/3731441635 >> 172.21.15.110:0/3731441635 conn(0x7f240c06b6c0 msgr2=0x7f240c151690 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:31:00.048 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:00.040+0000 7f2410971700 1 -- 172.21.15.110:0/3731441635 shutdown_connections 2023-10-06T03:31:00.048 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:00.040+0000 7f2410971700 1 -- 172.21.15.110:0/3731441635 wait complete. 2023-10-06T03:31:00.058 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-06T03:31:00.135 INFO:tasks.ceph.mgr.y.smithi110.stderr:2023-10-06T03:31:00.128+0000 7ff496339000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-06T03:31:00.170 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:00.164+0000 7f95dc9f9700 1 Processor -- start 2023-10-06T03:31:00.170 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:00.164+0000 7f95dc9f9700 1 -- start start 2023-10-06T03:31:00.171 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:00.164+0000 7f95dc9f9700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f95d8155a70 0x7f95d80ccb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:31:00.171 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:00.164+0000 7f95dc9f9700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95d80b07b0 con 0x7f95d8155e40 2023-10-06T03:31:00.172 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:00.164+0000 7f95dc9f9700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95d80afba0 con 0x7f95d80af210 2023-10-06T03:31:00.172 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:00.164+0000 7f95dc9f9700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f95d80af9c0 con 0x7f95d8155a70 2023-10-06T03:31:00.172 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:00.164+0000 7f95d659c700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f95d8155a70 0x7f95d80ccb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:31:00.172 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:00.168+0000 7f95d659c700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f95d8155a70 0x7f95d80ccb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:51240/0 (socket says 172.21.15.110:51240) 2023-10-06T03:31:00.172 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:00.168+0000 7f95d659c700 1 -- 172.21.15.110:0/1819998357 learned_addr learned my addr 172.21.15.110:0/1819998357 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:31:00.172 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:00.168+0000 7f95d5d9b700 1 --1- 172.21.15.110:0/1819998357 >> v1:172.21.15.110:6789/0 conn(0x7f95d8155e40 0x7f95d80aeb00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:40076/0 (socket says 172.21.15.110:40076) 2023-10-06T03:31:00.174 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:00.168+0000 7f95d659c700 1 -- 172.21.15.110:0/1819998357 >> v1:172.21.15.192:6789/0 conn(0x7f95d80af210 legacy=0x7f95d80b0c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:31:00.174 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:00.168+0000 7f95d659c700 1 -- 172.21.15.110:0/1819998357 >> v1:172.21.15.110:6789/0 conn(0x7f95d8155e40 legacy=0x7f95d80aeb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:31:00.174 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:00.168+0000 7f95d659c700 1 -- 172.21.15.110:0/1819998357 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95d80afe80 con 0x7f95d8155a70 2023-10-06T03:31:00.174 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:00.168+0000 7f95d559a700 1 -- 172.21.15.110:0/1819998357 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3975771929 0 0) 0x7f95d80b07b0 con 0x7f95d8155e40 2023-10-06T03:31:00.174 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:00.168+0000 7f95d659c700 1 --2- 172.21.15.110:0/1819998357 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f95d8155a70 0x7f95d80ccb80 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f95c80099e0 tx=0x7f95c8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f757639e3740b72a server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:31:00.174 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:00.168+0000 7f95d559a700 1 -- 172.21.15.110:0/1819998357 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95c800b820 con 0x7f95d8155a70 2023-10-06T03:31:00.174 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:00.168+0000 7f95d559a700 1 -- 172.21.15.110:0/1819998357 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f95c800b980 con 0x7f95d8155a70 2023-10-06T03:31:00.174 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:00.168+0000 7f95d559a700 1 -- 172.21.15.110:0/1819998357 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95c800bd20 con 0x7f95d8155a70 2023-10-06T03:31:00.175 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:00.168+0000 7f95dc9f9700 1 -- 172.21.15.110:0/1819998357 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f95d8155a70 msgr2=0x7f95d80ccb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:31:00.175 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:00.168+0000 7f95dc9f9700 1 --2- 172.21.15.110:0/1819998357 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f95d8155a70 0x7f95d80ccb80 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f95c80099e0 tx=0x7f95c8009250 comp rx=0 tx=0).stop 2023-10-06T03:31:00.175 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:00.168+0000 7f95dc9f9700 1 -- 172.21.15.110:0/1819998357 shutdown_connections 2023-10-06T03:31:00.175 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:00.168+0000 7f95dc9f9700 1 --2- 172.21.15.110:0/1819998357 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f95d8155a70 0x7f95d80ccb80 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:00.175 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:00.168+0000 7f95dc9f9700 1 -- 172.21.15.110:0/1819998357 >> 172.21.15.110:0/1819998357 conn(0x7f95d806b5f0 msgr2=0x7f95d806b9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:31:00.175 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:00.168+0000 7f95dc9f9700 1 -- 172.21.15.110:0/1819998357 shutdown_connections 2023-10-06T03:31:00.175 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:00.168+0000 7f95dc9f9700 1 -- 172.21.15.110:0/1819998357 wait complete. 2023-10-06T03:31:00.176 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:00.172+0000 7f95dc9f9700 1 Processor -- start 2023-10-06T03:31:00.176 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:00.172+0000 7f95dc9f9700 1 -- start start 2023-10-06T03:31:00.178 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:00.172+0000 7f95dc9f9700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f95d80af210 0x7f95d80c8280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:31:00.178 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:00.172+0000 7f95dc9f9700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f95d8155a70 0x7f95d8153200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:31:00.178 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:00.172+0000 7f95dc9f9700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f95d8155e40 0x7f95d80b1e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:31:00.178 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:00.172+0000 7f95dc9f9700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95d80c7df0 con 0x7f95d8155a70 2023-10-06T03:31:00.179 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:00.172+0000 7f95dc9f9700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95d80c71e0 con 0x7f95d80af210 2023-10-06T03:31:00.179 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:00.172+0000 7f95dc9f9700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95d80c74c0 con 0x7f95d8155e40 2023-10-06T03:31:00.179 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:00.172+0000 7f95d5d9b700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f95d8155a70 0x7f95d8153200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:31:00.179 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:00.172+0000 7f95d5d9b700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f95d8155a70 0x7f95d8153200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:51254/0 (socket says 172.21.15.110:51254) 2023-10-06T03:31:00.179 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:00.172+0000 7f95d5d9b700 1 -- 172.21.15.110:0/1291359830 learned_addr learned my addr 172.21.15.110:0/1291359830 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:31:00.179 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:00.172+0000 7f95d659c700 1 --2- 172.21.15.110:0/1291359830 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f95d80af210 0x7f95d80c8280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:31:00.179 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:00.172+0000 7f95d6d9d700 1 --2- 172.21.15.110:0/1291359830 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f95d8155e40 0x7f95d80b1e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:31:00.179 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:00.172+0000 7f95d5d9b700 1 -- 172.21.15.110:0/1291359830 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f95d8155e40 msgr2=0x7f95d80b1e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:31:00.179 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:00.172+0000 7f95d5d9b700 1 --2- 172.21.15.110:0/1291359830 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f95d8155e40 0x7f95d80b1e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:00.179 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:00.172+0000 7f95d5d9b700 1 -- 172.21.15.110:0/1291359830 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f95d80af210 msgr2=0x7f95d80c8280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:31:00.179 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:00.172+0000 7f95d5d9b700 1 --2- 172.21.15.110:0/1291359830 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f95d80af210 0x7f95d80c8280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:00.180 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:00.172+0000 7f95d5d9b700 1 -- 172.21.15.110:0/1291359830 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95c80094f0 con 0x7f95d8155a70 2023-10-06T03:31:00.180 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:00.172+0000 7f95d659c700 1 --2- 172.21.15.110:0/1291359830 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f95d80af210 0x7f95d80c8280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-06T03:31:00.180 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:00.172+0000 7f95d5d9b700 1 --2- 172.21.15.110:0/1291359830 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f95d8155a70 0x7f95d8153200 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f95cc00c560 tx=0x7f95cc007f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:31:00.180 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:00.172+0000 7f95c67fc700 1 -- 172.21.15.110:0/1291359830 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95cc005a70 con 0x7f95d8155a70 2023-10-06T03:31:00.180 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:00.172+0000 7f95c67fc700 1 -- 172.21.15.110:0/1291359830 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f95cc005bd0 con 0x7f95d8155a70 2023-10-06T03:31:00.180 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:00.172+0000 7f95c67fc700 1 -- 172.21.15.110:0/1291359830 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95cc0070d0 con 0x7f95d8155a70 2023-10-06T03:31:00.180 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:00.172+0000 7f95dc9f9700 1 -- 172.21.15.110:0/1291359830 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95d80c77a0 con 0x7f95d8155a70 2023-10-06T03:31:00.180 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:00.172+0000 7f95dc9f9700 1 -- 172.21.15.110:0/1291359830 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f95d8153740 con 0x7f95d8155a70 2023-10-06T03:31:00.181 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:00.172+0000 7f95dc9f9700 1 -- 172.21.15.110:0/1291359830 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f95d80c7fa0 con 0x7f95d8155a70 2023-10-06T03:31:00.181 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:00.176+0000 7f95c67fc700 1 -- 172.21.15.110:0/1291359830 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 10) v1 ==== 80118+0+0 (secure 0 0 0) 0x7f95cc0074c0 con 0x7f95d8155a70 2023-10-06T03:31:00.181 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:00.176+0000 7f95c67fc700 1 --2- 172.21.15.110:0/1291359830 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f95b8060700 0x7f95b8062bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:31:00.181 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:00.176+0000 7f95d659c700 1 -- 172.21.15.110:0/1291359830 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f95b8060700 msgr2=0x7f95b8062bb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.110:6832/33268 2023-10-06T03:31:00.182 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:00.176+0000 7f95c67fc700 1 -- 172.21.15.110:0/1291359830 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 6636+0+0 (secure 0 0 0) 0x7f95cc07c3b0 con 0x7f95d8155a70 2023-10-06T03:31:00.182 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:00.176+0000 7f95d659c700 1 --2- 172.21.15.110:0/1291359830 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f95b8060700 0x7f95b8062bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-06T03:31:00.188 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:00.184+0000 7f95c67fc700 1 -- 172.21.15.110:0/1291359830 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f95d80c7fa0 con 0x7f95d8155a70 2023-10-06T03:31:00.272 INFO:tasks.ceph.mgr.x.smithi192.stderr:2023-10-06T03:31:00.268+0000 7f76034c8000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-06T03:31:00.356 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:00.352+0000 7f95dc9f9700 1 -- 172.21.15.110:0/1291359830 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f95d8064770 con 0x7f95d8155a70 2023-10-06T03:31:00.357 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:00.352+0000 7f95c67fc700 1 -- 172.21.15.110:0/1291359830 <== mon.0 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v43) v1 ==== 100+0+11 (secure 0 0 0) 0x7f95cc0170a0 con 0x7f95d8155a70 2023-10-06T03:31:00.357 INFO:teuthology.orchestra.run.smithi110.stdout:pg_num: 64 2023-10-06T03:31:00.360 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:00.356+0000 7f95dc9f9700 1 -- 172.21.15.110:0/1291359830 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f95b8060700 msgr2=0x7f95b8062bb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-06T03:31:00.361 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:00.356+0000 7f95dc9f9700 1 --2- 172.21.15.110:0/1291359830 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f95b8060700 0x7f95b8062bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:00.361 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:00.356+0000 7f95dc9f9700 1 -- 172.21.15.110:0/1291359830 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f95d8155a70 msgr2=0x7f95d8153200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:31:00.361 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:00.356+0000 7f95dc9f9700 1 --2- 172.21.15.110:0/1291359830 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f95d8155a70 0x7f95d8153200 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f95cc00c560 tx=0x7f95cc007f60 comp rx=0 tx=0).stop 2023-10-06T03:31:00.361 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:00.356+0000 7f95dc9f9700 1 -- 172.21.15.110:0/1291359830 shutdown_connections 2023-10-06T03:31:00.361 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:00.356+0000 7f95dc9f9700 1 --2- 172.21.15.110:0/1291359830 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f95b8060700 0x7f95b8062bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:00.361 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:00.356+0000 7f95dc9f9700 1 --2- 172.21.15.110:0/1291359830 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f95d8155e40 0x7f95d80b1e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:00.361 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:00.356+0000 7f95dc9f9700 1 --2- 172.21.15.110:0/1291359830 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f95d8155a70 0x7f95d8153200 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:00.361 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:00.356+0000 7f95dc9f9700 1 --2- 172.21.15.110:0/1291359830 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f95d80af210 0x7f95d80c8280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:00.361 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:00.356+0000 7f95dc9f9700 1 -- 172.21.15.110:0/1291359830 >> 172.21.15.110:0/1291359830 conn(0x7f95d806b5f0 msgr2=0x7f95d8153b50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:31:00.362 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:00.356+0000 7f95dc9f9700 1 -- 172.21.15.110:0/1291359830 shutdown_connections 2023-10-06T03:31:00.362 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:00.356+0000 7f95dc9f9700 1 -- 172.21.15.110:0/1291359830 wait complete. 2023-10-06T03:31:00.375 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-06T03:31:00.435 INFO:tasks.ceph.mgr.x.smithi192.stderr:2023-10-06T03:31:00.432+0000 7f76034c8000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-06T03:31:00.464 INFO:tasks.ceph.mgr.y.smithi110.stderr:2023-10-06T03:31:00.460+0000 7ff496339000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-06T03:31:00.472 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:00.468+0000 7f8e43833700 1 Processor -- start 2023-10-06T03:31:00.472 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:00.468+0000 7f8e43833700 1 -- start start 2023-10-06T03:31:00.473 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:00.468+0000 7f8e43833700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8e3c156b60 0x7f8e3c0b3680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:31:00.473 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:00.468+0000 7f8e43833700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f8e3c156f30 0x7f8e3c15a930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:31:00.474 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:00.468+0000 7f8e43833700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e3c0ce470 con 0x7f8e3c15ae70 2023-10-06T03:31:00.474 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:00.468+0000 7f8e43833700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8e3c15b310 con 0x7f8e3c156f30 2023-10-06T03:31:00.474 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:00.468+0000 7f8e43833700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e3c0cd860 con 0x7f8e3c156b60 2023-10-06T03:31:00.474 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:00.468+0000 7f8e41dd0700 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f8e3c15ae70 0x7f8e3c0b0af0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:40090/0 (socket says 172.21.15.110:40090) 2023-10-06T03:31:00.474 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:00.468+0000 7f8e41dd0700 1 -- 172.21.15.110:0/4126222938 learned_addr learned my addr 172.21.15.110:0/4126222938 (peer_addr_for_me v1:172.21.15.110:0/0) 2023-10-06T03:31:00.474 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:00.468+0000 7f8e415cf700 1 --2- 172.21.15.110:0/4126222938 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8e3c156b60 0x7f8e3c0b3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:31:00.474 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:00.468+0000 7f8e40dce700 1 --2- 172.21.15.110:0/4126222938 >> v2:172.21.15.192:3300/0 conn(0x7f8e3c156f30 0x7f8e3c15a930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:31:00.474 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:00.468+0000 7f8e33fff700 1 -- 172.21.15.110:0/4126222938 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1721052359 0 0) 0x7f8e3c0ce470 con 0x7f8e3c15ae70 2023-10-06T03:31:00.474 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:00.468+0000 7f8e33fff700 1 -- 172.21.15.110:0/4126222938 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8e28003570 con 0x7f8e3c15ae70 2023-10-06T03:31:00.476 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:00.468+0000 7f8e33fff700 1 -- 172.21.15.110:0/4126222938 <== mon.0 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 477290702 0 0) 0x7f8e28003570 con 0x7f8e3c15ae70 2023-10-06T03:31:00.477 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:00.468+0000 7f8e33fff700 1 -- 172.21.15.110:0/4126222938 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8e3c156b60 msgr2=0x7f8e3c0b3680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:31:00.477 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:00.468+0000 7f8e33fff700 1 --2- 172.21.15.110:0/4126222938 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8e3c156b60 0x7f8e3c0b3680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:00.477 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:00.468+0000 7f8e33fff700 1 -- 172.21.15.110:0/4126222938 >> v2:172.21.15.192:3300/0 conn(0x7f8e3c156f30 msgr2=0x7f8e3c15a930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:31:00.477 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:00.468+0000 7f8e33fff700 1 --2- 172.21.15.110:0/4126222938 >> v2:172.21.15.192:3300/0 conn(0x7f8e3c156f30 0x7f8e3c15a930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:00.477 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:00.468+0000 7f8e33fff700 1 -- 172.21.15.110:0/4126222938 --> v1:172.21.15.110:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e3c0cdb40 con 0x7f8e3c15ae70 2023-10-06T03:31:00.477 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:00.468+0000 7f8e33fff700 1 -- 172.21.15.110:0/4126222938 <== mon.0 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 118603703 0 0) 0x7f8e340017f0 con 0x7f8e3c15ae70 2023-10-06T03:31:00.477 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:00.468+0000 7f8e33fff700 1 -- 172.21.15.110:0/4126222938 >> v1:172.21.15.110:6789/0 conn(0x7f8e3c15ae70 legacy=0x7f8e3c0b0af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:31:00.477 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:00.468+0000 7f8e33fff700 1 --2- 172.21.15.110:0/4126222938 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8e28003a60 0x7f8e28003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:31:00.477 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:00.468+0000 7f8e33fff700 1 --2- 172.21.15.110:0/4126222938 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8e3c156b60 0x7f8e28004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:31:00.477 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:00.468+0000 7f8e33fff700 1 --2- 172.21.15.110:0/4126222938 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8e28004d50 0x7f8e280051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:31:00.477 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:00.468+0000 7f8e33fff700 1 -- 172.21.15.110:0/4126222938 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e3c15b310 con 0x7f8e28004d50 2023-10-06T03:31:00.478 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:00.468+0000 7f8e33fff700 1 -- 172.21.15.110:0/4126222938 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e3c0cd860 con 0x7f8e28003a60 2023-10-06T03:31:00.478 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:00.468+0000 7f8e33fff700 1 -- 172.21.15.110:0/4126222938 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e2800bae0 con 0x7f8e3c156b60 2023-10-06T03:31:00.478 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:00.468+0000 7f8e41dd0700 1 --2- 172.21.15.110:0/4126222938 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8e28004d50 0x7f8e280051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:31:00.478 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:00.468+0000 7f8e415cf700 1 --2- 172.21.15.110:0/4126222938 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8e28003a60 0x7f8e28003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:31:00.478 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:00.468+0000 7f8e40dce700 1 --2- 172.21.15.110:0/4126222938 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8e3c156b60 0x7f8e28004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:31:00.478 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:00.468+0000 7f8e41dd0700 1 -- 172.21.15.110:0/4126222938 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8e3c156b60 msgr2=0x7f8e28004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:31:00.478 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:00.468+0000 7f8e41dd0700 1 --2- 172.21.15.110:0/4126222938 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8e3c156b60 0x7f8e28004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:00.478 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:00.468+0000 7f8e41dd0700 1 -- 172.21.15.110:0/4126222938 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8e28003a60 msgr2=0x7f8e28003e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:31:00.478 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:00.468+0000 7f8e41dd0700 1 --2- 172.21.15.110:0/4126222938 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8e28003a60 0x7f8e28003e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:00.478 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:00.468+0000 7f8e41dd0700 1 -- 172.21.15.110:0/4126222938 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e28003570 con 0x7f8e28004d50 2023-10-06T03:31:00.479 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:00.468+0000 7f8e41dd0700 1 --2- 172.21.15.110:0/4126222938 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8e28004d50 0x7f8e280051c0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f8e3400a970 tx=0x7f8e34002210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=31f2b6f5a3c939ff server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:31:00.479 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:00.468+0000 7f8e33fff700 1 -- 172.21.15.110:0/4126222938 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e34012440 con 0x7f8e28004d50 2023-10-06T03:31:00.479 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:00.472+0000 7f8e33fff700 1 -- 172.21.15.110:0/4126222938 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e340125c0 con 0x7f8e28004d50 2023-10-06T03:31:00.479 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:00.472+0000 7f8e33fff700 1 -- 172.21.15.110:0/4126222938 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e34012980 con 0x7f8e28004d50 2023-10-06T03:31:00.479 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:00.472+0000 7f8e43833700 1 -- 172.21.15.110:0/4126222938 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8e28004d50 msgr2=0x7f8e280051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:31:00.479 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:00.472+0000 7f8e43833700 1 --2- 172.21.15.110:0/4126222938 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8e28004d50 0x7f8e280051c0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f8e3400a970 tx=0x7f8e34002210 comp rx=0 tx=0).stop 2023-10-06T03:31:00.479 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:00.472+0000 7f8e41dd0700 1 -- 172.21.15.110:0/4126222938 reap_dead start 2023-10-06T03:31:00.479 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:00.472+0000 7f8e43833700 1 -- 172.21.15.110:0/4126222938 shutdown_connections 2023-10-06T03:31:00.479 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:00.472+0000 7f8e43833700 1 -- 172.21.15.110:0/4126222938 >> 172.21.15.110:0/4126222938 conn(0x7f8e3c069cd0 msgr2=0x7f8e3c06c760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:31:00.481 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:00.472+0000 7f8e43833700 1 -- 172.21.15.110:0/4126222938 shutdown_connections 2023-10-06T03:31:00.481 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:00.472+0000 7f8e43833700 1 -- 172.21.15.110:0/4126222938 wait complete. 2023-10-06T03:31:00.484 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:00.472+0000 7f8e43833700 1 Processor -- start 2023-10-06T03:31:00.484 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:00.472+0000 7f8e43833700 1 -- start start 2023-10-06T03:31:00.484 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:00.472+0000 7f8e43833700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8e3c147970 0x7f8e3c153820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:31:00.484 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:00.472+0000 7f8e43833700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8e3c166ec0 0x7f8e3c149530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:31:00.485 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:00.472+0000 7f8e43833700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8e3c0af0e0 0x7f8e3c0af560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:31:00.485 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:00.476+0000 7f8e43833700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e3c167eb0 con 0x7f8e3c166ec0 2023-10-06T03:31:00.485 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:00.476+0000 7f8e43833700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e3c0afaa0 con 0x7f8e3c0af0e0 2023-10-06T03:31:00.485 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:00.476+0000 7f8e43833700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e3c149a70 con 0x7f8e3c147970 2023-10-06T03:31:00.485 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:00.476+0000 7f8e40dce700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8e3c166ec0 0x7f8e3c149530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:31:00.485 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:00.476+0000 7f8e415cf700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8e3c147970 0x7f8e3c153820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:31:00.485 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:00.476+0000 7f8e40dce700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8e3c166ec0 0x7f8e3c149530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:51278/0 (socket says 172.21.15.110:51278) 2023-10-06T03:31:00.485 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:00.476+0000 7f8e40dce700 1 -- 172.21.15.110:0/2586360745 learned_addr learned my addr 172.21.15.110:0/2586360745 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:31:00.485 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:00.476+0000 7f8e41dd0700 1 --2- 172.21.15.110:0/2586360745 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8e3c0af0e0 0x7f8e3c0af560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:31:00.485 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:00.476+0000 7f8e40dce700 1 -- 172.21.15.110:0/2586360745 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8e3c147970 msgr2=0x7f8e3c153820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:31:00.486 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:00.476+0000 7f8e40dce700 1 --2- 172.21.15.110:0/2586360745 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8e3c147970 0x7f8e3c153820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:00.486 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:00.476+0000 7f8e40dce700 1 -- 172.21.15.110:0/2586360745 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8e3c0af0e0 msgr2=0x7f8e3c0af560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:31:00.486 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:00.476+0000 7f8e40dce700 1 --2- 172.21.15.110:0/2586360745 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8e3c0af0e0 0x7f8e3c0af560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:00.486 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:00.476+0000 7f8e40dce700 1 -- 172.21.15.110:0/2586360745 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e3c167610 con 0x7f8e3c166ec0 2023-10-06T03:31:00.486 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:00.476+0000 7f8e40dce700 1 --2- 172.21.15.110:0/2586360745 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8e3c166ec0 0x7f8e3c149530 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f8e2c011750 tx=0x7f8e2c008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:31:00.486 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:00.476+0000 7f8e317fa700 1 -- 172.21.15.110:0/2586360745 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e2c015670 con 0x7f8e3c166ec0 2023-10-06T03:31:00.486 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:00.476+0000 7f8e317fa700 1 -- 172.21.15.110:0/2586360745 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e2c0157d0 con 0x7f8e3c166ec0 2023-10-06T03:31:00.486 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:00.476+0000 7f8e43833700 1 -- 172.21.15.110:0/2586360745 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e3c1678f0 con 0x7f8e3c166ec0 2023-10-06T03:31:00.486 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:00.476+0000 7f8e317fa700 1 -- 172.21.15.110:0/2586360745 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e2c015b90 con 0x7f8e3c166ec0 2023-10-06T03:31:00.486 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:00.476+0000 7f8e43833700 1 -- 172.21.15.110:0/2586360745 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e3c154530 con 0x7f8e3c166ec0 2023-10-06T03:31:00.486 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:00.476+0000 7f8e43833700 1 -- 172.21.15.110:0/2586360745 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e3c064770 con 0x7f8e3c166ec0 2023-10-06T03:31:00.487 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:00.476+0000 7f8e317fa700 1 -- 172.21.15.110:0/2586360745 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 10) v1 ==== 80118+0+0 (secure 0 0 0) 0x7f8e2c013410 con 0x7f8e3c166ec0 2023-10-06T03:31:00.487 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:00.476+0000 7f8e317fa700 1 --2- 172.21.15.110:0/2586360745 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f8e20060700 0x7f8e20062bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:31:00.487 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:00.476+0000 7f8e415cf700 1 -- 172.21.15.110:0/2586360745 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f8e20060700 msgr2=0x7f8e20062bb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.110:6832/33268 2023-10-06T03:31:00.487 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:00.476+0000 7f8e317fa700 1 -- 172.21.15.110:0/2586360745 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 6636+0+0 (secure 0 0 0) 0x7f8e2c07d0a0 con 0x7f8e3c166ec0 2023-10-06T03:31:00.487 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:00.476+0000 7f8e415cf700 1 --2- 172.21.15.110:0/2586360745 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f8e20060700 0x7f8e20062bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-06T03:31:00.488 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:00.484+0000 7f8e317fa700 1 -- 172.21.15.110:0/2586360745 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f8e2c04d070 con 0x7f8e3c166ec0 2023-10-06T03:31:00.499 INFO:tasks.ceph.mgr.x.smithi192.stderr:2023-10-06T03:31:00.496+0000 7f76034c8000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-06T03:31:00.552 INFO:tasks.ceph.mgr.x.smithi192.stderr:2023-10-06T03:31:00.548+0000 7f76034c8000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-06T03:31:00.605 INFO:tasks.ceph.mgr.x.smithi192.stderr:2023-10-06T03:31:00.600+0000 7f76034c8000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-06T03:31:00.629 INFO:tasks.ceph.mgr.y.smithi110.stderr:2023-10-06T03:31:00.624+0000 7ff496339000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-06T03:31:00.682 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:00.676+0000 7f8e43833700 1 -- 172.21.15.110:0/2586360745 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8e3c06a470 con 0x7f8e3c166ec0 2023-10-06T03:31:00.683 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:00.676+0000 7f8e317fa700 1 -- 172.21.15.110:0/2586360745 <== mon.0 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) v1 ==== 94+0+4603 (secure 0 0 0) 0x7f8e2c050690 con 0x7f8e3c166ec0 2023-10-06T03:31:00.683 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-10-06T03:31:00.683 INFO:teuthology.orchestra.run.smithi110.stdout:{"epoch":16,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5107,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6834/884349347","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6833","nonce":884349347},{"type":"v1","addr":"172.21.15.192:6834","nonce":884349347}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":5110,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6835/4077204152","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6832","nonce":4077204152},{"type":"v1","addr":"172.21.15.192:6835","nonce":4077204152}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":5160,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/504583843","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6834","nonce":504583843},{"type":"v1","addr":"172.21.15.110:6835","nonce":504583843}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-06T03:30:34.014644+0000","modified":"2023-10-06T03:30:37.662700+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5163},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5163":{"gid":5163,"name":"a","rank":0,"incarnation":13,"state":"up:active","state_seq":4,"addr":"172.21.15.110:6837/1379510962","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6836","nonce":1379510962},{"type":"v1","addr":"172.21.15.110:6837","nonce":1379510962}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":2}]} 2023-10-06T03:31:00.684 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:00.680+0000 7f8e415cf700 1 -- 172.21.15.110:0/2586360745 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f8e20060700 msgr2=0x7f8e20062bb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.110:6832/33268 2023-10-06T03:31:00.684 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:00.680+0000 7f8e415cf700 1 --2- 172.21.15.110:0/2586360745 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f8e20060700 0x7f8e20062bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-06T03:31:00.685 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:00.680+0000 7f8e43833700 1 -- 172.21.15.110:0/2586360745 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f8e20060700 msgr2=0x7f8e20062bb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-06T03:31:00.685 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:00.680+0000 7f8e43833700 1 --2- 172.21.15.110:0/2586360745 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f8e20060700 0x7f8e20062bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:00.685 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:00.680+0000 7f8e43833700 1 -- 172.21.15.110:0/2586360745 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8e3c166ec0 msgr2=0x7f8e3c149530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:31:00.685 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:00.680+0000 7f8e43833700 1 --2- 172.21.15.110:0/2586360745 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8e3c166ec0 0x7f8e3c149530 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f8e2c011750 tx=0x7f8e2c008040 comp rx=0 tx=0).stop 2023-10-06T03:31:00.685 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:00.680+0000 7f8e43833700 1 -- 172.21.15.110:0/2586360745 shutdown_connections 2023-10-06T03:31:00.685 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:00.680+0000 7f8e43833700 1 --2- 172.21.15.110:0/2586360745 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f8e20060700 0x7f8e20062bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:00.685 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:00.680+0000 7f8e43833700 1 --2- 172.21.15.110:0/2586360745 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8e3c0af0e0 0x7f8e3c0af560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:00.685 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:00.680+0000 7f8e43833700 1 --2- 172.21.15.110:0/2586360745 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8e3c166ec0 0x7f8e3c149530 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:00.686 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:00.680+0000 7f8e43833700 1 --2- 172.21.15.110:0/2586360745 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8e3c147970 0x7f8e3c153820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:00.686 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:00.680+0000 7f8e43833700 1 -- 172.21.15.110:0/2586360745 >> 172.21.15.110:0/2586360745 conn(0x7f8e3c069cd0 msgr2=0x7f8e3c157300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:31:00.686 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:00.680+0000 7f8e43833700 1 -- 172.21.15.110:0/2586360745 shutdown_connections 2023-10-06T03:31:00.686 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:00.680+0000 7f8e43833700 1 -- 172.21.15.110:0/2586360745 wait complete. 2023-10-06T03:31:00.686 INFO:teuthology.orchestra.run.smithi110.stderr:dumped fsmap epoch 16 2023-10-06T03:31:00.694 INFO:tasks.ceph.mgr.y.smithi110.stderr:2023-10-06T03:31:00.688+0000 7ff496339000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-06T03:31:00.696 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-06T03:31:00.746 INFO:tasks.ceph.mgr.y.smithi110.stderr:2023-10-06T03:31:00.740+0000 7ff496339000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-06T03:31:00.780 INFO:tasks.ceph.mgr.x.smithi192.stderr:2023-10-06T03:31:00.776+0000 7f76034c8000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-06T03:31:00.794 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:00.788+0000 7f3e46aa4700 1 Processor -- start 2023-10-06T03:31:00.794 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:00.788+0000 7f3e46aa4700 1 -- start start 2023-10-06T03:31:00.796 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:00.788+0000 7f3e46aa4700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f3e40156ff0 0x7f3e400c8400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:31:00.796 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:00.788+0000 7f3e46aa4700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f3e400c8940 0x7f3e4006a7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:31:00.796 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:00.788+0000 7f3e46aa4700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e400abc40 con 0x7f3e40156c20 2023-10-06T03:31:00.796 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:00.788+0000 7f3e46aa4700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3e400abf20 con 0x7f3e400c8940 2023-10-06T03:31:00.796 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:00.788+0000 7f3e46aa4700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3e400ac200 con 0x7f3e40156ff0 2023-10-06T03:31:00.796 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:00.788+0000 7f3e44840700 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f3e40156c20 0x7f3e400add50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:40100/0 (socket says 172.21.15.110:40100) 2023-10-06T03:31:00.797 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:00.788+0000 7f3e45041700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f3e400c8940 0x7f3e4006a7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:31:00.797 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:00.788+0000 7f3e44840700 1 -- 172.21.15.110:0/2762703078 learned_addr learned my addr 172.21.15.110:0/2762703078 (peer_addr_for_me v1:172.21.15.110:0/0) 2023-10-06T03:31:00.797 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:00.788+0000 7f3e3ffff700 1 --2- 172.21.15.110:0/2762703078 >> v2:172.21.15.192:3300/0 conn(0x7f3e40156ff0 0x7f3e400c8400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:31:00.797 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:00.792+0000 7f3e45041700 1 -- 172.21.15.110:0/2762703078 >> v2:172.21.15.192:3300/0 conn(0x7f3e40156ff0 msgr2=0x7f3e400c8400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:31:00.797 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:00.792+0000 7f3e45041700 1 --2- 172.21.15.110:0/2762703078 >> v2:172.21.15.192:3300/0 conn(0x7f3e40156ff0 0x7f3e400c8400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:00.797 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:00.792+0000 7f3e45041700 1 -- 172.21.15.110:0/2762703078 >> v1:172.21.15.110:6789/0 conn(0x7f3e40156c20 legacy=0x7f3e400add50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:31:00.797 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:00.792+0000 7f3e45041700 1 -- 172.21.15.110:0/2762703078 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e400ac4e0 con 0x7f3e400c8940 2023-10-06T03:31:00.798 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:00.792+0000 7f3e3f7fe700 1 -- 172.21.15.110:0/2762703078 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3683952377 0 0) 0x7f3e400abc40 con 0x7f3e40156c20 2023-10-06T03:31:00.798 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:00.792+0000 7f3e45041700 1 --2- 172.21.15.110:0/2762703078 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3e400c8940 0x7f3e4006a7c0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f3e4006da70 tx=0x7f3e38009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d80cedc09169a93f server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:31:00.798 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:00.792+0000 7f3e3f7fe700 1 -- 172.21.15.110:0/2762703078 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e3800e070 con 0x7f3e400c8940 2023-10-06T03:31:00.798 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:00.792+0000 7f3e3f7fe700 1 -- 172.21.15.110:0/2762703078 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e3800bb60 con 0x7f3e400c8940 2023-10-06T03:31:00.798 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:00.792+0000 7f3e3f7fe700 1 -- 172.21.15.110:0/2762703078 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e38012430 con 0x7f3e400c8940 2023-10-06T03:31:00.801 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:00.792+0000 7f3e46aa4700 1 -- 172.21.15.110:0/2762703078 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3e400c8940 msgr2=0x7f3e4006a7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:31:00.801 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:00.792+0000 7f3e46aa4700 1 --2- 172.21.15.110:0/2762703078 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3e400c8940 0x7f3e4006a7c0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f3e4006da70 tx=0x7f3e38009780 comp rx=0 tx=0).stop 2023-10-06T03:31:00.801 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:00.792+0000 7f3e46aa4700 1 -- 172.21.15.110:0/2762703078 shutdown_connections 2023-10-06T03:31:00.801 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:00.792+0000 7f3e46aa4700 1 --2- 172.21.15.110:0/2762703078 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3e400c8940 0x7f3e4006a7c0 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:00.802 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:00.792+0000 7f3e46aa4700 1 --2- 172.21.15.110:0/2762703078 >> v2:172.21.15.192:3300/0 conn(0x7f3e40156ff0 0x7f3e400c8400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:00.802 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:00.792+0000 7f3e46aa4700 1 -- 172.21.15.110:0/2762703078 >> 172.21.15.110:0/2762703078 conn(0x7f3e40067040 msgr2=0x7f3e40067410 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:31:00.802 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:00.792+0000 7f3e46aa4700 1 -- 172.21.15.110:0/2762703078 shutdown_connections 2023-10-06T03:31:00.802 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:00.792+0000 7f3e46aa4700 1 -- 172.21.15.110:0/2762703078 wait complete. 2023-10-06T03:31:00.802 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:00.792+0000 7f3e46aa4700 1 Processor -- start 2023-10-06T03:31:00.802 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:00.792+0000 7f3e46aa4700 1 -- start start 2023-10-06T03:31:00.802 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:00.792+0000 7f3e46aa4700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3e400c8940 0x7f3e4014c380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:31:00.802 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:00.792+0000 7f3e46aa4700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3e40156c20 0x7f3e400bffb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:31:00.803 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:00.792+0000 7f3e46aa4700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3e40156ff0 0x7f3e400c04f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:31:00.803 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:00.792+0000 7f3e46aa4700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e400c17f0 con 0x7f3e400c8940 2023-10-06T03:31:00.803 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:00.792+0000 7f3e46aa4700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e400c4e40 con 0x7f3e40156c20 2023-10-06T03:31:00.803 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:00.792+0000 7f3e46aa4700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e400c0b70 con 0x7f3e40156ff0 2023-10-06T03:31:00.803 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:00.792+0000 7f3e44840700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3e400c8940 0x7f3e4014c380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:31:00.803 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:00.792+0000 7f3e44840700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3e400c8940 0x7f3e4014c380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:51290/0 (socket says 172.21.15.110:51290) 2023-10-06T03:31:00.803 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:00.792+0000 7f3e44840700 1 -- 172.21.15.110:0/1724269401 learned_addr learned my addr 172.21.15.110:0/1724269401 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:31:00.803 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:00.792+0000 7f3e3ffff700 1 --2- 172.21.15.110:0/1724269401 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3e40156c20 0x7f3e400bffb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:31:00.803 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:00.796+0000 7f3e45041700 1 --2- 172.21.15.110:0/1724269401 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3e40156ff0 0x7f3e400c04f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:31:00.804 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:00.796+0000 7f3e44840700 1 -- 172.21.15.110:0/1724269401 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3e40156ff0 msgr2=0x7f3e400c04f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:31:00.804 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:00.796+0000 7f3e44840700 1 --2- 172.21.15.110:0/1724269401 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3e40156ff0 0x7f3e400c04f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:00.804 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:00.796+0000 7f3e44840700 1 -- 172.21.15.110:0/1724269401 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3e40156c20 msgr2=0x7f3e400bffb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:31:00.804 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:00.796+0000 7f3e44840700 1 --2- 172.21.15.110:0/1724269401 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3e40156c20 0x7f3e400bffb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:00.804 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:00.796+0000 7f3e44840700 1 -- 172.21.15.110:0/1724269401 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e38009a20 con 0x7f3e400c8940 2023-10-06T03:31:00.804 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:00.796+0000 7f3e44840700 1 --2- 172.21.15.110:0/1724269401 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3e400c8940 0x7f3e4014c380 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f3e3400c7e0 tx=0x7f3e34007df0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:31:00.804 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:00.796+0000 7f3e3cff9700 1 -- 172.21.15.110:0/1724269401 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e34005b30 con 0x7f3e400c8940 2023-10-06T03:31:00.804 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:00.796+0000 7f3e3cff9700 1 -- 172.21.15.110:0/1724269401 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e34005c90 con 0x7f3e400c8940 2023-10-06T03:31:00.805 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:00.796+0000 7f3e46aa4700 1 -- 172.21.15.110:0/1724269401 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e400c0e50 con 0x7f3e400c8940 2023-10-06T03:31:00.805 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:00.796+0000 7f3e3cff9700 1 -- 172.21.15.110:0/1724269401 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e34007300 con 0x7f3e400c8940 2023-10-06T03:31:00.805 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:00.796+0000 7f3e46aa4700 1 -- 172.21.15.110:0/1724269401 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e400c1410 con 0x7f3e400c8940 2023-10-06T03:31:00.805 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:00.796+0000 7f3e46aa4700 1 -- 172.21.15.110:0/1724269401 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e40156820 con 0x7f3e400c8940 2023-10-06T03:31:00.806 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:00.796+0000 7f3e3cff9700 1 -- 172.21.15.110:0/1724269401 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 10) v1 ==== 80118+0+0 (secure 0 0 0) 0x7f3e3401b3b0 con 0x7f3e400c8940 2023-10-06T03:31:00.806 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:00.796+0000 7f3e3cff9700 1 --2- 172.21.15.110:0/1724269401 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f3e24060700 0x7f3e24062bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:31:00.806 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:00.796+0000 7f3e3ffff700 1 -- 172.21.15.110:0/1724269401 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f3e24060700 msgr2=0x7f3e24062bb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.110:6832/33268 2023-10-06T03:31:00.806 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:00.796+0000 7f3e3cff9700 1 -- 172.21.15.110:0/1724269401 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 6636+0+0 (secure 0 0 0) 0x7f3e3407c2a0 con 0x7f3e400c8940 2023-10-06T03:31:00.806 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:00.796+0000 7f3e3ffff700 1 --2- 172.21.15.110:0/1724269401 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f3e24060700 0x7f3e24062bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-06T03:31:00.806 INFO:tasks.ceph.mgr.y.smithi110.stderr:2023-10-06T03:31:00.796+0000 7ff496339000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-06T03:31:00.809 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:00.804+0000 7f3e3cff9700 1 -- 172.21.15.110:0/1724269401 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f3e40156820 con 0x7f3e400c8940 2023-10-06T03:31:00.833 INFO:tasks.ceph.mgr.x.smithi192.stderr:2023-10-06T03:31:00.828+0000 7f76034c8000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-06T03:31:00.895 INFO:tasks.ceph.mgr.x.smithi192.stderr:2023-10-06T03:31:00.892+0000 7f76034c8000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-06T03:31:00.975 INFO:tasks.ceph.mgr.y.smithi110.stderr:2023-10-06T03:31:00.968+0000 7ff496339000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-06T03:31:00.976 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:00.972+0000 7f3e46aa4700 1 -- 172.21.15.110:0/1724269401 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3e400c1930 con 0x7f3e400c8940 2023-10-06T03:31:00.978 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:00.972+0000 7f3e3cff9700 1 -- 172.21.15.110:0/1724269401 <== mon.0 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v43) v1 ==== 74+0+17491 (secure 0 0 0) 0x7f3e400c1930 con 0x7f3e400c8940 2023-10-06T03:31:00.979 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-10-06T03:31:00.979 INFO:teuthology.orchestra.run.smithi110.stdout:{"epoch":43,"fsid":"93b8f6ac-4c92-4b45-b8e4-b59617024aa2","created":"2023-10-06T03:29:10.519583+0000","modified":"2023-10-06T03:30:46.697260+0000","last_up_change":"2023-10-06T03:29:21.960511+0000","last_in_change":"2023-10-06T03:29:12.560436+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-06T03:29:20.703743+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2023-10-06T03:29:28.307281+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2023-10-06T03:30:31.807897+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"41","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":7,"pool_name":"cephfs_data","create_time":"2023-10-06T03:30:32.991334+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"42","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"5866eb6c-2adc-46e4-be7a-e5bcda0f4065","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6824","nonce":33394},{"type":"v1","addr":"172.21.15.110:6825","nonce":33394}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6826","nonce":33394},{"type":"v1","addr":"172.21.15.110:6827","nonce":33394}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6830","nonce":33394},{"type":"v1","addr":"172.21.15.110:6831","nonce":33394}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6828","nonce":33394},{"type":"v1","addr":"172.21.15.110:6829","nonce":33394}]},"public_addr":"172.21.15.110:6825/33394","cluster_addr":"172.21.15.110:6827/33394","heartbeat_back_addr":"172.21.15.110:6831/33394","heartbeat_front_addr":"172.21.15.110:6829/33394","state":["exists","up"]},{"osd":1,"uuid":"a38de118-062b-4a05-9782-2ceddf6ff4df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6800","nonce":33396},{"type":"v1","addr":"172.21.15.110:6801","nonce":33396}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6802","nonce":33396},{"type":"v1","addr":"172.21.15.110:6803","nonce":33396}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6806","nonce":33396},{"type":"v1","addr":"172.21.15.110:6807","nonce":33396}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6804","nonce":33396},{"type":"v1","addr":"172.21.15.110:6805","nonce":33396}]},"public_addr":"172.21.15.110:6801/33396","cluster_addr":"172.21.15.110:6803/33396","heartbeat_back_addr":"172.21.15.110:6807/33396","heartbeat_front_addr":"172.21.15.110:6805/33396","state":["exists","up"]},{"osd":2,"uuid":"56e39be2-3407-4b69-877e-cf16617b06d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6816","nonce":33397},{"type":"v1","addr":"172.21.15.110:6817","nonce":33397}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6818","nonce":33397},{"type":"v1","addr":"172.21.15.110:6819","nonce":33397}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6822","nonce":33397},{"type":"v1","addr":"172.21.15.110:6823","nonce":33397}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6820","nonce":33397},{"type":"v1","addr":"172.21.15.110:6821","nonce":33397}]},"public_addr":"172.21.15.110:6817/33397","cluster_addr":"172.21.15.110:6819/33397","heartbeat_back_addr":"172.21.15.110:6823/33397","heartbeat_front_addr":"172.21.15.110:6821/33397","state":["exists","up"]},{"osd":3,"uuid":"01c82d13-c1d4-4bde-b55e-3e0089b40115","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6808","nonce":33395},{"type":"v1","addr":"172.21.15.110:6809","nonce":33395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6810","nonce":33395},{"type":"v1","addr":"172.21.15.110:6811","nonce":33395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6814","nonce":33395},{"type":"v1","addr":"172.21.15.110:6815","nonce":33395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6812","nonce":33395},{"type":"v1","addr":"172.21.15.110:6813","nonce":33395}]},"public_addr":"172.21.15.110:6809/33395","cluster_addr":"172.21.15.110:6811/33395","heartbeat_back_addr":"172.21.15.110:6815/33395","heartbeat_front_addr":"172.21.15.110:6813/33395","state":["exists","up"]},{"osd":4,"uuid":"86983430-5fa3-4d17-998f-03e2cd843237","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":33683},{"type":"v1","addr":"172.21.15.192:6817","nonce":33683}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":33683},{"type":"v1","addr":"172.21.15.192:6819","nonce":33683}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":33683},{"type":"v1","addr":"172.21.15.192:6823","nonce":33683}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6820","nonce":33683},{"type":"v1","addr":"172.21.15.192:6821","nonce":33683}]},"public_addr":"172.21.15.192:6817/33683","cluster_addr":"172.21.15.192:6819/33683","heartbeat_back_addr":"172.21.15.192:6823/33683","heartbeat_front_addr":"172.21.15.192:6821/33683","state":["exists","up"]},{"osd":5,"uuid":"6945257f-821d-4619-a63f-ebe857eba6f4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":33684},{"type":"v1","addr":"172.21.15.192:6801","nonce":33684}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":33684},{"type":"v1","addr":"172.21.15.192:6803","nonce":33684}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":33684},{"type":"v1","addr":"172.21.15.192:6807","nonce":33684}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6804","nonce":33684},{"type":"v1","addr":"172.21.15.192:6805","nonce":33684}]},"public_addr":"172.21.15.192:6801/33684","cluster_addr":"172.21.15.192:6803/33684","heartbeat_back_addr":"172.21.15.192:6807/33684","heartbeat_front_addr":"172.21.15.192:6805/33684","state":["exists","up"]},{"osd":6,"uuid":"7854d1ae-7060-4780-8f97-73cea5e7a12f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6824","nonce":33686},{"type":"v1","addr":"172.21.15.192:6825","nonce":33686}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":33686},{"type":"v1","addr":"172.21.15.192:6827","nonce":33686}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":33686},{"type":"v1","addr":"172.21.15.192:6831","nonce":33686}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6828","nonce":33686},{"type":"v1","addr":"172.21.15.192:6829","nonce":33686}]},"public_addr":"172.21.15.192:6825/33686","cluster_addr":"172.21.15.192:6827/33686","heartbeat_back_addr":"172.21.15.192:6831/33686","heartbeat_front_addr":"172.21.15.192:6829/33686","state":["exists","up"]},{"osd":7,"uuid":"f1ebdfd5-b859-482b-9f97-c18c4a26f445","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6808","nonce":33685},{"type":"v1","addr":"172.21.15.192:6809","nonce":33685}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":33685},{"type":"v1","addr":"172.21.15.192:6811","nonce":33685}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":33685},{"type":"v1","addr":"172.21.15.192:6815","nonce":33685}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6812","nonce":33685},{"type":"v1","addr":"172.21.15.192:6813","nonce":33685}]},"public_addr":"172.21.15.192:6809/33685","cluster_addr":"172.21.15.192:6811/33685","heartbeat_back_addr":"172.21.15.192:6815/33685","heartbeat_front_addr":"172.21.15.192:6813/33685","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T03:29:18.002048+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T03:29:17.478576+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T03:29:18.034944+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T03:29:17.628513+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T03:29:18.047338+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T03:29:17.772349+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T03:29:17.883958+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T03:29:18.151838+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.110:6832/33268":"2023-10-07T03:30:46.697197+0000","172.21.15.110:0/463196587":"2023-10-07T03:30:46.697197+0000","172.21.15.110:6833/33268":"2023-10-07T03:30:46.697197+0000","172.21.15.110:0/1139019390":"2023-10-07T03:30:46.697197+0000","172.21.15.110:0/1318624357":"2023-10-07T03:30:46.697197+0000","172.21.15.110:0/1857995858":"2023-10-07T03:30:46.697197+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-06T03:31:00.981 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:00.976+0000 7f3e46aa4700 1 -- 172.21.15.110:0/1724269401 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f3e24060700 msgr2=0x7f3e24062bb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-06T03:31:00.981 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:00.976+0000 7f3e46aa4700 1 --2- 172.21.15.110:0/1724269401 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f3e24060700 0x7f3e24062bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:00.981 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:00.976+0000 7f3e46aa4700 1 -- 172.21.15.110:0/1724269401 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3e400c8940 msgr2=0x7f3e4014c380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:31:00.981 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:00.976+0000 7f3e46aa4700 1 --2- 172.21.15.110:0/1724269401 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3e400c8940 0x7f3e4014c380 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f3e3400c7e0 tx=0x7f3e34007df0 comp rx=0 tx=0).stop 2023-10-06T03:31:00.982 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:00.976+0000 7f3e46aa4700 1 -- 172.21.15.110:0/1724269401 shutdown_connections 2023-10-06T03:31:00.982 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:00.976+0000 7f3e46aa4700 1 --2- 172.21.15.110:0/1724269401 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f3e24060700 0x7f3e24062bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:00.982 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:00.976+0000 7f3e46aa4700 1 --2- 172.21.15.110:0/1724269401 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3e40156ff0 0x7f3e400c04f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:00.982 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:00.976+0000 7f3e46aa4700 1 --2- 172.21.15.110:0/1724269401 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3e40156c20 0x7f3e400bffb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:00.982 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:00.976+0000 7f3e46aa4700 1 --2- 172.21.15.110:0/1724269401 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3e400c8940 0x7f3e4014c380 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:00.982 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:00.976+0000 7f3e46aa4700 1 -- 172.21.15.110:0/1724269401 >> 172.21.15.110:0/1724269401 conn(0x7f3e40067040 msgr2=0x7f3e4014ce70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:31:00.982 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:00.976+0000 7f3e46aa4700 1 -- 172.21.15.110:0/1724269401 shutdown_connections 2023-10-06T03:31:00.983 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:00.976+0000 7f3e46aa4700 1 -- 172.21.15.110:0/1724269401 wait complete. 2023-10-06T03:31:00.993 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2023-10-06T03:31:00.993 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-06T03:31:01.029 INFO:tasks.ceph.mgr.y.smithi110.stderr:2023-10-06T03:31:01.024+0000 7ff496339000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-06T03:31:01.043 INFO:tasks.ceph.mgr.x.smithi192.stderr:2023-10-06T03:31:01.040+0000 7f76034c8000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-06T03:31:01.091 INFO:tasks.ceph.mgr.y.smithi110.stderr:2023-10-06T03:31:01.084+0000 7ff496339000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-06T03:31:01.093 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:01.088+0000 7f9e9b6af700 1 Processor -- start 2023-10-06T03:31:01.093 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:01.088+0000 7f9e9b6af700 1 -- start start 2023-10-06T03:31:01.095 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:01.088+0000 7f9e9b6af700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9e94158ef0 0x7f9e940cb100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:31:01.095 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:01.088+0000 7f9e9b6af700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f9e940d0450 0x7f9e940b3380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:31:01.096 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:01.088+0000 7f9e9b6af700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e940cd2d0 con 0x7f9e941592c0 2023-10-06T03:31:01.096 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:01.088+0000 7f9e9b6af700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e940ce7f0 con 0x7f9e940d0450 2023-10-06T03:31:01.096 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:01.088+0000 7f9e9b6af700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e940cd5b0 con 0x7f9e94158ef0 2023-10-06T03:31:01.096 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:01.088+0000 7f9e99c4c700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f9e940d0450 0x7f9e940b3380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:31:01.096 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:01.088+0000 7f9e9944b700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9e94158ef0 0x7f9e940cb100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:31:01.096 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:01.088+0000 7f9e9944b700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9e94158ef0 0x7f9e940cb100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3301/0 says I am v2:172.21.15.110:54608/0 (socket says 172.21.15.110:54608) 2023-10-06T03:31:01.097 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:01.088+0000 7f9e99c4c700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f9e940d0450 0x7f9e940b3380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.110:54920/0 (socket says 172.21.15.110:54920) 2023-10-06T03:31:01.097 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:01.088+0000 7f9e9944b700 1 -- 172.21.15.110:0/604276400 learned_addr learned my addr 172.21.15.110:0/604276400 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:31:01.097 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:01.088+0000 7f9e9944b700 1 -- 172.21.15.110:0/604276400 >> v2:172.21.15.192:3300/0 conn(0x7f9e940d0450 msgr2=0x7f9e940b3380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:31:01.097 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:01.088+0000 7f9e9944b700 1 --2- 172.21.15.110:0/604276400 >> v2:172.21.15.192:3300/0 conn(0x7f9e940d0450 0x7f9e940b3380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:01.097 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:01.088+0000 7f9e9944b700 1 -- 172.21.15.110:0/604276400 >> v1:172.21.15.192:6789/0 conn(0x7f9e941592c0 legacy=0x7f9e940cfd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:31:01.097 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:01.088+0000 7f9e9944b700 1 -- 172.21.15.110:0/604276400 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e940ca900 con 0x7f9e94158ef0 2023-10-06T03:31:01.097 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:01.088+0000 7f9e99c4c700 1 --2- 172.21.15.110:0/604276400 >> v2:172.21.15.192:3300/0 conn(0x7f9e940d0450 0x7f9e940b3380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-06T03:31:01.097 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:01.088+0000 7f9e9944b700 1 --2- 172.21.15.110:0/604276400 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9e94158ef0 0x7f9e940cb100 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f9e900099b0 tx=0x7f9e90009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9cd5104edcf19040 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:31:01.097 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:01.092+0000 7f9e8bfff700 1 -- 172.21.15.110:0/604276400 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e90009ea0 con 0x7f9e94158ef0 2023-10-06T03:31:01.097 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:01.092+0000 7f9e8bfff700 1 -- 172.21.15.110:0/604276400 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e9000b860 con 0x7f9e94158ef0 2023-10-06T03:31:01.097 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:01.092+0000 7f9e8bfff700 1 -- 172.21.15.110:0/604276400 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e9000bc00 con 0x7f9e94158ef0 2023-10-06T03:31:01.100 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:01.092+0000 7f9e9b6af700 1 -- 172.21.15.110:0/604276400 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9e94158ef0 msgr2=0x7f9e940cb100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:31:01.100 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:01.092+0000 7f9e9b6af700 1 --2- 172.21.15.110:0/604276400 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9e94158ef0 0x7f9e940cb100 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f9e900099b0 tx=0x7f9e90009250 comp rx=0 tx=0).stop 2023-10-06T03:31:01.100 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:01.092+0000 7f9e9b6af700 1 -- 172.21.15.110:0/604276400 shutdown_connections 2023-10-06T03:31:01.100 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:01.092+0000 7f9e9b6af700 1 --2- 172.21.15.110:0/604276400 >> v2:172.21.15.192:3300/0 conn(0x7f9e940d0450 0x7f9e940b3380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:01.100 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:01.092+0000 7f9e9b6af700 1 --2- 172.21.15.110:0/604276400 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9e94158ef0 0x7f9e940cb100 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:01.100 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:01.092+0000 7f9e9b6af700 1 -- 172.21.15.110:0/604276400 >> 172.21.15.110:0/604276400 conn(0x7f9e94067040 msgr2=0x7f9e94067440 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:31:01.100 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:01.092+0000 7f9e9b6af700 1 -- 172.21.15.110:0/604276400 shutdown_connections 2023-10-06T03:31:01.101 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:01.092+0000 7f9e9b6af700 1 -- 172.21.15.110:0/604276400 wait complete. 2023-10-06T03:31:01.101 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:01.092+0000 7f9e9b6af700 1 Processor -- start 2023-10-06T03:31:01.101 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:01.092+0000 7f9e9b6af700 1 -- start start 2023-10-06T03:31:01.101 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:01.092+0000 7f9e9b6af700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9e940d0450 0x7f9e94142250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:31:01.101 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:01.092+0000 7f9e9b6af700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9e94158ef0 0x7f9e940c8620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:31:01.101 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:01.092+0000 7f9e9b6af700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9e941592c0 0x7f9e94156f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:31:01.101 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:01.092+0000 7f9e9b6af700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e940c8b60 con 0x7f9e94158ef0 2023-10-06T03:31:01.101 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:01.092+0000 7f9e9b6af700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e94157ca0 con 0x7f9e940d0450 2023-10-06T03:31:01.101 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:01.092+0000 7f9e9b6af700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e94157f80 con 0x7f9e941592c0 2023-10-06T03:31:01.101 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:01.092+0000 7f9e98c4a700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9e94158ef0 0x7f9e940c8620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:31:01.102 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:01.092+0000 7f9e9944b700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9e940d0450 0x7f9e94142250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:31:01.102 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:01.092+0000 7f9e9944b700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9e940d0450 0x7f9e94142250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.110:54924/0 (socket says 172.21.15.110:54924) 2023-10-06T03:31:01.102 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:01.092+0000 7f9e98c4a700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9e94158ef0 0x7f9e940c8620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:38630/0 (socket says 172.21.15.110:38630) 2023-10-06T03:31:01.102 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:01.092+0000 7f9e99c4c700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9e941592c0 0x7f9e94156f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:31:01.102 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:01.092+0000 7f9e9944b700 1 -- 172.21.15.110:0/2970448444 learned_addr learned my addr 172.21.15.110:0/2970448444 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:31:01.102 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:01.096+0000 7f9e9944b700 1 -- 172.21.15.110:0/2970448444 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9e941592c0 msgr2=0x7f9e94156f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:31:01.102 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:01.096+0000 7f9e9944b700 1 --2- 172.21.15.110:0/2970448444 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9e941592c0 0x7f9e94156f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:01.102 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:01.096+0000 7f9e9944b700 1 -- 172.21.15.110:0/2970448444 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9e94158ef0 msgr2=0x7f9e940c8620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:31:01.102 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:01.096+0000 7f9e9944b700 1 --2- 172.21.15.110:0/2970448444 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9e94158ef0 0x7f9e940c8620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:01.102 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:01.096+0000 7f9e9944b700 1 -- 172.21.15.110:0/2970448444 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e8c009a20 con 0x7f9e940d0450 2023-10-06T03:31:01.103 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:01.096+0000 7f9e9944b700 1 --2- 172.21.15.110:0/2970448444 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9e940d0450 0x7f9e94142250 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f9e940cd6f0 tx=0x7f9e90017000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:31:01.103 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:01.096+0000 7f9e897fa700 1 -- 172.21.15.110:0/2970448444 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e90000d10 con 0x7f9e940d0450 2023-10-06T03:31:01.103 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:01.096+0000 7f9e897fa700 1 -- 172.21.15.110:0/2970448444 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e9000bbe0 con 0x7f9e940d0450 2023-10-06T03:31:01.103 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:01.096+0000 7f9e9b6af700 1 -- 172.21.15.110:0/2970448444 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e900094f0 con 0x7f9e940d0450 2023-10-06T03:31:01.103 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:01.096+0000 7f9e897fa700 1 -- 172.21.15.110:0/2970448444 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e90011670 con 0x7f9e940d0450 2023-10-06T03:31:01.103 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:01.096+0000 7f9e9b6af700 1 -- 172.21.15.110:0/2970448444 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e94158540 con 0x7f9e940d0450 2023-10-06T03:31:01.104 INFO:tasks.ceph.mgr.x.smithi192.stderr:2023-10-06T03:31:01.100+0000 7f76034c8000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-06T03:31:01.104 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:01.096+0000 7f9e9b6af700 1 -- 172.21.15.110:0/2970448444 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e94064770 con 0x7f9e940d0450 2023-10-06T03:31:01.104 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:01.096+0000 7f9e897fa700 1 -- 172.21.15.110:0/2970448444 <== mon.1 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 10) v1 ==== 80118+0+0 (secure 0 0 0) 0x7f9e90011aa0 con 0x7f9e940d0450 2023-10-06T03:31:01.105 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:01.096+0000 7f9e897fa700 1 --2- 172.21.15.110:0/2970448444 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f9e78060700 0x7f9e78062bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:31:01.105 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:01.096+0000 7f9e98c4a700 1 -- 172.21.15.110:0/2970448444 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f9e78060700 msgr2=0x7f9e78062bb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.110:6832/33268 2023-10-06T03:31:01.105 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:01.096+0000 7f9e897fa700 1 -- 172.21.15.110:0/2970448444 <== mon.1 v2:172.21.15.192:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 6636+0+0 (secure 0 0 0) 0x7f9e9007e6d0 con 0x7f9e940d0450 2023-10-06T03:31:01.105 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:01.096+0000 7f9e98c4a700 1 --2- 172.21.15.110:0/2970448444 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f9e78060700 0x7f9e78062bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-06T03:31:01.115 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:01.108+0000 7f9e897fa700 1 -- 172.21.15.110:0/2970448444 <== mon.1 v2:172.21.15.192:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f9e9004e6a0 con 0x7f9e940d0450 2023-10-06T03:31:01.240 INFO:tasks.ceph.mgr.y.smithi110.stderr:2023-10-06T03:31:01.236+0000 7ff496339000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-06T03:31:01.301 INFO:tasks.ceph.mgr.y.smithi110.stderr:2023-10-06T03:31:01.296+0000 7ff496339000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-06T03:31:01.303 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:01.300+0000 7f9e98c4a700 1 -- 172.21.15.110:0/2970448444 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f9e78060700 msgr2=0x7f9e78062bb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.110:6832/33268 2023-10-06T03:31:01.304 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:01.300+0000 7f9e98c4a700 1 --2- 172.21.15.110:0/2970448444 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f9e78060700 0x7f9e78062bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-06T03:31:01.311 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:01.304+0000 7f9e9b6af700 1 -- 172.21.15.110:0/2970448444 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9e940cdfa0 con 0x7f9e940d0450 2023-10-06T03:31:01.312 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:01.308+0000 7f9e897fa700 1 -- 172.21.15.110:0/2970448444 <== mon.1 v2:172.21.15.192:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) v1 ==== 94+0+4603 (secure 0 0 0) 0x7f9e90051cc0 con 0x7f9e940d0450 2023-10-06T03:31:01.312 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-10-06T03:31:01.313 INFO:teuthology.orchestra.run.smithi110.stdout:{"epoch":16,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5107,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6834/884349347","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6833","nonce":884349347},{"type":"v1","addr":"172.21.15.192:6834","nonce":884349347}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":5110,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6835/4077204152","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6832","nonce":4077204152},{"type":"v1","addr":"172.21.15.192:6835","nonce":4077204152}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":5160,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/504583843","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6834","nonce":504583843},{"type":"v1","addr":"172.21.15.110:6835","nonce":504583843}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-06T03:30:34.014644+0000","modified":"2023-10-06T03:30:37.662700+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5163},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5163":{"gid":5163,"name":"a","rank":0,"incarnation":13,"state":"up:active","state_seq":4,"addr":"172.21.15.110:6837/1379510962","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6836","nonce":1379510962},{"type":"v1","addr":"172.21.15.110:6837","nonce":1379510962}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":2}]} 2023-10-06T03:31:01.315 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:01.308+0000 7f9e9b6af700 1 -- 172.21.15.110:0/2970448444 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f9e78060700 msgr2=0x7f9e78062bb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-06T03:31:01.315 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:01.308+0000 7f9e9b6af700 1 --2- 172.21.15.110:0/2970448444 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f9e78060700 0x7f9e78062bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:01.315 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:01.308+0000 7f9e9b6af700 1 -- 172.21.15.110:0/2970448444 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9e940d0450 msgr2=0x7f9e94142250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:31:01.315 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:01.308+0000 7f9e9b6af700 1 --2- 172.21.15.110:0/2970448444 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9e940d0450 0x7f9e94142250 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f9e940cd6f0 tx=0x7f9e90017000 comp rx=0 tx=0).stop 2023-10-06T03:31:01.315 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:01.308+0000 7f9e9b6af700 1 -- 172.21.15.110:0/2970448444 shutdown_connections 2023-10-06T03:31:01.315 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:01.308+0000 7f9e9b6af700 1 --2- 172.21.15.110:0/2970448444 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f9e78060700 0x7f9e78062bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:01.315 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:01.308+0000 7f9e9b6af700 1 --2- 172.21.15.110:0/2970448444 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9e941592c0 0x7f9e94156f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:01.315 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:01.308+0000 7f9e9b6af700 1 --2- 172.21.15.110:0/2970448444 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9e94158ef0 0x7f9e940c8620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:01.315 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:01.308+0000 7f9e9b6af700 1 --2- 172.21.15.110:0/2970448444 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9e940d0450 0x7f9e94142250 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:01.316 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:01.308+0000 7f9e9b6af700 1 -- 172.21.15.110:0/2970448444 >> 172.21.15.110:0/2970448444 conn(0x7f9e94067040 msgr2=0x7f9e940b3430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:31:01.316 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:01.308+0000 7f9e9b6af700 1 -- 172.21.15.110:0/2970448444 shutdown_connections 2023-10-06T03:31:01.316 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:01.308+0000 7f9e9b6af700 1 -- 172.21.15.110:0/2970448444 wait complete. 2023-10-06T03:31:01.316 INFO:teuthology.orchestra.run.smithi110.stderr:dumped fsmap epoch 16 2023-10-06T03:31:01.326 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-06T03:31:01.476 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:01.472+0000 7f4dd5b8c700 1 Processor -- start 2023-10-06T03:31:01.476 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:01.472+0000 7f4dd5b8c700 1 -- start start 2023-10-06T03:31:01.478 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:01.472+0000 7f4dd5b8c700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f4dd0158dd0 0x7f4dd006bba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:31:01.478 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:01.472+0000 7f4dd5b8c700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4dd0062f90 0x7f4dd006ae50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:31:01.478 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:01.472+0000 7f4dd5b8c700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f4dd01591a0 0x7f4dd006b390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:31:01.478 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:01.472+0000 7f4dd5b8c700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4dd00cb4c0 con 0x7f4dd01591a0 2023-10-06T03:31:01.478 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:01.472+0000 7f4dd5b8c700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4dd00ca8b0 con 0x7f4dd0158dd0 2023-10-06T03:31:01.479 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:01.472+0000 7f4dd5b8c700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4dd00cab90 con 0x7f4dd0062f90 2023-10-06T03:31:01.479 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:01.472+0000 7f4dcffff700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f4dd01591a0 0x7f4dd006b390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:31:01.479 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:01.472+0000 7f4dcf7fe700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f4dd0158dd0 0x7f4dd006bba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:31:01.479 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:01.472+0000 7f4dcffff700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f4dd01591a0 0x7f4dd006b390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:38634/0 (socket says 172.21.15.110:38634) 2023-10-06T03:31:01.479 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:01.472+0000 7f4dcffff700 1 -- 172.21.15.110:0/3775292910 learned_addr learned my addr 172.21.15.110:0/3775292910 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:31:01.479 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:01.472+0000 7f4dceffd700 1 --2- 172.21.15.110:0/3775292910 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4dd0062f90 0x7f4dd006ae50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:31:01.479 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:01.472+0000 7f4dcffff700 1 -- 172.21.15.110:0/3775292910 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4dd0062f90 msgr2=0x7f4dd006ae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:31:01.479 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:01.472+0000 7f4dcffff700 1 --2- 172.21.15.110:0/3775292910 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4dd0062f90 0x7f4dd006ae50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:01.479 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:01.472+0000 7f4dcffff700 1 -- 172.21.15.110:0/3775292910 >> v2:172.21.15.192:3300/0 conn(0x7f4dd0158dd0 msgr2=0x7f4dd006bba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:31:01.480 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:01.472+0000 7f4dcffff700 1 --2- 172.21.15.110:0/3775292910 >> v2:172.21.15.192:3300/0 conn(0x7f4dd0158dd0 0x7f4dd006bba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:01.480 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:01.472+0000 7f4dcffff700 1 -- 172.21.15.110:0/3775292910 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4dd00cae70 con 0x7f4dd01591a0 2023-10-06T03:31:01.480 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:01.472+0000 7f4dcf7fe700 1 --2- 172.21.15.110:0/3775292910 >> v2:172.21.15.192:3300/0 conn(0x7f4dd0158dd0 0x7f4dd006bba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-06T03:31:01.480 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:01.472+0000 7f4dcffff700 1 --2- 172.21.15.110:0/3775292910 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4dd01591a0 0x7f4dd006b390 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f4dd006a2a0 tx=0x7f4dc0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e2e10e7abf7a99d server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:31:01.480 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:01.472+0000 7f4dce7fc700 1 -- 172.21.15.110:0/3775292910 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dc000b640 con 0x7f4dd01591a0 2023-10-06T03:31:01.480 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:01.472+0000 7f4dce7fc700 1 -- 172.21.15.110:0/3775292910 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4dc000b7a0 con 0x7f4dd01591a0 2023-10-06T03:31:01.480 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:01.472+0000 7f4dce7fc700 1 -- 172.21.15.110:0/3775292910 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dc000bb60 con 0x7f4dd01591a0 2023-10-06T03:31:01.481 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:01.472+0000 7f4dd5b8c700 1 -- 172.21.15.110:0/3775292910 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4dd01591a0 msgr2=0x7f4dd006b390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:31:01.481 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:01.472+0000 7f4dd5b8c700 1 --2- 172.21.15.110:0/3775292910 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4dd01591a0 0x7f4dd006b390 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f4dd006a2a0 tx=0x7f4dc0009250 comp rx=0 tx=0).stop 2023-10-06T03:31:01.481 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:01.472+0000 7f4dd5b8c700 1 -- 172.21.15.110:0/3775292910 shutdown_connections 2023-10-06T03:31:01.481 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:01.472+0000 7f4dd5b8c700 1 --2- 172.21.15.110:0/3775292910 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4dd01591a0 0x7f4dd006b390 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:01.481 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:01.472+0000 7f4dd5b8c700 1 --2- 172.21.15.110:0/3775292910 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4dd0062f90 0x7f4dd006ae50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:01.481 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:01.472+0000 7f4dd5b8c700 1 --2- 172.21.15.110:0/3775292910 >> v2:172.21.15.192:3300/0 conn(0x7f4dd0158dd0 0x7f4dd006bba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:01.482 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:01.472+0000 7f4dd5b8c700 1 -- 172.21.15.110:0/3775292910 >> 172.21.15.110:0/3775292910 conn(0x7f4dd00cc3b0 msgr2=0x7f4dd00cc780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:31:01.482 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:01.476+0000 7f4dd5b8c700 1 -- 172.21.15.110:0/3775292910 shutdown_connections 2023-10-06T03:31:01.482 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:01.476+0000 7f4dd5b8c700 1 -- 172.21.15.110:0/3775292910 wait complete. 2023-10-06T03:31:01.482 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:01.476+0000 7f4dd5b8c700 1 Processor -- start 2023-10-06T03:31:01.482 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:01.476+0000 7f4dd5b8c700 1 -- start start 2023-10-06T03:31:01.484 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:01.476+0000 7f4dd5b8c700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4dd0062f90 0x7f4dd0150be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:31:01.484 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:01.476+0000 7f4dd5b8c700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4dd0158dd0 0x7f4dd0151120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:31:01.484 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:01.476+0000 7f4dd5b8c700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4dd01591a0 0x7f4dd0169000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:31:01.484 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:01.476+0000 7f4dd5b8c700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4dd00af520 con 0x7f4dd01591a0 2023-10-06T03:31:01.485 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:01.476+0000 7f4dd5b8c700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4dd0151980 con 0x7f4dd0158dd0 2023-10-06T03:31:01.485 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:01.476+0000 7f4dd5b8c700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4dd0151c60 con 0x7f4dd0062f90 2023-10-06T03:31:01.485 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:01.480+0000 7f4dcffff700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4dd01591a0 0x7f4dd0169000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:31:01.485 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:01.480+0000 7f4dcffff700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4dd01591a0 0x7f4dd0169000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:38644/0 (socket says 172.21.15.110:38644) 2023-10-06T03:31:01.485 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:01.480+0000 7f4dcffff700 1 -- 172.21.15.110:0/1280613229 learned_addr learned my addr 172.21.15.110:0/1280613229 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:31:01.485 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:01.480+0000 7f4dcf7fe700 1 --2- 172.21.15.110:0/1280613229 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4dd0062f90 0x7f4dd0150be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:31:01.485 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:01.480+0000 7f4dceffd700 1 --2- 172.21.15.110:0/1280613229 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4dd0158dd0 0x7f4dd0151120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:31:01.485 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:01.480+0000 7f4dcffff700 1 -- 172.21.15.110:0/1280613229 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4dd0062f90 msgr2=0x7f4dd0150be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:31:01.485 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:01.480+0000 7f4dcffff700 1 --2- 172.21.15.110:0/1280613229 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4dd0062f90 0x7f4dd0150be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:01.485 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:01.480+0000 7f4dcffff700 1 -- 172.21.15.110:0/1280613229 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4dd0158dd0 msgr2=0x7f4dd0151120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:31:01.485 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:01.480+0000 7f4dcffff700 1 --2- 172.21.15.110:0/1280613229 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4dd0158dd0 0x7f4dd0151120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:01.485 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:01.480+0000 7f4dcffff700 1 -- 172.21.15.110:0/1280613229 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4dc8009a20 con 0x7f4dd01591a0 2023-10-06T03:31:01.486 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:01.480+0000 7f4dcffff700 1 --2- 172.21.15.110:0/1280613229 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4dd01591a0 0x7f4dd0169000 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f4dc00094c0 tx=0x7f4dc0017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:31:01.486 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:01.480+0000 7f4db77fe700 1 -- 172.21.15.110:0/1280613229 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dc000b910 con 0x7f4dd01591a0 2023-10-06T03:31:01.486 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:01.480+0000 7f4db77fe700 1 -- 172.21.15.110:0/1280613229 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4dc000ba70 con 0x7f4dd01591a0 2023-10-06T03:31:01.486 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:01.480+0000 7f4dd5b8c700 1 -- 172.21.15.110:0/1280613229 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4dc00094f0 con 0x7f4dd01591a0 2023-10-06T03:31:01.486 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:01.480+0000 7f4db77fe700 1 -- 172.21.15.110:0/1280613229 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dc0026680 con 0x7f4dd01591a0 2023-10-06T03:31:01.486 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:01.480+0000 7f4dd5b8c700 1 -- 172.21.15.110:0/1280613229 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4dd0152220 con 0x7f4dd01591a0 2023-10-06T03:31:01.487 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:01.480+0000 7f4dd5b8c700 1 -- 172.21.15.110:0/1280613229 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4dd00cb900 con 0x7f4dd01591a0 2023-10-06T03:31:01.487 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:01.480+0000 7f4db77fe700 1 -- 172.21.15.110:0/1280613229 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 10) v1 ==== 80118+0+0 (secure 0 0 0) 0x7f4dc0023030 con 0x7f4dd01591a0 2023-10-06T03:31:01.487 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:01.480+0000 7f4db77fe700 1 --2- 172.21.15.110:0/1280613229 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f4dbc060700 0x7f4dbc062bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:31:01.487 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:01.480+0000 7f4dcf7fe700 1 -- 172.21.15.110:0/1280613229 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f4dbc060700 msgr2=0x7f4dbc062bb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.110:6832/33268 2023-10-06T03:31:01.487 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:01.480+0000 7f4db77fe700 1 -- 172.21.15.110:0/1280613229 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 6636+0+0 (secure 0 0 0) 0x7f4dc007e680 con 0x7f4dd01591a0 2023-10-06T03:31:01.487 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:01.480+0000 7f4dcf7fe700 1 --2- 172.21.15.110:0/1280613229 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f4dbc060700 0x7f4dbc062bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-06T03:31:01.493 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:01.488+0000 7f4db77fe700 1 -- 172.21.15.110:0/1280613229 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f4dd00cb900 con 0x7f4dd01591a0 2023-10-06T03:31:01.688 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:01.684+0000 7f4dcf7fe700 1 -- 172.21.15.110:0/1280613229 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f4dbc060700 msgr2=0x7f4dbc062bb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.110:6832/33268 2023-10-06T03:31:01.688 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:01.684+0000 7f4dcf7fe700 1 --2- 172.21.15.110:0/1280613229 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f4dbc060700 0x7f4dbc062bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-06T03:31:01.688 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:01.684+0000 7f4dd5b8c700 1 -- 172.21.15.110:0/1280613229 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4dd00cb900 con 0x7f4dd01591a0 2023-10-06T03:31:01.688 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:01.684+0000 7f4db77fe700 1 -- 172.21.15.110:0/1280613229 <== mon.0 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) v1 ==== 94+0+4603 (secure 0 0 0) 0x7f4dd00cb900 con 0x7f4dd01591a0 2023-10-06T03:31:01.689 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-10-06T03:31:01.689 INFO:teuthology.orchestra.run.smithi110.stdout:{"epoch":16,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5107,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6834/884349347","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6833","nonce":884349347},{"type":"v1","addr":"172.21.15.192:6834","nonce":884349347}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":5110,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6835/4077204152","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6832","nonce":4077204152},{"type":"v1","addr":"172.21.15.192:6835","nonce":4077204152}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":5160,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/504583843","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6834","nonce":504583843},{"type":"v1","addr":"172.21.15.110:6835","nonce":504583843}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-06T03:30:34.014644+0000","modified":"2023-10-06T03:30:37.662700+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5163},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5163":{"gid":5163,"name":"a","rank":0,"incarnation":13,"state":"up:active","state_seq":4,"addr":"172.21.15.110:6837/1379510962","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6836","nonce":1379510962},{"type":"v1","addr":"172.21.15.110:6837","nonce":1379510962}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":2}]} 2023-10-06T03:31:01.691 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:01.684+0000 7f4dd5b8c700 1 -- 172.21.15.110:0/1280613229 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f4dbc060700 msgr2=0x7f4dbc062bb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-06T03:31:01.691 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:01.684+0000 7f4dd5b8c700 1 --2- 172.21.15.110:0/1280613229 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f4dbc060700 0x7f4dbc062bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:01.691 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:01.684+0000 7f4dd5b8c700 1 -- 172.21.15.110:0/1280613229 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4dd01591a0 msgr2=0x7f4dd0169000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:31:01.691 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:01.684+0000 7f4dd5b8c700 1 --2- 172.21.15.110:0/1280613229 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4dd01591a0 0x7f4dd0169000 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f4dc00094c0 tx=0x7f4dc0017000 comp rx=0 tx=0).stop 2023-10-06T03:31:01.691 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:01.684+0000 7f4dd5b8c700 1 -- 172.21.15.110:0/1280613229 shutdown_connections 2023-10-06T03:31:01.692 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:01.684+0000 7f4dd5b8c700 1 --2- 172.21.15.110:0/1280613229 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f4dbc060700 0x7f4dbc062bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:01.692 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:01.684+0000 7f4dd5b8c700 1 --2- 172.21.15.110:0/1280613229 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4dd01591a0 0x7f4dd0169000 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:01.692 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:01.684+0000 7f4dd5b8c700 1 --2- 172.21.15.110:0/1280613229 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4dd0158dd0 0x7f4dd0151120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:01.692 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:01.684+0000 7f4dd5b8c700 1 --2- 172.21.15.110:0/1280613229 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4dd0062f90 0x7f4dd0150be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:01.692 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:01.684+0000 7f4dd5b8c700 1 -- 172.21.15.110:0/1280613229 >> 172.21.15.110:0/1280613229 conn(0x7f4dd00cc3b0 msgr2=0x7f4dd0157f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:31:01.692 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:01.684+0000 7f4dd5b8c700 1 -- 172.21.15.110:0/1280613229 shutdown_connections 2023-10-06T03:31:01.692 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:01.684+0000 7f4dd5b8c700 1 -- 172.21.15.110:0/1280613229 wait complete. 2023-10-06T03:31:01.692 INFO:teuthology.orchestra.run.smithi110.stderr:dumped fsmap epoch 16 2023-10-06T03:31:01.698 INFO:tasks.ceph.mgr.x.smithi192.stderr:2023-10-06T03:31:01.696+0000 7f76034c8000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-06T03:31:01.703 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-06T03:31:01.806 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:01.800+0000 7f508d923700 1 Processor -- start 2023-10-06T03:31:01.806 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:01.800+0000 7f508d923700 1 -- start start 2023-10-06T03:31:01.808 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:01.800+0000 7f508d923700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f5088159060 0x7f50880aff00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:31:01.808 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:01.800+0000 7f508d923700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5088068270 0x7f50880a88c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:31:01.808 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:01.800+0000 7f508d923700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f508806d830 con 0x7f5088158c90 2023-10-06T03:31:01.808 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:01.800+0000 7f508d923700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f50880afa70 con 0x7f5088159060 2023-10-06T03:31:01.809 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:01.800+0000 7f508d923700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50880af140 con 0x7f5088068270 2023-10-06T03:31:01.809 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:01.800+0000 7f5087fff700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f5088159060 0x7f50880aff00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:31:01.809 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:01.800+0000 7f5087fff700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f5088159060 0x7f50880aff00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:38646/0 (socket says 172.21.15.110:38646) 2023-10-06T03:31:01.809 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:01.800+0000 7f5087fff700 1 -- 172.21.15.110:0/2635489376 learned_addr learned my addr 172.21.15.110:0/2635489376 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:31:01.809 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:01.800+0000 7f508d122700 1 --2- 172.21.15.110:0/2635489376 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5088068270 0x7f50880a88c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:31:01.809 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:01.800+0000 7f5087fff700 1 -- 172.21.15.110:0/2635489376 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5088068270 msgr2=0x7f50880a88c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:31:01.809 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:01.800+0000 7f5087fff700 1 --2- 172.21.15.110:0/2635489376 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5088068270 0x7f50880a88c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:01.809 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:01.800+0000 7f5087fff700 1 -- 172.21.15.110:0/2635489376 >> v1:172.21.15.192:6789/0 conn(0x7f5088158c90 legacy=0x7f508806d120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:31:01.809 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:01.800+0000 7f5087fff700 1 -- 172.21.15.110:0/2635489376 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50880af420 con 0x7f5088159060 2023-10-06T03:31:01.810 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:01.800+0000 7f5087fff700 1 --2- 172.21.15.110:0/2635489376 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5088159060 0x7f50880aff00 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f507c0099b0 tx=0x7f507c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=344dff9c8859b1b0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:31:01.810 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:01.804+0000 7f50877fe700 1 -- 172.21.15.110:0/2635489376 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f507c00b860 con 0x7f5088159060 2023-10-06T03:31:01.810 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:01.804+0000 7f50877fe700 1 -- 172.21.15.110:0/2635489376 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f507c00b9c0 con 0x7f5088159060 2023-10-06T03:31:01.810 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:01.804+0000 7f50877fe700 1 -- 172.21.15.110:0/2635489376 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f507c00bd60 con 0x7f5088159060 2023-10-06T03:31:01.810 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:01.804+0000 7f508d923700 1 -- 172.21.15.110:0/2635489376 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5088159060 msgr2=0x7f50880aff00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:31:01.810 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:01.804+0000 7f508d923700 1 --2- 172.21.15.110:0/2635489376 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5088159060 0x7f50880aff00 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f507c0099b0 tx=0x7f507c009250 comp rx=0 tx=0).stop 2023-10-06T03:31:01.810 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:01.804+0000 7f508d923700 1 -- 172.21.15.110:0/2635489376 shutdown_connections 2023-10-06T03:31:01.810 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:01.804+0000 7f508d923700 1 --2- 172.21.15.110:0/2635489376 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5088068270 0x7f50880a88c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:01.810 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:01.804+0000 7f508d923700 1 --2- 172.21.15.110:0/2635489376 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5088159060 0x7f50880aff00 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:01.811 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:01.804+0000 7f508d923700 1 -- 172.21.15.110:0/2635489376 >> 172.21.15.110:0/2635489376 conn(0x7f50880cccb0 msgr2=0x7f50880adbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:31:01.811 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:01.804+0000 7f508d923700 1 -- 172.21.15.110:0/2635489376 shutdown_connections 2023-10-06T03:31:01.811 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:01.804+0000 7f508d923700 1 -- 172.21.15.110:0/2635489376 wait complete. 2023-10-06T03:31:01.813 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:01.804+0000 7f508d923700 1 Processor -- start 2023-10-06T03:31:01.813 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:01.804+0000 7f508d923700 1 -- start start 2023-10-06T03:31:01.813 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:01.804+0000 7f508d923700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5088068270 0x7f5088157e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:31:01.813 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:01.804+0000 7f508d923700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5088158c90 0x7f508816c1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:31:01.814 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:01.804+0000 7f508d923700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5088159060 0x7f508816cac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:31:01.814 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:01.808+0000 7f508d923700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50880c13a0 con 0x7f5088159060 2023-10-06T03:31:01.814 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:01.808+0000 7f508d923700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f508815b0c0 con 0x7f5088158c90 2023-10-06T03:31:01.815 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:01.808+0000 7f508d923700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f508815a7d0 con 0x7f5088068270 2023-10-06T03:31:01.815 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:01.808+0000 7f508d122700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5088159060 0x7f508816cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:31:01.815 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:01.808+0000 7f508d122700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5088159060 0x7f508816cac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:38652/0 (socket says 172.21.15.110:38652) 2023-10-06T03:31:01.815 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:01.808+0000 7f508d122700 1 -- 172.21.15.110:0/2301341802 learned_addr learned my addr 172.21.15.110:0/2301341802 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:31:01.816 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:01.808+0000 7f508c921700 1 --2- 172.21.15.110:0/2301341802 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5088068270 0x7f5088157e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:31:01.816 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:01.808+0000 7f5087fff700 1 --2- 172.21.15.110:0/2301341802 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5088158c90 0x7f508816c1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:31:01.816 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:01.808+0000 7f508d122700 1 -- 172.21.15.110:0/2301341802 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5088068270 msgr2=0x7f5088157e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:31:01.816 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:01.808+0000 7f508d122700 1 --2- 172.21.15.110:0/2301341802 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5088068270 0x7f5088157e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:01.816 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:01.808+0000 7f508d122700 1 -- 172.21.15.110:0/2301341802 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5088158c90 msgr2=0x7f508816c1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:31:01.817 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:01.808+0000 7f508d122700 1 --2- 172.21.15.110:0/2301341802 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5088158c90 0x7f508816c1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:01.817 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:01.808+0000 7f508d122700 1 -- 172.21.15.110:0/2301341802 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f507c0094f0 con 0x7f5088159060 2023-10-06T03:31:01.817 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:01.808+0000 7f508d122700 1 --2- 172.21.15.110:0/2301341802 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5088159060 0x7f508816cac0 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f5080007f30 tx=0x7f50800098e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:31:01.817 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:01.808+0000 7f5084ff9700 1 -- 172.21.15.110:0/2301341802 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f508000eb90 con 0x7f5088159060 2023-10-06T03:31:01.817 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:01.808+0000 7f5084ff9700 1 -- 172.21.15.110:0/2301341802 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f508000ecf0 con 0x7f5088159060 2023-10-06T03:31:01.817 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:01.808+0000 7f508d923700 1 -- 172.21.15.110:0/2301341802 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f508815aa70 con 0x7f5088159060 2023-10-06T03:31:01.817 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:01.808+0000 7f5084ff9700 1 -- 172.21.15.110:0/2301341802 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5080014670 con 0x7f5088159060 2023-10-06T03:31:01.817 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:01.808+0000 7f508d923700 1 -- 172.21.15.110:0/2301341802 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50881584a0 con 0x7f5088159060 2023-10-06T03:31:01.818 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:01.808+0000 7f508d923700 1 -- 172.21.15.110:0/2301341802 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f508815b230 con 0x7f5088159060 2023-10-06T03:31:01.818 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:01.808+0000 7f5084ff9700 1 -- 172.21.15.110:0/2301341802 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 10) v1 ==== 80118+0+0 (secure 0 0 0) 0x7f5080014a20 con 0x7f5088159060 2023-10-06T03:31:01.818 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:01.808+0000 7f5084ff9700 1 --2- 172.21.15.110:0/2301341802 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f5074060700 0x7f5074062bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:31:01.818 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:01.808+0000 7f508c921700 1 -- 172.21.15.110:0/2301341802 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f5074060700 msgr2=0x7f5074062bb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.110:6832/33268 2023-10-06T03:31:01.818 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:01.808+0000 7f5084ff9700 1 -- 172.21.15.110:0/2301341802 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 6636+0+0 (secure 0 0 0) 0x7f508007d530 con 0x7f5088159060 2023-10-06T03:31:01.819 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:01.808+0000 7f508c921700 1 --2- 172.21.15.110:0/2301341802 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f5074060700 0x7f5074062bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-06T03:31:01.820 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:01.816+0000 7f5084ff9700 1 -- 172.21.15.110:0/2301341802 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f508004d500 con 0x7f5088159060 2023-10-06T03:31:01.904 INFO:tasks.ceph.mgr.y.smithi110.stderr:2023-10-06T03:31:01.900+0000 7ff496339000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-06T03:31:01.938 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:01.932+0000 7f5084ff9700 1 -- 172.21.15.110:0/2301341802 <== mon.0 v2:172.21.15.110:3300/0 7 ==== mgrmap(e 11) v1 ==== 79670+0+0 (secure 0 0 0) 0x7f508004d8d0 con 0x7f5088159060 2023-10-06T03:31:01.938 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:01.932+0000 7f5084ff9700 1 -- 172.21.15.110:0/2301341802 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f5074060700 msgr2=0x7f5074062bb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-06T03:31:01.938 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:01.932+0000 7f5084ff9700 1 --2- 172.21.15.110:0/2301341802 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f5074060700 0x7f5074062bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:01.995 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:01.988+0000 7f508d923700 1 -- 172.21.15.110:0/2301341802 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f50880af700 con 0x7f5088159060 2023-10-06T03:31:01.997 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:01.992+0000 7f5084ff9700 1 -- 172.21.15.110:0/2301341802 <== mon.0 v2:172.21.15.110:3300/0 8 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v44) v1 ==== 74+0+17932 (secure 0 0 0) 0x7f508003a190 con 0x7f5088159060 2023-10-06T03:31:01.997 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-10-06T03:31:01.998 INFO:teuthology.orchestra.run.smithi110.stdout:{"epoch":44,"fsid":"93b8f6ac-4c92-4b45-b8e4-b59617024aa2","created":"2023-10-06T03:29:10.519583+0000","modified":"2023-10-06T03:31:01.923613+0000","last_up_change":"2023-10-06T03:29:21.960511+0000","last_in_change":"2023-10-06T03:29:12.560436+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-06T03:29:20.703743+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2023-10-06T03:29:28.307281+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2023-10-06T03:30:31.807897+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"41","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":7,"pool_name":"cephfs_data","create_time":"2023-10-06T03:30:32.991334+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"42","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"5866eb6c-2adc-46e4-be7a-e5bcda0f4065","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6824","nonce":33394},{"type":"v1","addr":"172.21.15.110:6825","nonce":33394}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6826","nonce":33394},{"type":"v1","addr":"172.21.15.110:6827","nonce":33394}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6830","nonce":33394},{"type":"v1","addr":"172.21.15.110:6831","nonce":33394}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6828","nonce":33394},{"type":"v1","addr":"172.21.15.110:6829","nonce":33394}]},"public_addr":"172.21.15.110:6825/33394","cluster_addr":"172.21.15.110:6827/33394","heartbeat_back_addr":"172.21.15.110:6831/33394","heartbeat_front_addr":"172.21.15.110:6829/33394","state":["exists","up"]},{"osd":1,"uuid":"a38de118-062b-4a05-9782-2ceddf6ff4df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6800","nonce":33396},{"type":"v1","addr":"172.21.15.110:6801","nonce":33396}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6802","nonce":33396},{"type":"v1","addr":"172.21.15.110:6803","nonce":33396}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6806","nonce":33396},{"type":"v1","addr":"172.21.15.110:6807","nonce":33396}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6804","nonce":33396},{"type":"v1","addr":"172.21.15.110:6805","nonce":33396}]},"public_addr":"172.21.15.110:6801/33396","cluster_addr":"172.21.15.110:6803/33396","heartbeat_back_addr":"172.21.15.110:6807/33396","heartbeat_front_addr":"172.21.15.110:6805/33396","state":["exists","up"]},{"osd":2,"uuid":"56e39be2-3407-4b69-877e-cf16617b06d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6816","nonce":33397},{"type":"v1","addr":"172.21.15.110:6817","nonce":33397}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6818","nonce":33397},{"type":"v1","addr":"172.21.15.110:6819","nonce":33397}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6822","nonce":33397},{"type":"v1","addr":"172.21.15.110:6823","nonce":33397}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6820","nonce":33397},{"type":"v1","addr":"172.21.15.110:6821","nonce":33397}]},"public_addr":"172.21.15.110:6817/33397","cluster_addr":"172.21.15.110:6819/33397","heartbeat_back_addr":"172.21.15.110:6823/33397","heartbeat_front_addr":"172.21.15.110:6821/33397","state":["exists","up"]},{"osd":3,"uuid":"01c82d13-c1d4-4bde-b55e-3e0089b40115","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6808","nonce":33395},{"type":"v1","addr":"172.21.15.110:6809","nonce":33395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6810","nonce":33395},{"type":"v1","addr":"172.21.15.110:6811","nonce":33395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6814","nonce":33395},{"type":"v1","addr":"172.21.15.110:6815","nonce":33395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6812","nonce":33395},{"type":"v1","addr":"172.21.15.110:6813","nonce":33395}]},"public_addr":"172.21.15.110:6809/33395","cluster_addr":"172.21.15.110:6811/33395","heartbeat_back_addr":"172.21.15.110:6815/33395","heartbeat_front_addr":"172.21.15.110:6813/33395","state":["exists","up"]},{"osd":4,"uuid":"86983430-5fa3-4d17-998f-03e2cd843237","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":33683},{"type":"v1","addr":"172.21.15.192:6817","nonce":33683}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":33683},{"type":"v1","addr":"172.21.15.192:6819","nonce":33683}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":33683},{"type":"v1","addr":"172.21.15.192:6823","nonce":33683}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6820","nonce":33683},{"type":"v1","addr":"172.21.15.192:6821","nonce":33683}]},"public_addr":"172.21.15.192:6817/33683","cluster_addr":"172.21.15.192:6819/33683","heartbeat_back_addr":"172.21.15.192:6823/33683","heartbeat_front_addr":"172.21.15.192:6821/33683","state":["exists","up"]},{"osd":5,"uuid":"6945257f-821d-4619-a63f-ebe857eba6f4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":33684},{"type":"v1","addr":"172.21.15.192:6801","nonce":33684}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":33684},{"type":"v1","addr":"172.21.15.192:6803","nonce":33684}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":33684},{"type":"v1","addr":"172.21.15.192:6807","nonce":33684}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6804","nonce":33684},{"type":"v1","addr":"172.21.15.192:6805","nonce":33684}]},"public_addr":"172.21.15.192:6801/33684","cluster_addr":"172.21.15.192:6803/33684","heartbeat_back_addr":"172.21.15.192:6807/33684","heartbeat_front_addr":"172.21.15.192:6805/33684","state":["exists","up"]},{"osd":6,"uuid":"7854d1ae-7060-4780-8f97-73cea5e7a12f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6824","nonce":33686},{"type":"v1","addr":"172.21.15.192:6825","nonce":33686}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":33686},{"type":"v1","addr":"172.21.15.192:6827","nonce":33686}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":33686},{"type":"v1","addr":"172.21.15.192:6831","nonce":33686}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6828","nonce":33686},{"type":"v1","addr":"172.21.15.192:6829","nonce":33686}]},"public_addr":"172.21.15.192:6825/33686","cluster_addr":"172.21.15.192:6827/33686","heartbeat_back_addr":"172.21.15.192:6831/33686","heartbeat_front_addr":"172.21.15.192:6829/33686","state":["exists","up"]},{"osd":7,"uuid":"f1ebdfd5-b859-482b-9f97-c18c4a26f445","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6808","nonce":33685},{"type":"v1","addr":"172.21.15.192:6809","nonce":33685}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":33685},{"type":"v1","addr":"172.21.15.192:6811","nonce":33685}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":33685},{"type":"v1","addr":"172.21.15.192:6815","nonce":33685}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6812","nonce":33685},{"type":"v1","addr":"172.21.15.192:6813","nonce":33685}]},"public_addr":"172.21.15.192:6809/33685","cluster_addr":"172.21.15.192:6811/33685","heartbeat_back_addr":"172.21.15.192:6815/33685","heartbeat_front_addr":"172.21.15.192:6813/33685","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T03:29:18.002048+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T03:29:17.478576+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T03:29:18.034944+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T03:29:17.628513+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T03:29:18.047338+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T03:29:17.772349+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T03:29:17.883958+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T03:29:18.151838+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.110:0/3461792981":"2023-10-07T03:31:01.923482+0000","172.21.15.110:0/1315925152":"2023-10-07T03:31:01.923482+0000","172.21.15.110:0/3495425172":"2023-10-07T03:31:01.923482+0000","172.21.15.110:0/3149105774":"2023-10-07T03:31:01.923482+0000","172.21.15.110:0/3193317457":"2023-10-07T03:31:01.923482+0000","172.21.15.110:0/2107091982":"2023-10-07T03:31:01.923482+0000","172.21.15.110:0/2944476429":"2023-10-07T03:31:01.923482+0000","172.21.15.110:0/1857995858":"2023-10-07T03:30:46.697197+0000","172.21.15.110:0/1318624357":"2023-10-07T03:30:46.697197+0000","172.21.15.110:0/1139019390":"2023-10-07T03:30:46.697197+0000","172.21.15.110:6833/33268":"2023-10-07T03:30:46.697197+0000","172.21.15.110:0/463196587":"2023-10-07T03:30:46.697197+0000","172.21.15.110:6832/33268":"2023-10-07T03:30:46.697197+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-06T03:31:02.000 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:01.996+0000 7f508d923700 1 -- 172.21.15.110:0/2301341802 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5088159060 msgr2=0x7f508816cac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:31:02.001 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:01.996+0000 7f508d923700 1 --2- 172.21.15.110:0/2301341802 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5088159060 0x7f508816cac0 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f5080007f30 tx=0x7f50800098e0 comp rx=0 tx=0).stop 2023-10-06T03:31:02.001 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:01.996+0000 7f508d923700 1 -- 172.21.15.110:0/2301341802 shutdown_connections 2023-10-06T03:31:02.001 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:01.996+0000 7f508d923700 1 --2- 172.21.15.110:0/2301341802 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f5074060700 0x7f5074062bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:02.001 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:01.996+0000 7f508d923700 1 --2- 172.21.15.110:0/2301341802 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5088159060 0x7f508816cac0 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:02.001 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:01.996+0000 7f508d923700 1 --2- 172.21.15.110:0/2301341802 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5088158c90 0x7f508816c1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:02.001 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:01.996+0000 7f508d923700 1 --2- 172.21.15.110:0/2301341802 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f5088068270 0x7f5088157e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:02.001 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:01.996+0000 7f508d923700 1 -- 172.21.15.110:0/2301341802 >> 172.21.15.110:0/2301341802 conn(0x7f50880cccb0 msgr2=0x7f5088153de0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:31:02.001 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:01.996+0000 7f508d923700 1 -- 172.21.15.110:0/2301341802 shutdown_connections 2023-10-06T03:31:02.001 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:01.996+0000 7f508d923700 1 -- 172.21.15.110:0/2301341802 wait complete. 2023-10-06T03:31:02.007 INFO:tasks.ceph.mgr.y.smithi110.stderr:2023-10-06T03:31:02.000+0000 7ff462d7c700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-06T03:31:02.007 INFO:tasks.ceph.mgr.y.smithi110.stderr:2023-10-06T03:31:02.000+0000 7ff462d7c700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-06T03:31:02.007 INFO:tasks.ceph.mgr.y.smithi110.stderr:2023-10-06T03:31:02.000+0000 7ff462d7c700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-06T03:31:02.007 INFO:tasks.ceph.mgr.y.smithi110.stderr:2023-10-06T03:31:02.000+0000 7ff462d7c700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-06T03:31:02.007 INFO:tasks.ceph.mgr.y.smithi110.stderr:2023-10-06T03:31:02.000+0000 7ff462d7c700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-06T03:31:02.008 INFO:tasks.ceph.mgr.y.smithi110.stderr:2023-10-06T03:31:02.004+0000 7ff45fd76700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-06T03:31:02.008 INFO:tasks.ceph.mgr.y.smithi110.stderr:2023-10-06T03:31:02.004+0000 7ff45fd76700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-06T03:31:02.008 INFO:tasks.ceph.mgr.y.smithi110.stderr:2023-10-06T03:31:02.004+0000 7ff45fd76700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-06T03:31:02.008 INFO:tasks.ceph.mgr.y.smithi110.stderr:2023-10-06T03:31:02.004+0000 7ff45fd76700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-06T03:31:02.008 INFO:tasks.ceph.mgr.y.smithi110.stderr:2023-10-06T03:31:02.004+0000 7ff45fd76700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-06T03:31:02.015 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2023-10-06T03:31:02.108 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:02.104+0000 7fc5699d7700 1 Processor -- start 2023-10-06T03:31:02.108 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:02.104+0000 7fc5699d7700 1 -- start start 2023-10-06T03:31:02.109 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:02.104+0000 7fc5699d7700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fc5640b5320 0x7fc5640cf3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:31:02.110 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:02.104+0000 7fc5699d7700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5640cdf40 con 0x7fc564158da0 2023-10-06T03:31:02.110 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:02.104+0000 7fc5699d7700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5640cc790 con 0x7fc5640d03e0 2023-10-06T03:31:02.110 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:02.104+0000 7fc5699d7700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc5640b5aa0 con 0x7fc5640b5320 2023-10-06T03:31:02.110 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:02.104+0000 7fc5689d5700 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7fc564158da0 0x7fc5640cd830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:60302/0 (socket says 172.21.15.110:60302) 2023-10-06T03:31:02.110 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:02.104+0000 7fc5689d5700 1 -- 172.21.15.110:0/570718822 learned_addr learned my addr 172.21.15.110:0/570718822 (peer_addr_for_me v1:172.21.15.110:0/0) 2023-10-06T03:31:02.110 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:02.104+0000 7fc5691d6700 1 --2- 172.21.15.110:0/570718822 >> v2:172.21.15.192:3300/0 conn(0x7fc5640b5320 0x7fc5640cf3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:31:02.110 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:02.104+0000 7fc5637fe700 1 -- 172.21.15.110:0/570718822 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 865962517 0 0) 0x7fc5640cdf40 con 0x7fc564158da0 2023-10-06T03:31:02.110 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:02.104+0000 7fc5637fe700 1 -- 172.21.15.110:0/570718822 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc54c003570 con 0x7fc564158da0 2023-10-06T03:31:02.110 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:02.104+0000 7fc5637fe700 1 -- 172.21.15.110:0/570718822 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2277200785 0 0) 0x7fc5640cc790 con 0x7fc5640d03e0 2023-10-06T03:31:02.110 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:02.104+0000 7fc5637fe700 1 -- 172.21.15.110:0/570718822 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc5640cdf40 con 0x7fc5640d03e0 2023-10-06T03:31:02.111 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:02.104+0000 7fc5637fe700 1 -- 172.21.15.110:0/570718822 <== mon.0 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2590940828 0 0) 0x7fc54c003570 con 0x7fc564158da0 2023-10-06T03:31:02.111 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:02.104+0000 7fc5691d6700 1 -- 172.21.15.110:0/570718822 >> v1:172.21.15.192:6789/0 conn(0x7fc5640d03e0 legacy=0x7fc5640b4c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:31:02.111 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:02.104+0000 7fc5691d6700 1 -- 172.21.15.110:0/570718822 >> v1:172.21.15.110:6789/0 conn(0x7fc564158da0 legacy=0x7fc5640cd830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:31:02.111 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:02.104+0000 7fc5691d6700 1 -- 172.21.15.110:0/570718822 --> v2:172.21.15.192:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5640cca70 con 0x7fc5640b5320 2023-10-06T03:31:02.112 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:02.104+0000 7fc5691d6700 1 --2- 172.21.15.110:0/570718822 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc5640b5320 0x7fc5640cf3c0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fc55c0098e0 tx=0x7fc55c0091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=598089f125bfd6b7 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:31:02.112 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:02.104+0000 7fc5637fe700 1 -- 172.21.15.110:0/570718822 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5580015f0 con 0x7fc5640b5320 2023-10-06T03:31:02.112 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:02.104+0000 7fc5637fe700 1 -- 172.21.15.110:0/570718822 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc55c011420 con 0x7fc5640b5320 2023-10-06T03:31:02.112 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:02.104+0000 7fc5637fe700 1 -- 172.21.15.110:0/570718822 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc55c0117a0 con 0x7fc5640b5320 2023-10-06T03:31:02.112 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:02.104+0000 7fc5699d7700 1 -- 172.21.15.110:0/570718822 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc5640b5320 msgr2=0x7fc5640cf3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:31:02.113 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:02.108+0000 7fc5699d7700 1 --2- 172.21.15.110:0/570718822 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc5640b5320 0x7fc5640cf3c0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fc55c0098e0 tx=0x7fc55c0091a0 comp rx=0 tx=0).stop 2023-10-06T03:31:02.113 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:02.108+0000 7fc5699d7700 1 -- 172.21.15.110:0/570718822 shutdown_connections 2023-10-06T03:31:02.113 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:02.108+0000 7fc5699d7700 1 --2- 172.21.15.110:0/570718822 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc5640b5320 0x7fc5640cf3c0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:02.113 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:02.108+0000 7fc5699d7700 1 -- 172.21.15.110:0/570718822 >> 172.21.15.110:0/570718822 conn(0x7fc56406c760 msgr2=0x7fc56406cb60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:31:02.113 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:02.108+0000 7fc5699d7700 1 -- 172.21.15.110:0/570718822 shutdown_connections 2023-10-06T03:31:02.113 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:02.108+0000 7fc5699d7700 1 -- 172.21.15.110:0/570718822 wait complete. 2023-10-06T03:31:02.113 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:02.108+0000 7fc5699d7700 1 Processor -- start 2023-10-06T03:31:02.114 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:02.108+0000 7fc5699d7700 1 -- start start 2023-10-06T03:31:02.115 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:02.108+0000 7fc5699d7700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc5640b5320 0x7fc564153860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:31:02.116 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:02.108+0000 7fc5699d7700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc5640d03e0 0x7fc56415da10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:31:02.116 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:02.108+0000 7fc5699d7700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc564158da0 0x7fc5641690b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:31:02.116 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:02.108+0000 7fc5699d7700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc56415e500 con 0x7fc5640d03e0 2023-10-06T03:31:02.116 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:02.108+0000 7fc5699d7700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc56415df50 con 0x7fc5640b5320 2023-10-06T03:31:02.116 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:02.108+0000 7fc5699d7700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc56415e220 con 0x7fc564158da0 2023-10-06T03:31:02.116 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:02.108+0000 7fc563fff700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc5640d03e0 0x7fc56415da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:31:02.116 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:02.108+0000 7fc563fff700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc5640d03e0 0x7fc56415da10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:38772/0 (socket says 172.21.15.110:38772) 2023-10-06T03:31:02.116 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:02.108+0000 7fc563fff700 1 -- 172.21.15.110:0/3344084368 learned_addr learned my addr 172.21.15.110:0/3344084368 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:31:02.116 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:02.108+0000 7fc5689d5700 1 --2- 172.21.15.110:0/3344084368 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc5640b5320 0x7fc564153860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:31:02.116 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:02.108+0000 7fc5691d6700 1 --2- 172.21.15.110:0/3344084368 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc564158da0 0x7fc5641690b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:31:02.117 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:02.108+0000 7fc563fff700 1 -- 172.21.15.110:0/3344084368 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc564158da0 msgr2=0x7fc5641690b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:31:02.117 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:02.108+0000 7fc563fff700 1 --2- 172.21.15.110:0/3344084368 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc564158da0 0x7fc5641690b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:02.117 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:02.108+0000 7fc563fff700 1 -- 172.21.15.110:0/3344084368 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc5640b5320 msgr2=0x7fc564153860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:31:02.117 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:02.108+0000 7fc563fff700 1 --2- 172.21.15.110:0/3344084368 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc5640b5320 0x7fc564153860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:02.117 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:02.108+0000 7fc563fff700 1 -- 172.21.15.110:0/3344084368 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc55c009420 con 0x7fc5640d03e0 2023-10-06T03:31:02.117 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:02.108+0000 7fc5691d6700 1 --2- 172.21.15.110:0/3344084368 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc564158da0 0x7fc5641690b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-06T03:31:02.117 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:02.112+0000 7fc563fff700 1 --2- 172.21.15.110:0/3344084368 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc5640d03e0 0x7fc56415da10 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7fc55400cb10 tx=0x7fc554002820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:31:02.117 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:02.112+0000 7fc560ff9700 1 -- 172.21.15.110:0/3344084368 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc55400a7e0 con 0x7fc5640d03e0 2023-10-06T03:31:02.117 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:02.112+0000 7fc560ff9700 1 -- 172.21.15.110:0/3344084368 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc55400a940 con 0x7fc5640d03e0 2023-10-06T03:31:02.117 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:02.112+0000 7fc5699d7700 1 -- 172.21.15.110:0/3344084368 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5640b2d10 con 0x7fc5640d03e0 2023-10-06T03:31:02.117 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:02.112+0000 7fc560ff9700 1 -- 172.21.15.110:0/3344084368 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc55400ad00 con 0x7fc5640d03e0 2023-10-06T03:31:02.117 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:02.112+0000 7fc5699d7700 1 -- 172.21.15.110:0/3344084368 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5640b32d0 con 0x7fc5640d03e0 2023-10-06T03:31:02.118 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:02.112+0000 7fc560ff9700 1 -- 172.21.15.110:0/3344084368 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 11) v1 ==== 79670+0+0 (secure 0 0 0) 0x7fc554012630 con 0x7fc5640d03e0 2023-10-06T03:31:02.118 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:02.112+0000 7fc5699d7700 1 -- 172.21.15.110:0/3344084368 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5640b34f0 con 0x7fc5640d03e0 2023-10-06T03:31:02.118 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:02.112+0000 7fc560ff9700 1 -- 172.21.15.110:0/3344084368 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 6937+0+0 (secure 0 0 0) 0x7fc55407cad0 con 0x7fc5640d03e0 2023-10-06T03:31:02.124 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:02.120+0000 7fc560ff9700 1 -- 172.21.15.110:0/3344084368 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fc55404cec0 con 0x7fc5640d03e0 2023-10-06T03:31:02.317 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:02.312+0000 7fc5699d7700 1 -- 172.21.15.110:0/3344084368 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7fc564071020 con 0x7fc5640d03e0 2023-10-06T03:31:02.319 INFO:tasks.ceph.mon.a.smithi110.stderr:2023-10-06T03:31:02.312+0000 7f5c90378700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-10-06T03:31:02.330 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:02.324+0000 7fc560ff9700 1 -- 172.21.15.110:0/3344084368 <== mon.0 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v17) v1 ==== 161+0+0 (secure 0 0 0) 0x7fc55400ae60 con 0x7fc5640d03e0 2023-10-06T03:31:02.331 INFO:tasks.ceph.mds.a.smithi110.stderr: -3653> 2023-10-06T03:30:37.592+0000 7fab283cf700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-06T03:31:02.332 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:02.328+0000 7fc5699d7700 1 -- 172.21.15.110:0/3344084368 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc5640d03e0 msgr2=0x7fc56415da10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:31:02.332 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:02.328+0000 7fc5699d7700 1 --2- 172.21.15.110:0/3344084368 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc5640d03e0 0x7fc56415da10 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7fc55400cb10 tx=0x7fc554002820 comp rx=0 tx=0).stop 2023-10-06T03:31:02.332 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:02.328+0000 7fc5699d7700 1 -- 172.21.15.110:0/3344084368 shutdown_connections 2023-10-06T03:31:02.332 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:02.328+0000 7fc5699d7700 1 --2- 172.21.15.110:0/3344084368 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc564158da0 0x7fc5641690b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:02.332 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:02.328+0000 7fc5699d7700 1 --2- 172.21.15.110:0/3344084368 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc5640d03e0 0x7fc56415da10 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:02.333 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:02.328+0000 7fc5699d7700 1 --2- 172.21.15.110:0/3344084368 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc5640b5320 0x7fc564153860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:02.333 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:02.328+0000 7fc5699d7700 1 -- 172.21.15.110:0/3344084368 >> 172.21.15.110:0/3344084368 conn(0x7fc56406c760 msgr2=0x7fc56406cb30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:31:02.333 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:02.328+0000 7fc5699d7700 1 -- 172.21.15.110:0/3344084368 shutdown_connections 2023-10-06T03:31:02.333 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:02.328+0000 7fc5699d7700 1 -- 172.21.15.110:0/3344084368 wait complete. 2023-10-06T03:31:02.333 INFO:teuthology.orchestra.run.smithi110.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2023-10-06T03:31:02.344 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2023-10-06T03:31:02.357 INFO:tasks.ceph.mds.a.smithi110.stdout:starting mds.a at 2023-10-06T03:31:02.446 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:02.440+0000 7f57aa136700 1 Processor -- start 2023-10-06T03:31:02.446 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:02.440+0000 7f57aa136700 1 -- start start 2023-10-06T03:31:02.448 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:02.444+0000 7f57aa136700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f57a4159140 0x7f57a406b6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:31:02.448 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:02.444+0000 7f57aa136700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57a406b240 con 0x7f57a4158d70 2023-10-06T03:31:02.448 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:02.444+0000 7f57aa136700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57a406a630 con 0x7f57a40cfc00 2023-10-06T03:31:02.448 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:02.444+0000 7f57aa136700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f57a406a910 con 0x7f57a4159140 2023-10-06T03:31:02.449 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:02.444+0000 7f57a2ffd700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f57a4159140 0x7f57a406b6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:31:02.449 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:02.444+0000 7f57a37fe700 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f57a4158d70 0x7f57a40ce7b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:60306/0 (socket says 172.21.15.110:60306) 2023-10-06T03:31:02.449 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:02.444+0000 7f57a37fe700 1 -- 172.21.15.110:0/917384839 learned_addr learned my addr 172.21.15.110:0/917384839 (peer_addr_for_me v1:172.21.15.110:0/0) 2023-10-06T03:31:02.449 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:02.444+0000 7f57a27fc700 1 -- 172.21.15.110:0/917384839 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3724368471 0 0) 0x7f57a406b240 con 0x7f57a4158d70 2023-10-06T03:31:02.449 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:02.444+0000 7f57a27fc700 1 -- 172.21.15.110:0/917384839 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f578c003570 con 0x7f57a4158d70 2023-10-06T03:31:02.449 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:02.444+0000 7f57a2ffd700 1 -- 172.21.15.110:0/917384839 >> v1:172.21.15.192:6789/0 conn(0x7f57a40cfc00 legacy=0x7f57a40d0040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:31:02.449 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:02.444+0000 7f57a2ffd700 1 -- 172.21.15.110:0/917384839 >> v1:172.21.15.110:6789/0 conn(0x7f57a4158d70 legacy=0x7f57a40ce7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:31:02.449 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:02.444+0000 7f57a27fc700 1 -- 172.21.15.110:0/917384839 <== mon.0 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 423995811 0 0) 0x7f578c003570 con 0x7f57a4158d70 2023-10-06T03:31:02.449 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:02.444+0000 7f57a2ffd700 1 -- 172.21.15.110:0/917384839 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57a406abf0 con 0x7f57a4159140 2023-10-06T03:31:02.450 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:02.444+0000 7f57a2ffd700 1 --2- 172.21.15.110:0/917384839 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f57a4159140 0x7f57a406b6d0 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f57980098e0 tx=0x7f57980091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=85f414331ee644b0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:31:02.451 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:02.444+0000 7f57a27fc700 1 -- 172.21.15.110:0/917384839 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5794001470 con 0x7f57a4159140 2023-10-06T03:31:02.451 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:02.444+0000 7f57a27fc700 1 -- 172.21.15.110:0/917384839 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5798011420 con 0x7f57a4159140 2023-10-06T03:31:02.451 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:02.444+0000 7f57a27fc700 1 -- 172.21.15.110:0/917384839 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57980117c0 con 0x7f57a4159140 2023-10-06T03:31:02.451 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:02.444+0000 7f57aa136700 1 -- 172.21.15.110:0/917384839 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f57a4159140 msgr2=0x7f57a406b6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:31:02.451 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:02.444+0000 7f57aa136700 1 --2- 172.21.15.110:0/917384839 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f57a4159140 0x7f57a406b6d0 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f57980098e0 tx=0x7f57980091a0 comp rx=0 tx=0).stop 2023-10-06T03:31:02.451 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:02.444+0000 7f57aa136700 1 -- 172.21.15.110:0/917384839 shutdown_connections 2023-10-06T03:31:02.451 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:02.444+0000 7f57aa136700 1 --2- 172.21.15.110:0/917384839 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f57a4159140 0x7f57a406b6d0 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:02.451 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:02.444+0000 7f57aa136700 1 -- 172.21.15.110:0/917384839 >> 172.21.15.110:0/917384839 conn(0x7f57a4068590 msgr2=0x7f57a4068990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:31:02.451 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:02.444+0000 7f57aa136700 1 -- 172.21.15.110:0/917384839 shutdown_connections 2023-10-06T03:31:02.451 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:02.444+0000 7f57aa136700 1 -- 172.21.15.110:0/917384839 wait complete. 2023-10-06T03:31:02.452 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:02.448+0000 7f57aa136700 1 Processor -- start 2023-10-06T03:31:02.452 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:02.448+0000 7f57aa136700 1 -- start start 2023-10-06T03:31:02.455 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:02.448+0000 7f57aa136700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f57a40cfc00 0x7f57a415c3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:31:02.455 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:02.448+0000 7f57aa136700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f57a4158d70 0x7f57a40b4550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:31:02.455 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:02.448+0000 7f57aa136700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f57a4159140 0x7f57a4154970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:31:02.455 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:02.448+0000 7f57aa136700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57a416a410 con 0x7f57a40cfc00 2023-10-06T03:31:02.455 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:02.448+0000 7f57aa136700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57a40b4a90 con 0x7f57a4159140 2023-10-06T03:31:02.455 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:02.448+0000 7f57aa136700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57a4154eb0 con 0x7f57a4158d70 2023-10-06T03:31:02.455 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:02.448+0000 7f57a37fe700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f57a40cfc00 0x7f57a415c3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:31:02.455 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:02.448+0000 7f57a37fe700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f57a40cfc00 0x7f57a415c3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:38786/0 (socket says 172.21.15.110:38786) 2023-10-06T03:31:02.455 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:02.448+0000 7f57a37fe700 1 -- 172.21.15.110:0/201494877 learned_addr learned my addr 172.21.15.110:0/201494877 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:31:02.455 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:02.448+0000 7f57a2ffd700 1 --2- 172.21.15.110:0/201494877 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f57a4158d70 0x7f57a40b4550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:31:02.456 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:02.448+0000 7f57a3fff700 1 --2- 172.21.15.110:0/201494877 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f57a4159140 0x7f57a4154970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:31:02.456 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:02.448+0000 7f57a37fe700 1 -- 172.21.15.110:0/201494877 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f57a4158d70 msgr2=0x7f57a40b4550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:31:02.456 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:02.448+0000 7f57a37fe700 1 --2- 172.21.15.110:0/201494877 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f57a4158d70 0x7f57a40b4550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:02.456 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:02.448+0000 7f57a37fe700 1 -- 172.21.15.110:0/201494877 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f57a4159140 msgr2=0x7f57a4154970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:31:02.456 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:02.448+0000 7f57a37fe700 1 --2- 172.21.15.110:0/201494877 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f57a4159140 0x7f57a4154970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:02.456 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:02.448+0000 7f57a37fe700 1 -- 172.21.15.110:0/201494877 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5798009420 con 0x7f57a40cfc00 2023-10-06T03:31:02.457 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:02.452+0000 7f57a37fe700 1 --2- 172.21.15.110:0/201494877 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f57a40cfc00 0x7f57a415c3e0 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f579400cb90 tx=0x7f5794001ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:31:02.457 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:02.452+0000 7f578b7fe700 1 -- 172.21.15.110:0/201494877 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5794007150 con 0x7f57a40cfc00 2023-10-06T03:31:02.457 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:02.452+0000 7f578b7fe700 1 -- 172.21.15.110:0/201494877 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57940072d0 con 0x7f57a40cfc00 2023-10-06T03:31:02.457 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:02.452+0000 7f57aa136700 1 -- 172.21.15.110:0/201494877 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57a4169ae0 con 0x7f57a40cfc00 2023-10-06T03:31:02.457 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:02.452+0000 7f578b7fe700 1 -- 172.21.15.110:0/201494877 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5794007690 con 0x7f57a40cfc00 2023-10-06T03:31:02.457 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:02.452+0000 7f57aa136700 1 -- 172.21.15.110:0/201494877 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57a416a0a0 con 0x7f57a40cfc00 2023-10-06T03:31:02.458 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:02.452+0000 7f578b7fe700 1 -- 172.21.15.110:0/201494877 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 11) v1 ==== 79670+0+0 (secure 0 0 0) 0x7f5794004a40 con 0x7f57a40cfc00 2023-10-06T03:31:02.458 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:02.452+0000 7f57aa136700 1 -- 172.21.15.110:0/201494877 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f57a4064770 con 0x7f57a40cfc00 2023-10-06T03:31:02.459 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:02.452+0000 7f578b7fe700 1 -- 172.21.15.110:0/201494877 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 7023+0+0 (secure 0 0 0) 0x7f579407bc50 con 0x7f57a40cfc00 2023-10-06T03:31:02.465 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:02.460+0000 7f578b7fe700 1 -- 172.21.15.110:0/201494877 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f579404c070 con 0x7f57a40cfc00 2023-10-06T03:31:02.651 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:02.644+0000 7f57aa136700 1 -- 172.21.15.110:0/201494877 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f57a406aed0 con 0x7f57a40cfc00 2023-10-06T03:31:02.948 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:02.944+0000 7f578b7fe700 1 -- 172.21.15.110:0/201494877 <== mon.0 v2:172.21.15.110:3300/0 7 ==== mgrmap(e 12) v1 ==== 79797+0+0 (secure 0 0 0) 0x7f579404bb20 con 0x7f57a40cfc00 2023-10-06T03:31:02.948 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:02.944+0000 7f578b7fe700 1 --2- 172.21.15.110:0/201494877 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f5790066970 0x7f5790068d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:31:02.949 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:02.944+0000 7f57a2ffd700 1 --2- 172.21.15.110:0/201494877 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f5790066970 0x7f5790068d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:31:02.952 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:02.948+0000 7f57a2ffd700 1 --2- 172.21.15.110:0/201494877 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f5790066970 0x7f5790068d50 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f57980091e0 tx=0x7f5798016000 comp rx=0 tx=0).ready entity=mgr.5505 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:31:03.330 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:03.324+0000 7f578b7fe700 1 -- 172.21.15.110:0/201494877 <== mon.0 v2:172.21.15.110:3300/0 8 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v18) v1 ==== 104+0+0 (secure 0 0 0) 0x7f57940786f0 con 0x7f57a40cfc00 2023-10-06T03:31:03.334 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:03.328+0000 7f57aa136700 1 -- 172.21.15.110:0/201494877 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f5790066970 msgr2=0x7f5790068d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:31:03.335 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:03.328+0000 7f57aa136700 1 --2- 172.21.15.110:0/201494877 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f5790066970 0x7f5790068d50 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f57980091e0 tx=0x7f5798016000 comp rx=0 tx=0).stop 2023-10-06T03:31:03.335 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:03.328+0000 7f57aa136700 1 -- 172.21.15.110:0/201494877 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f57a40cfc00 msgr2=0x7f57a415c3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:31:03.335 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:03.328+0000 7f57aa136700 1 --2- 172.21.15.110:0/201494877 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f57a40cfc00 0x7f57a415c3e0 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f579400cb90 tx=0x7f5794001ad0 comp rx=0 tx=0).stop 2023-10-06T03:31:03.335 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:03.328+0000 7f57aa136700 1 -- 172.21.15.110:0/201494877 shutdown_connections 2023-10-06T03:31:03.335 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:03.328+0000 7f57aa136700 1 --2- 172.21.15.110:0/201494877 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f5790066970 0x7f5790068d50 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:03.335 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:03.328+0000 7f57aa136700 1 --2- 172.21.15.110:0/201494877 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f57a4159140 0x7f57a4154970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:03.335 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:03.328+0000 7f57aa136700 1 --2- 172.21.15.110:0/201494877 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f57a4158d70 0x7f57a40b4550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:03.335 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:03.328+0000 7f57aa136700 1 --2- 172.21.15.110:0/201494877 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f57a40cfc00 0x7f57a415c3e0 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:03.335 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:03.328+0000 7f57aa136700 1 -- 172.21.15.110:0/201494877 >> 172.21.15.110:0/201494877 conn(0x7f57a4068590 msgr2=0x7f57a4157ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:31:03.335 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:03.328+0000 7f57aa136700 1 -- 172.21.15.110:0/201494877 shutdown_connections 2023-10-06T03:31:03.336 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:03.328+0000 7f57aa136700 1 -- 172.21.15.110:0/201494877 wait complete. 2023-10-06T03:31:03.348 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2023-10-06T03:31:03.464 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:03.460+0000 7f6c9aa93700 1 Processor -- start 2023-10-06T03:31:03.464 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:03.460+0000 7f6c9aa93700 1 -- start start 2023-10-06T03:31:03.466 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:03.460+0000 7f6c9aa93700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f6c940b01c0 0x7f6c940cc250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:31:03.466 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:03.460+0000 7f6c9aa93700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6c940cc790 0x7f6c940ca810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:31:03.466 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:03.460+0000 7f6c9aa93700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c940ca130 con 0x7f6c940cad50 2023-10-06T03:31:03.466 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:03.460+0000 7f6c9aa93700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6c940ccbe0 con 0x7f6c940b01c0 2023-10-06T03:31:03.466 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:03.460+0000 7f6c9aa93700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c940c9520 con 0x7f6c940cc790 2023-10-06T03:31:03.466 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:03.460+0000 7f6c9882f700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f6c940b01c0 0x7f6c940cc250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:31:03.466 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:03.460+0000 7f6c9882f700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f6c940b01c0 0x7f6c940cc250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:38800/0 (socket says 172.21.15.110:38800) 2023-10-06T03:31:03.467 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:03.460+0000 7f6c9882f700 1 -- 172.21.15.110:0/433827163 learned_addr learned my addr 172.21.15.110:0/433827163 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:31:03.467 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:03.460+0000 7f6c99030700 1 --1- 172.21.15.110:0/433827163 >> v1:172.21.15.110:6789/0 conn(0x7f6c940cad50 0x7f6c9415cb50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:60314/0 (socket says 172.21.15.110:60314) 2023-10-06T03:31:03.467 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:03.460+0000 7f6c93fff700 1 --2- 172.21.15.110:0/433827163 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6c940cc790 0x7f6c940ca810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:31:03.467 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:03.460+0000 7f6c937fe700 1 -- 172.21.15.110:0/433827163 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2425677214 0 0) 0x7f6c940ca130 con 0x7f6c940cad50 2023-10-06T03:31:03.467 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:03.460+0000 7f6c937fe700 1 -- 172.21.15.110:0/433827163 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6c84003570 con 0x7f6c940cad50 2023-10-06T03:31:03.467 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:03.460+0000 7f6c9882f700 1 -- 172.21.15.110:0/433827163 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6c940cc790 msgr2=0x7f6c940ca810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:31:03.467 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:03.460+0000 7f6c9882f700 1 --2- 172.21.15.110:0/433827163 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6c940cc790 0x7f6c940ca810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:03.467 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:03.460+0000 7f6c937fe700 1 -- 172.21.15.110:0/433827163 <== mon.0 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4144200052 0 0) 0x7f6c84003570 con 0x7f6c940cad50 2023-10-06T03:31:03.467 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:03.460+0000 7f6c9882f700 1 -- 172.21.15.110:0/433827163 >> v1:172.21.15.110:6789/0 conn(0x7f6c940cad50 legacy=0x7f6c9415cb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:31:03.467 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:03.460+0000 7f6c9882f700 1 -- 172.21.15.110:0/433827163 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c940c9800 con 0x7f6c940b01c0 2023-10-06T03:31:03.469 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:03.460+0000 7f6c9882f700 1 --2- 172.21.15.110:0/433827163 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6c940b01c0 0x7f6c940cc250 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f6c880099b0 tx=0x7f6c88009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b7b6c6d1a2b47f server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:31:03.469 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:03.460+0000 7f6c937fe700 1 -- 172.21.15.110:0/433827163 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c8c0017f0 con 0x7f6c940b01c0 2023-10-06T03:31:03.469 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:03.460+0000 7f6c937fe700 1 -- 172.21.15.110:0/433827163 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c88011420 con 0x7f6c940b01c0 2023-10-06T03:31:03.469 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:03.460+0000 7f6c937fe700 1 -- 172.21.15.110:0/433827163 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c880117a0 con 0x7f6c940b01c0 2023-10-06T03:31:03.469 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:03.460+0000 7f6c9aa93700 1 -- 172.21.15.110:0/433827163 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6c940b01c0 msgr2=0x7f6c940cc250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:31:03.469 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:03.460+0000 7f6c9aa93700 1 --2- 172.21.15.110:0/433827163 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6c940b01c0 0x7f6c940cc250 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f6c880099b0 tx=0x7f6c88009250 comp rx=0 tx=0).stop 2023-10-06T03:31:03.469 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:03.460+0000 7f6c9aa93700 1 -- 172.21.15.110:0/433827163 shutdown_connections 2023-10-06T03:31:03.469 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:03.460+0000 7f6c9aa93700 1 --2- 172.21.15.110:0/433827163 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6c940cc790 0x7f6c940ca810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:03.470 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:03.460+0000 7f6c9aa93700 1 --2- 172.21.15.110:0/433827163 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6c940b01c0 0x7f6c940cc250 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:03.470 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:03.460+0000 7f6c9aa93700 1 -- 172.21.15.110:0/433827163 >> 172.21.15.110:0/433827163 conn(0x7f6c94069d20 msgr2=0x7f6c9406c760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:31:03.470 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:03.464+0000 7f6c9aa93700 1 -- 172.21.15.110:0/433827163 shutdown_connections 2023-10-06T03:31:03.470 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:03.464+0000 7f6c9aa93700 1 -- 172.21.15.110:0/433827163 wait complete. 2023-10-06T03:31:03.470 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:03.464+0000 7f6c9aa93700 1 Processor -- start 2023-10-06T03:31:03.470 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:03.464+0000 7f6c9aa93700 1 -- start start 2023-10-06T03:31:03.472 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:03.464+0000 7f6c9aa93700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6c940b01c0 0x7f6c94156970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:31:03.472 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:03.464+0000 7f6c9aa93700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6c940cad50 0x7f6c9415c950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:31:03.472 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:03.464+0000 7f6c9aa93700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6c940cc790 0x7f6c9415ce90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:31:03.472 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:03.464+0000 7f6c9aa93700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c94169bf0 con 0x7f6c940b01c0 2023-10-06T03:31:03.472 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:03.464+0000 7f6c9aa93700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c94169e60 con 0x7f6c940cc790 2023-10-06T03:31:03.473 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:03.464+0000 7f6c9aa93700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c940c62f0 con 0x7f6c940cad50 2023-10-06T03:31:03.473 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:03.464+0000 7f6c9882f700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6c940b01c0 0x7f6c94156970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:31:03.473 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:03.464+0000 7f6c9882f700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6c940b01c0 0x7f6c94156970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:38806/0 (socket says 172.21.15.110:38806) 2023-10-06T03:31:03.473 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:03.464+0000 7f6c9882f700 1 -- 172.21.15.110:0/3215954662 learned_addr learned my addr 172.21.15.110:0/3215954662 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:31:03.473 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:03.464+0000 7f6c99030700 1 --2- 172.21.15.110:0/3215954662 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6c940cc790 0x7f6c9415ce90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:31:03.473 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:03.464+0000 7f6c93fff700 1 --2- 172.21.15.110:0/3215954662 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6c940cad50 0x7f6c9415c950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:31:03.473 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:03.464+0000 7f6c9882f700 1 -- 172.21.15.110:0/3215954662 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6c940cad50 msgr2=0x7f6c9415c950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:31:03.473 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:03.464+0000 7f6c9882f700 1 --2- 172.21.15.110:0/3215954662 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6c940cad50 0x7f6c9415c950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:03.473 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:03.464+0000 7f6c9882f700 1 -- 172.21.15.110:0/3215954662 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6c940cc790 msgr2=0x7f6c9415ce90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:31:03.473 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:03.464+0000 7f6c9882f700 1 --2- 172.21.15.110:0/3215954662 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6c940cc790 0x7f6c9415ce90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:03.473 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:03.464+0000 7f6c9882f700 1 -- 172.21.15.110:0/3215954662 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c880094f0 con 0x7f6c940b01c0 2023-10-06T03:31:03.473 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:03.464+0000 7f6c99030700 1 --2- 172.21.15.110:0/3215954662 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6c940cc790 0x7f6c9415ce90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-06T03:31:03.474 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:03.464+0000 7f6c9882f700 1 --2- 172.21.15.110:0/3215954662 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6c940b01c0 0x7f6c94156970 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f6c8800a010 tx=0x7f6c88016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:31:03.474 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:03.464+0000 7f6c90ff9700 1 -- 172.21.15.110:0/3215954662 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c8801cb80 con 0x7f6c940b01c0 2023-10-06T03:31:03.474 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:03.464+0000 7f6c90ff9700 1 -- 172.21.15.110:0/3215954662 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c8801cce0 con 0x7f6c940b01c0 2023-10-06T03:31:03.474 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:03.464+0000 7f6c9aa93700 1 -- 172.21.15.110:0/3215954662 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c940c5a50 con 0x7f6c940b01c0 2023-10-06T03:31:03.474 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:03.468+0000 7f6c90ff9700 1 -- 172.21.15.110:0/3215954662 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c8801a680 con 0x7f6c940b01c0 2023-10-06T03:31:03.474 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:03.468+0000 7f6c9aa93700 1 -- 172.21.15.110:0/3215954662 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c940c6010 con 0x7f6c940b01c0 2023-10-06T03:31:03.475 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:03.468+0000 7f6c9aa93700 1 -- 172.21.15.110:0/3215954662 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c9406d730 con 0x7f6c940b01c0 2023-10-06T03:31:03.475 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:03.468+0000 7f6c90ff9700 1 -- 172.21.15.110:0/3215954662 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 12) v1 ==== 79797+0+0 (secure 0 0 0) 0x7f6c8801aa30 con 0x7f6c940b01c0 2023-10-06T03:31:03.475 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:03.468+0000 7f6c90ff9700 1 --2- 172.21.15.110:0/3215954662 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f6c780603d0 0x7f6c78062880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:31:03.475 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:03.468+0000 7f6c90ff9700 1 -- 172.21.15.110:0/3215954662 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 7023+0+0 (secure 0 0 0) 0x7f6c8807e010 con 0x7f6c940b01c0 2023-10-06T03:31:03.475 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:03.468+0000 7f6c93fff700 1 --2- 172.21.15.110:0/3215954662 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f6c780603d0 0x7f6c78062880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:31:03.476 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:03.472+0000 7f6c93fff700 1 --2- 172.21.15.110:0/3215954662 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f6c780603d0 0x7f6c78062880 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f6c9415d5f0 tx=0x7f6c800091a0 comp rx=0 tx=0).ready entity=mgr.5505 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:31:03.479 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:03.472+0000 7f6c90ff9700 1 -- 172.21.15.110:0/3215954662 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f6c8804e390 con 0x7f6c940b01c0 2023-10-06T03:31:03.641 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:03.636+0000 7f6c9aa93700 1 -- 172.21.15.110:0/3215954662 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f6c94064770 con 0x7f6c940b01c0 2023-10-06T03:31:04.396 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:04.392+0000 7f6c90ff9700 1 -- 172.21.15.110:0/3215954662 <== mon.0 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v46) v1 ==== 181+0+0 (secure 0 0 0) 0x7f6c880519b0 con 0x7f6c940b01c0 2023-10-06T03:31:04.403 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:04.396+0000 7f6c9aa93700 1 -- 172.21.15.110:0/3215954662 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f6c780603d0 msgr2=0x7f6c78062880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:31:04.403 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:04.396+0000 7f6c9aa93700 1 --2- 172.21.15.110:0/3215954662 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f6c780603d0 0x7f6c78062880 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f6c9415d5f0 tx=0x7f6c800091a0 comp rx=0 tx=0).stop 2023-10-06T03:31:04.403 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:04.396+0000 7f6c9aa93700 1 -- 172.21.15.110:0/3215954662 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6c940b01c0 msgr2=0x7f6c94156970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:31:04.403 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:04.396+0000 7f6c9aa93700 1 --2- 172.21.15.110:0/3215954662 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6c940b01c0 0x7f6c94156970 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f6c8800a010 tx=0x7f6c88016000 comp rx=0 tx=0).stop 2023-10-06T03:31:04.403 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:04.396+0000 7f6c9aa93700 1 -- 172.21.15.110:0/3215954662 shutdown_connections 2023-10-06T03:31:04.403 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:04.396+0000 7f6c9aa93700 1 --2- 172.21.15.110:0/3215954662 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f6c780603d0 0x7f6c78062880 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:04.403 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:04.396+0000 7f6c9aa93700 1 --2- 172.21.15.110:0/3215954662 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6c940cc790 0x7f6c9415ce90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:04.404 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:04.396+0000 7f6c9aa93700 1 --2- 172.21.15.110:0/3215954662 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6c940cad50 0x7f6c9415c950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:04.404 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:04.396+0000 7f6c9aa93700 1 --2- 172.21.15.110:0/3215954662 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6c940b01c0 0x7f6c94156970 secure :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f6c8800a010 tx=0x7f6c88016000 comp rx=0 tx=0).stop 2023-10-06T03:31:04.404 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:04.396+0000 7f6c9aa93700 1 -- 172.21.15.110:0/3215954662 >> 172.21.15.110:0/3215954662 conn(0x7f6c94069d20 msgr2=0x7f6c940c3800 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:31:04.404 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:04.396+0000 7f6c9aa93700 1 -- 172.21.15.110:0/3215954662 shutdown_connections 2023-10-06T03:31:04.404 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:04.396+0000 7f6c9aa93700 1 -- 172.21.15.110:0/3215954662 wait complete. 2023-10-06T03:31:04.405 INFO:teuthology.orchestra.run.smithi110.stderr:pool 'cephfs_metadata' removed 2023-10-06T03:31:04.417 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2023-10-06T03:31:04.530 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:04.524+0000 7f1358eee700 1 Processor -- start 2023-10-06T03:31:04.530 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:04.524+0000 7f1358eee700 1 -- start start 2023-10-06T03:31:04.533 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:04.528+0000 7f1358eee700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f13540abfe0 0x7f1354069b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:31:04.533 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:04.528+0000 7f1358eee700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f135406ce70 0x7f135415aa00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:31:04.533 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:04.528+0000 7f1358eee700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1354069670 con 0x7f13540abc10 2023-10-06T03:31:04.533 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:04.528+0000 7f1358eee700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f135415b7a0 con 0x7f13540abfe0 2023-10-06T03:31:04.533 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:04.528+0000 7f1358eee700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f13540ad3c0 con 0x7f135406ce70 2023-10-06T03:31:04.533 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:04.528+0000 7f134affd700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f13540abfe0 0x7f1354069b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:31:04.533 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:04.528+0000 7f134affd700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f13540abfe0 0x7f1354069b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:38812/0 (socket says 172.21.15.110:38812) 2023-10-06T03:31:04.533 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:04.528+0000 7f134affd700 1 -- 172.21.15.110:0/2934824283 learned_addr learned my addr 172.21.15.110:0/2934824283 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:31:04.534 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:04.528+0000 7f1353fff700 1 --2- 172.21.15.110:0/2934824283 >> v2:172.21.15.192:3300/0 conn(0x7f135406ce70 0x7f135415aa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:31:04.535 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:04.528+0000 7f134affd700 1 -- 172.21.15.110:0/2934824283 >> v2:172.21.15.192:3300/0 conn(0x7f135406ce70 msgr2=0x7f135415aa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:31:04.535 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:04.528+0000 7f134affd700 1 --2- 172.21.15.110:0/2934824283 >> v2:172.21.15.192:3300/0 conn(0x7f135406ce70 0x7f135415aa00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:04.535 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:04.528+0000 7f134affd700 1 -- 172.21.15.110:0/2934824283 >> v1:172.21.15.192:6789/0 conn(0x7f13540abc10 legacy=0x7f135406c760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:31:04.535 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:04.528+0000 7f134affd700 1 -- 172.21.15.110:0/2934824283 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1354068d40 con 0x7f13540abfe0 2023-10-06T03:31:04.535 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:04.528+0000 7f134affd700 1 --2- 172.21.15.110:0/2934824283 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f13540abfe0 0x7f1354069b00 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f135406b450 tx=0x7f133c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b27690c44268317 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:31:04.535 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:04.528+0000 7f1352ffd700 1 -- 172.21.15.110:0/2934824283 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f133c009e70 con 0x7f13540abfe0 2023-10-06T03:31:04.536 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:04.528+0000 7f1352ffd700 1 -- 172.21.15.110:0/2934824283 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f133c00b8f0 con 0x7f13540abfe0 2023-10-06T03:31:04.536 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:04.528+0000 7f1352ffd700 1 -- 172.21.15.110:0/2934824283 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f133c00bc90 con 0x7f13540abfe0 2023-10-06T03:31:04.536 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:04.528+0000 7f1358eee700 1 -- 172.21.15.110:0/2934824283 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f13540abfe0 msgr2=0x7f1354069b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:31:04.536 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:04.528+0000 7f1358eee700 1 --2- 172.21.15.110:0/2934824283 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f13540abfe0 0x7f1354069b00 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f135406b450 tx=0x7f133c009250 comp rx=0 tx=0).stop 2023-10-06T03:31:04.536 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:04.528+0000 7f1358eee700 1 -- 172.21.15.110:0/2934824283 shutdown_connections 2023-10-06T03:31:04.536 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:04.528+0000 7f1358eee700 1 --2- 172.21.15.110:0/2934824283 >> v2:172.21.15.192:3300/0 conn(0x7f135406ce70 0x7f135415aa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:04.536 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:04.528+0000 7f1358eee700 1 --2- 172.21.15.110:0/2934824283 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f13540abfe0 0x7f1354069b00 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:04.536 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:04.528+0000 7f1358eee700 1 -- 172.21.15.110:0/2934824283 >> 172.21.15.110:0/2934824283 conn(0x7f135406b4c0 msgr2=0x7f135406b8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:31:04.536 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:04.528+0000 7f1358eee700 1 -- 172.21.15.110:0/2934824283 shutdown_connections 2023-10-06T03:31:04.536 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:04.528+0000 7f1358eee700 1 -- 172.21.15.110:0/2934824283 wait complete. 2023-10-06T03:31:04.537 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:04.532+0000 7f1358eee700 1 Processor -- start 2023-10-06T03:31:04.537 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:04.532+0000 7f1358eee700 1 -- start start 2023-10-06T03:31:04.538 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:04.532+0000 7f1358eee700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f135406ce70 0x7f13540a8e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:31:04.538 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:04.532+0000 7f1358eee700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f13540abc10 0x7f135414d0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:31:04.539 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:04.532+0000 7f1358eee700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f13540abfe0 0x7f13540adcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:31:04.539 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:04.532+0000 7f1358eee700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13540a9370 con 0x7f13540abfe0 2023-10-06T03:31:04.539 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:04.532+0000 7f1358eee700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13540a95d0 con 0x7f13540abc10 2023-10-06T03:31:04.539 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:04.532+0000 7f1358eee700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13540a98b0 con 0x7f135406ce70 2023-10-06T03:31:04.539 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:04.532+0000 7f1353fff700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f13540abfe0 0x7f13540adcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:31:04.539 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:04.532+0000 7f1353fff700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f13540abfe0 0x7f13540adcd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:38814/0 (socket says 172.21.15.110:38814) 2023-10-06T03:31:04.539 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:04.532+0000 7f13537fe700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f135406ce70 0x7f13540a8e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:31:04.539 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:04.532+0000 7f1353fff700 1 -- 172.21.15.110:0/877011814 learned_addr learned my addr 172.21.15.110:0/877011814 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:31:04.539 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:04.532+0000 7f134affd700 1 --2- 172.21.15.110:0/877011814 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f13540abc10 0x7f135414d0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:31:04.540 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:04.532+0000 7f1353fff700 1 -- 172.21.15.110:0/877011814 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f135406ce70 msgr2=0x7f13540a8e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:31:04.540 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:04.532+0000 7f1353fff700 1 --2- 172.21.15.110:0/877011814 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f135406ce70 0x7f13540a8e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:04.541 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:04.532+0000 7f1353fff700 1 -- 172.21.15.110:0/877011814 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f13540abc10 msgr2=0x7f135414d0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:31:04.541 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:04.532+0000 7f1353fff700 1 --2- 172.21.15.110:0/877011814 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f13540abc10 0x7f135414d0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:04.541 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:04.532+0000 7f1353fff700 1 -- 172.21.15.110:0/877011814 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f133c0094f0 con 0x7f13540abfe0 2023-10-06T03:31:04.541 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:04.532+0000 7f1353fff700 1 --2- 172.21.15.110:0/877011814 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f13540abfe0 0x7f13540adcd0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f134c009d40 tx=0x7f134c00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:31:04.541 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:04.536+0000 7f134bfff700 1 -- 172.21.15.110:0/877011814 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f134c009ea0 con 0x7f13540abfe0 2023-10-06T03:31:04.541 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:04.536+0000 7f134bfff700 1 -- 172.21.15.110:0/877011814 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f134c00f040 con 0x7f13540abfe0 2023-10-06T03:31:04.541 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:04.536+0000 7f1358eee700 1 -- 172.21.15.110:0/877011814 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13540a9b90 con 0x7f13540abfe0 2023-10-06T03:31:04.541 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:04.536+0000 7f134bfff700 1 -- 172.21.15.110:0/877011814 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f134c00ccf0 con 0x7f13540abfe0 2023-10-06T03:31:04.541 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:04.536+0000 7f1358eee700 1 -- 172.21.15.110:0/877011814 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13540aa150 con 0x7f13540abfe0 2023-10-06T03:31:04.542 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:04.536+0000 7f1358eee700 1 -- 172.21.15.110:0/877011814 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f13540aa340 con 0x7f13540abfe0 2023-10-06T03:31:04.542 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:04.536+0000 7f134bfff700 1 -- 172.21.15.110:0/877011814 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 12) v1 ==== 79797+0+0 (secure 0 0 0) 0x7f134c014690 con 0x7f13540abfe0 2023-10-06T03:31:04.543 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:04.536+0000 7f134bfff700 1 --2- 172.21.15.110:0/877011814 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f13340603a0 0x7f1334062850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:31:04.543 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:04.536+0000 7f134bfff700 1 -- 172.21.15.110:0/877011814 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 6585+0+0 (secure 0 0 0) 0x7f134c07d260 con 0x7f13540abfe0 2023-10-06T03:31:04.543 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:04.540+0000 7f13537fe700 1 --2- 172.21.15.110:0/877011814 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f13340603a0 0x7f1334062850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:31:04.544 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:04.540+0000 7f13537fe700 1 --2- 172.21.15.110:0/877011814 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f13340603a0 0x7f1334062850 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f13440039f0 tx=0x7f1344007ec0 comp rx=0 tx=0).ready entity=mgr.5505 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:31:04.549 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:04.544+0000 7f134bfff700 1 -- 172.21.15.110:0/877011814 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f134c04d790 con 0x7f13540abfe0 2023-10-06T03:31:04.711 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:04.708+0000 7f1358eee700 1 -- 172.21.15.110:0/877011814 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f1354069020 con 0x7f13540abfe0 2023-10-06T03:31:04.963 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:04.956+0000 7f134bfff700 1 -- 172.21.15.110:0/877011814 <== mon.0 v2:172.21.15.110:3300/0 7 ==== mgrmap(e 13) v1 ==== 80061+0+0 (secure 0 0 0) 0x7f134c04db60 con 0x7f13540abfe0 2023-10-06T03:31:05.406 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:05.400+0000 7f134bfff700 1 -- 172.21.15.110:0/877011814 <== mon.0 v2:172.21.15.110:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v47) v1 ==== 169+0+0 (secure 0 0 0) 0x7f134c051220 con 0x7f13540abfe0 2023-10-06T03:31:05.416 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:05.412+0000 7f1358eee700 1 -- 172.21.15.110:0/877011814 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f13340603a0 msgr2=0x7f1334062850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:31:05.417 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:05.412+0000 7f1358eee700 1 --2- 172.21.15.110:0/877011814 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f13340603a0 0x7f1334062850 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f13440039f0 tx=0x7f1344007ec0 comp rx=0 tx=0).stop 2023-10-06T03:31:05.417 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:05.412+0000 7f1358eee700 1 -- 172.21.15.110:0/877011814 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f13540abfe0 msgr2=0x7f13540adcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:31:05.417 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:05.412+0000 7f1358eee700 1 --2- 172.21.15.110:0/877011814 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f13540abfe0 0x7f13540adcd0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f134c009d40 tx=0x7f134c00b040 comp rx=0 tx=0).stop 2023-10-06T03:31:05.417 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:05.412+0000 7f1358eee700 1 -- 172.21.15.110:0/877011814 shutdown_connections 2023-10-06T03:31:05.418 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:05.412+0000 7f1358eee700 1 --2- 172.21.15.110:0/877011814 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f13340603a0 0x7f1334062850 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:05.418 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:05.412+0000 7f1358eee700 1 --2- 172.21.15.110:0/877011814 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f13540abfe0 0x7f13540adcd0 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:05.418 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:05.412+0000 7f1358eee700 1 --2- 172.21.15.110:0/877011814 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f13540abc10 0x7f135414d0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:05.418 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:05.412+0000 7f1358eee700 1 --2- 172.21.15.110:0/877011814 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f135406ce70 0x7f13540a8e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:05.418 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:05.412+0000 7f1358eee700 1 -- 172.21.15.110:0/877011814 >> 172.21.15.110:0/877011814 conn(0x7f135406b4c0 msgr2=0x7f13540ccf90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:31:05.418 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:05.412+0000 7f1358eee700 1 -- 172.21.15.110:0/877011814 shutdown_connections 2023-10-06T03:31:05.418 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:05.412+0000 7f1358eee700 1 -- 172.21.15.110:0/877011814 wait complete. 2023-10-06T03:31:05.419 INFO:teuthology.orchestra.run.smithi110.stderr:pool 'cephfs_data' removed 2023-10-06T03:31:05.431 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/allow_m_granularity 2023-10-06T03:31:05.537 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:05.532+0000 7f9505460700 1 Processor -- start 2023-10-06T03:31:05.537 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:05.532+0000 7f9505460700 1 -- start start 2023-10-06T03:31:05.538 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:05.532+0000 7f9505460700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f9500156c50 0x7f95000c06b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:31:05.538 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:05.532+0000 7f9505460700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f95000cb7b0 0x7f95000cbbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:31:05.539 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:05.532+0000 7f9505460700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f950006a7a0 con 0x7f95000b11c0 2023-10-06T03:31:05.539 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:05.532+0000 7f9505460700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f95000adcb0 con 0x7f9500156c50 2023-10-06T03:31:05.539 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:05.532+0000 7f9505460700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f95000cc130 con 0x7f95000cb7b0 2023-10-06T03:31:05.539 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:05.532+0000 7f94fffff700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f9500156c50 0x7f95000c06b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:31:05.539 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:05.532+0000 7f94fffff700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f9500156c50 0x7f95000c06b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:38822/0 (socket says 172.21.15.110:38822) 2023-10-06T03:31:05.539 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:05.532+0000 7f94fffff700 1 -- 172.21.15.110:0/574905454 learned_addr learned my addr 172.21.15.110:0/574905454 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:31:05.539 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:05.532+0000 7f9504c5f700 1 --2- 172.21.15.110:0/574905454 >> v2:172.21.15.192:3300/0 conn(0x7f95000cb7b0 0x7f95000cbbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:31:05.539 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:05.532+0000 7f94fffff700 1 -- 172.21.15.110:0/574905454 >> v2:172.21.15.192:3300/0 conn(0x7f95000cb7b0 msgr2=0x7f95000cbbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:31:05.539 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:05.532+0000 7f94fffff700 1 --2- 172.21.15.110:0/574905454 >> v2:172.21.15.192:3300/0 conn(0x7f95000cb7b0 0x7f95000cbbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:05.539 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:05.532+0000 7f94fffff700 1 -- 172.21.15.110:0/574905454 >> v1:172.21.15.192:6789/0 conn(0x7f95000b11c0 legacy=0x7f9500157020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:31:05.540 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:05.532+0000 7f94fffff700 1 -- 172.21.15.110:0/574905454 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95000adec0 con 0x7f9500156c50 2023-10-06T03:31:05.541 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:05.532+0000 7f94fffff700 1 --2- 172.21.15.110:0/574905454 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9500156c50 0x7f95000c06b0 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f95000697c0 tx=0x7f94f8009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb018d30d9b5b02d server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:31:05.541 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:05.532+0000 7f94feffd700 1 -- 172.21.15.110:0/574905454 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94f800e070 con 0x7f9500156c50 2023-10-06T03:31:05.541 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:05.536+0000 7f94feffd700 1 -- 172.21.15.110:0/574905454 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94f800ba20 con 0x7f9500156c50 2023-10-06T03:31:05.541 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:05.536+0000 7f94feffd700 1 -- 172.21.15.110:0/574905454 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94f80092e0 con 0x7f9500156c50 2023-10-06T03:31:05.541 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:05.536+0000 7f9505460700 1 -- 172.21.15.110:0/574905454 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9500156c50 msgr2=0x7f95000c06b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:31:05.541 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:05.536+0000 7f9505460700 1 --2- 172.21.15.110:0/574905454 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9500156c50 0x7f95000c06b0 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f95000697c0 tx=0x7f94f8009840 comp rx=0 tx=0).stop 2023-10-06T03:31:05.541 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:05.536+0000 7f9505460700 1 -- 172.21.15.110:0/574905454 shutdown_connections 2023-10-06T03:31:05.542 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:05.536+0000 7f9505460700 1 --2- 172.21.15.110:0/574905454 >> v2:172.21.15.192:3300/0 conn(0x7f95000cb7b0 0x7f95000cbbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:05.542 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:05.536+0000 7f9505460700 1 --2- 172.21.15.110:0/574905454 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9500156c50 0x7f95000c06b0 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:05.542 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:05.536+0000 7f9505460700 1 -- 172.21.15.110:0/574905454 >> 172.21.15.110:0/574905454 conn(0x7f950006c760 msgr2=0x7f950006cb60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:31:05.542 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:05.536+0000 7f9505460700 1 -- 172.21.15.110:0/574905454 shutdown_connections 2023-10-06T03:31:05.542 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:05.536+0000 7f9505460700 1 -- 172.21.15.110:0/574905454 wait complete. 2023-10-06T03:31:05.542 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:05.536+0000 7f9505460700 1 Processor -- start 2023-10-06T03:31:05.543 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:05.536+0000 7f9505460700 1 -- start start 2023-10-06T03:31:05.544 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:05.536+0000 7f9505460700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f95000b11c0 0x7f950015f930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:31:05.544 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:05.536+0000 7f9505460700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f95000cb7b0 0x7f9500159070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:31:05.544 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:05.540+0000 7f9505460700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9500156c50 0x7f950015b070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:31:05.544 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:05.540+0000 7f9505460700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9500157020 con 0x7f95000cb7b0 2023-10-06T03:31:05.544 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:05.540+0000 7f9505460700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95001599e0 con 0x7f9500156c50 2023-10-06T03:31:05.544 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:05.540+0000 7f9505460700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9500159cc0 con 0x7f95000b11c0 2023-10-06T03:31:05.544 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:05.540+0000 7f94ff7fe700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f95000cb7b0 0x7f9500159070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:31:05.545 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:05.540+0000 7f94ff7fe700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f95000cb7b0 0x7f9500159070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:38828/0 (socket says 172.21.15.110:38828) 2023-10-06T03:31:05.545 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:05.540+0000 7f94ff7fe700 1 -- 172.21.15.110:0/526774803 learned_addr learned my addr 172.21.15.110:0/526774803 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:31:05.545 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:05.540+0000 7f9504c5f700 1 --2- 172.21.15.110:0/526774803 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9500156c50 0x7f950015b070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:31:05.545 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:05.540+0000 7f94fffff700 1 --2- 172.21.15.110:0/526774803 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f95000b11c0 0x7f950015f930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:31:05.545 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:05.540+0000 7f94ff7fe700 1 -- 172.21.15.110:0/526774803 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f95000b11c0 msgr2=0x7f950015f930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:31:05.545 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:05.540+0000 7f94ff7fe700 1 --2- 172.21.15.110:0/526774803 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f95000b11c0 0x7f950015f930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:05.545 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:05.540+0000 7f94ff7fe700 1 -- 172.21.15.110:0/526774803 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9500156c50 msgr2=0x7f950015b070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:31:05.545 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:05.540+0000 7f94ff7fe700 1 --2- 172.21.15.110:0/526774803 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9500156c50 0x7f950015b070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:05.545 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:05.540+0000 7f94ff7fe700 1 -- 172.21.15.110:0/526774803 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94f8009ae0 con 0x7f95000cb7b0 2023-10-06T03:31:05.546 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:05.540+0000 7f94ff7fe700 1 --2- 172.21.15.110:0/526774803 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f95000cb7b0 0x7f9500159070 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f94f400c590 tx=0x7f94f400c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:31:05.546 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:05.540+0000 7f94e7fff700 1 -- 172.21.15.110:0/526774803 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94f4011590 con 0x7f95000cb7b0 2023-10-06T03:31:05.546 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:05.540+0000 7f9505460700 1 -- 172.21.15.110:0/526774803 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9500159fa0 con 0x7f95000cb7b0 2023-10-06T03:31:05.547 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:05.540+0000 7f94e7fff700 1 -- 172.21.15.110:0/526774803 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94f4011710 con 0x7f95000cb7b0 2023-10-06T03:31:05.547 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:05.540+0000 7f94e7fff700 1 -- 172.21.15.110:0/526774803 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94f4011ad0 con 0x7f95000cb7b0 2023-10-06T03:31:05.547 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:05.540+0000 7f9505460700 1 -- 172.21.15.110:0/526774803 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f950015fe70 con 0x7f95000cb7b0 2023-10-06T03:31:05.547 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:05.540+0000 7f9505460700 1 -- 172.21.15.110:0/526774803 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9500153fa0 con 0x7f95000cb7b0 2023-10-06T03:31:05.547 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:05.544+0000 7f94e7fff700 1 -- 172.21.15.110:0/526774803 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 13) v1 ==== 80061+0+0 (secure 0 0 0) 0x7f94f401b3b0 con 0x7f95000cb7b0 2023-10-06T03:31:05.548 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:05.544+0000 7f94e7fff700 1 --2- 172.21.15.110:0/526774803 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f94ec060770 0x7f94ec062c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:31:05.548 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:05.544+0000 7f94e7fff700 1 -- 172.21.15.110:0/526774803 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 6171+0+0 (secure 0 0 0) 0x7f94f407beb0 con 0x7f95000cb7b0 2023-10-06T03:31:05.548 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:05.544+0000 7f94fffff700 1 --2- 172.21.15.110:0/526774803 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f94ec060770 0x7f94ec062c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:31:05.548 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:05.544+0000 7f94fffff700 1 --2- 172.21.15.110:0/526774803 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f94ec060770 0x7f94ec062c20 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f94f8009880 tx=0x7f94f8018000 comp rx=0 tx=0).ready entity=mgr.5505 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:31:05.556 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:05.552+0000 7f94e7fff700 1 -- 172.21.15.110:0/526774803 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f9500153fa0 con 0x7f95000cb7b0 2023-10-06T03:31:05.724 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:05.720+0000 7f9505460700 1 -- 172.21.15.110:0/526774803 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"} v 0) v1 -- 0x7f9500153fa0 con 0x7f95000cb7b0 2023-10-06T03:31:05.731 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:05.728+0000 7f94e7fff700 1 -- 172.21.15.110:0/526774803 <== mon.0 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"}]=0 v13) v1 ==== 120+0+0 (secure 0 0 0) 0x7f9500153fa0 con 0x7f95000cb7b0 2023-10-06T03:31:05.735 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:05.728+0000 7f9505460700 1 -- 172.21.15.110:0/526774803 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f94ec060770 msgr2=0x7f94ec062c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:31:05.735 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:05.728+0000 7f9505460700 1 --2- 172.21.15.110:0/526774803 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f94ec060770 0x7f94ec062c20 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f94f8009880 tx=0x7f94f8018000 comp rx=0 tx=0).stop 2023-10-06T03:31:05.735 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:05.728+0000 7f9505460700 1 -- 172.21.15.110:0/526774803 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f95000cb7b0 msgr2=0x7f9500159070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:31:05.735 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:05.728+0000 7f9505460700 1 --2- 172.21.15.110:0/526774803 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f95000cb7b0 0x7f9500159070 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f94f400c590 tx=0x7f94f400c8a0 comp rx=0 tx=0).stop 2023-10-06T03:31:05.736 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:05.728+0000 7f9505460700 1 -- 172.21.15.110:0/526774803 shutdown_connections 2023-10-06T03:31:05.736 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:05.728+0000 7f9505460700 1 --2- 172.21.15.110:0/526774803 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f94ec060770 0x7f94ec062c20 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:05.736 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:05.728+0000 7f9505460700 1 --2- 172.21.15.110:0/526774803 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9500156c50 0x7f950015b070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:05.736 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:05.728+0000 7f9505460700 1 --2- 172.21.15.110:0/526774803 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f95000cb7b0 0x7f9500159070 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:05.736 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:05.728+0000 7f9505460700 1 --2- 172.21.15.110:0/526774803 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f95000b11c0 0x7f950015f930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:05.736 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:05.728+0000 7f9505460700 1 -- 172.21.15.110:0/526774803 >> 172.21.15.110:0/526774803 conn(0x7f950006c760 msgr2=0x7f95001556a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:31:05.736 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:05.728+0000 7f9505460700 1 -- 172.21.15.110:0/526774803 shutdown_connections 2023-10-06T03:31:05.737 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:05.728+0000 7f9505460700 1 -- 172.21.15.110:0/526774803 wait complete. 2023-10-06T03:31:05.751 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2023-10-06T03:31:05.858 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:05.852+0000 7f9de7eb2700 1 Processor -- start 2023-10-06T03:31:05.859 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:05.852+0000 7f9de7eb2700 1 -- start start 2023-10-06T03:31:05.860 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:05.856+0000 7f9de7eb2700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9de0159140 0x7f9de006b6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:31:05.860 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:05.856+0000 7f9de7eb2700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9de006b240 con 0x7f9de00cfc00 2023-10-06T03:31:05.860 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:05.856+0000 7f9de7eb2700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9de006a630 con 0x7f9de0158d70 2023-10-06T03:31:05.860 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:05.856+0000 7f9de7eb2700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9de006a910 con 0x7f9de0159140 2023-10-06T03:31:05.860 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:05.856+0000 7f9de644f700 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f9de00cfc00 0x7f9de00d0080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:60318/0 (socket says 172.21.15.110:60318) 2023-10-06T03:31:05.860 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:05.856+0000 7f9de644f700 1 -- 172.21.15.110:0/1075570499 learned_addr learned my addr 172.21.15.110:0/1075570499 (peer_addr_for_me v1:172.21.15.110:0/0) 2023-10-06T03:31:05.862 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:05.856+0000 7f9de544d700 1 --2- 172.21.15.110:0/1075570499 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9de0159140 0x7f9de006b6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:31:05.863 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:05.856+0000 7f9de4c4c700 1 -- 172.21.15.110:0/1075570499 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2448722216 0 0) 0x7f9de006b240 con 0x7f9de00cfc00 2023-10-06T03:31:05.863 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:05.856+0000 7f9de4c4c700 1 -- 172.21.15.110:0/1075570499 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9dcc003570 con 0x7f9de00cfc00 2023-10-06T03:31:05.863 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:05.856+0000 7f9de4c4c700 1 -- 172.21.15.110:0/1075570499 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4071047730 0 0) 0x7f9de006a630 con 0x7f9de0158d70 2023-10-06T03:31:05.863 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:05.856+0000 7f9de4c4c700 1 -- 172.21.15.110:0/1075570499 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9de006b240 con 0x7f9de0158d70 2023-10-06T03:31:05.863 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:05.856+0000 7f9de4c4c700 1 -- 172.21.15.110:0/1075570499 <== mon.0 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1187977618 0 0) 0x7f9dcc003570 con 0x7f9de00cfc00 2023-10-06T03:31:05.863 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:05.856+0000 7f9de544d700 1 -- 172.21.15.110:0/1075570499 >> v1:172.21.15.192:6789/0 conn(0x7f9de0158d70 legacy=0x7f9de00ce7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:31:05.863 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:05.856+0000 7f9de544d700 1 -- 172.21.15.110:0/1075570499 >> v1:172.21.15.110:6789/0 conn(0x7f9de00cfc00 legacy=0x7f9de00d0080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:31:05.863 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:05.856+0000 7f9de544d700 1 -- 172.21.15.110:0/1075570499 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9de006abf0 con 0x7f9de0159140 2023-10-06T03:31:05.863 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:05.856+0000 7f9de544d700 1 --2- 172.21.15.110:0/1075570499 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9de0159140 0x7f9de006b6d0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f9dd0009910 tx=0x7f9dd00091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=920952a8fc00c131 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:31:05.863 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:05.856+0000 7f9de4c4c700 1 -- 172.21.15.110:0/1075570499 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9dd80017f0 con 0x7f9de0159140 2023-10-06T03:31:05.864 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:05.856+0000 7f9de4c4c700 1 -- 172.21.15.110:0/1075570499 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9dd0011420 con 0x7f9de0159140 2023-10-06T03:31:05.864 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:05.856+0000 7f9de4c4c700 1 -- 172.21.15.110:0/1075570499 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9dd00117c0 con 0x7f9de0159140 2023-10-06T03:31:05.864 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:05.856+0000 7f9de7eb2700 1 -- 172.21.15.110:0/1075570499 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9de0159140 msgr2=0x7f9de006b6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:31:05.864 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:05.856+0000 7f9de7eb2700 1 --2- 172.21.15.110:0/1075570499 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9de0159140 0x7f9de006b6d0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f9dd0009910 tx=0x7f9dd00091a0 comp rx=0 tx=0).stop 2023-10-06T03:31:05.864 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:05.856+0000 7f9de7eb2700 1 -- 172.21.15.110:0/1075570499 shutdown_connections 2023-10-06T03:31:05.864 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:05.856+0000 7f9de7eb2700 1 --2- 172.21.15.110:0/1075570499 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9de0159140 0x7f9de006b6d0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:05.864 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:05.856+0000 7f9de7eb2700 1 -- 172.21.15.110:0/1075570499 >> 172.21.15.110:0/1075570499 conn(0x7f9de0068590 msgr2=0x7f9de0068990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:31:05.864 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:05.856+0000 7f9de7eb2700 1 -- 172.21.15.110:0/1075570499 shutdown_connections 2023-10-06T03:31:05.864 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:05.856+0000 7f9de7eb2700 1 -- 172.21.15.110:0/1075570499 wait complete. 2023-10-06T03:31:05.865 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:05.860+0000 7f9de7eb2700 1 Processor -- start 2023-10-06T03:31:05.865 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:05.860+0000 7f9de7eb2700 1 -- start start 2023-10-06T03:31:05.867 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:05.860+0000 7f9de7eb2700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9de00cfc00 0x7f9de015cce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:31:05.867 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:05.860+0000 7f9de7eb2700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9de0158d70 0x7f9de0159f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:31:05.867 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:05.860+0000 7f9de7eb2700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9de0159140 0x7f9de015a4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:31:05.867 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:05.860+0000 7f9de7eb2700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9de015b6e0 con 0x7f9de0159140 2023-10-06T03:31:05.867 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:05.860+0000 7f9de7eb2700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9de00b4120 con 0x7f9de00cfc00 2023-10-06T03:31:05.867 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:05.860+0000 7f9de7eb2700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9de015ab60 con 0x7f9de0158d70 2023-10-06T03:31:05.867 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:05.860+0000 7f9de644f700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9de0159140 0x7f9de015a4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:31:05.868 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:05.860+0000 7f9de644f700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9de0159140 0x7f9de015a4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:38842/0 (socket says 172.21.15.110:38842) 2023-10-06T03:31:05.868 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:05.860+0000 7f9de5c4e700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9de00cfc00 0x7f9de015cce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:31:05.868 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:05.860+0000 7f9de644f700 1 -- 172.21.15.110:0/2009007456 learned_addr learned my addr 172.21.15.110:0/2009007456 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:31:05.868 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:05.860+0000 7f9de544d700 1 --2- 172.21.15.110:0/2009007456 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9de0158d70 0x7f9de0159f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:31:05.868 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:05.860+0000 7f9de644f700 1 -- 172.21.15.110:0/2009007456 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9de0158d70 msgr2=0x7f9de0159f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:31:05.868 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:05.860+0000 7f9de644f700 1 --2- 172.21.15.110:0/2009007456 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9de0158d70 0x7f9de0159f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:05.868 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:05.860+0000 7f9de644f700 1 -- 172.21.15.110:0/2009007456 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9de00cfc00 msgr2=0x7f9de015cce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:31:05.868 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:05.860+0000 7f9de644f700 1 --2- 172.21.15.110:0/2009007456 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9de00cfc00 0x7f9de015cce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:05.868 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:05.860+0000 7f9de644f700 1 -- 172.21.15.110:0/2009007456 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9dd0009420 con 0x7f9de0159140 2023-10-06T03:31:05.868 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:05.860+0000 7f9de644f700 1 --2- 172.21.15.110:0/2009007456 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9de0159140 0x7f9de015a4c0 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f9dd800ccf0 tx=0x7f9dd8007d20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:31:05.869 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:05.860+0000 7f9dd5ffb700 1 -- 172.21.15.110:0/2009007456 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9dd8004dc0 con 0x7f9de0159140 2023-10-06T03:31:05.869 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:05.860+0000 7f9dd5ffb700 1 -- 172.21.15.110:0/2009007456 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9dd800a6d0 con 0x7f9de0159140 2023-10-06T03:31:05.869 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:05.860+0000 7f9de7eb2700 1 -- 172.21.15.110:0/2009007456 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9de015b120 con 0x7f9de0159140 2023-10-06T03:31:05.869 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:05.860+0000 7f9dd5ffb700 1 -- 172.21.15.110:0/2009007456 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9dd800aa90 con 0x7f9de0159140 2023-10-06T03:31:05.869 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:05.860+0000 7f9de7eb2700 1 -- 172.21.15.110:0/2009007456 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9de00c7470 con 0x7f9de0159140 2023-10-06T03:31:05.869 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:05.864+0000 7f9de7eb2700 1 -- 172.21.15.110:0/2009007456 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9de00c76e0 con 0x7f9de0159140 2023-10-06T03:31:05.870 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:05.864+0000 7f9dd5ffb700 1 -- 172.21.15.110:0/2009007456 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 13) v1 ==== 80061+0+0 (secure 0 0 0) 0x7f9dd8007150 con 0x7f9de0159140 2023-10-06T03:31:05.871 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:05.864+0000 7f9dd5ffb700 1 --2- 172.21.15.110:0/2009007456 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f9dc40606a0 0x7f9dc4062b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:31:05.871 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:05.864+0000 7f9de5c4e700 1 --2- 172.21.15.110:0/2009007456 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f9dc40606a0 0x7f9dc4062b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:31:05.871 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:05.864+0000 7f9dd5ffb700 1 -- 172.21.15.110:0/2009007456 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 6171+0+0 (secure 0 0 0) 0x7f9dd807c6e0 con 0x7f9de0159140 2023-10-06T03:31:05.871 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:05.868+0000 7f9de5c4e700 1 --2- 172.21.15.110:0/2009007456 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f9dc40606a0 0x7f9dc4062b50 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f9ddc007fd0 tx=0x7f9ddc007d50 comp rx=0 tx=0).ready entity=mgr.5505 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:31:05.879 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:05.876+0000 7f9dd5ffb700 1 -- 172.21.15.110:0/2009007456 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f9dd804c9d0 con 0x7f9de0159140 2023-10-06T03:31:06.044 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:06.040+0000 7f9de7eb2700 1 -- 172.21.15.110:0/2009007456 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f9de0064770 con 0x7f9de0159140 2023-10-06T03:31:06.053 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:06.048+0000 7f9dd5ffb700 1 -- 172.21.15.110:0/2009007456 <== mon.0 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v14) v1 ==== 104+0+0 (secure 0 0 0) 0x7f9dd804fff0 con 0x7f9de0159140 2023-10-06T03:31:06.056 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:06.052+0000 7f9de7eb2700 1 -- 172.21.15.110:0/2009007456 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f9dc40606a0 msgr2=0x7f9dc4062b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:31:06.056 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:06.052+0000 7f9de7eb2700 1 --2- 172.21.15.110:0/2009007456 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f9dc40606a0 0x7f9dc4062b50 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f9ddc007fd0 tx=0x7f9ddc007d50 comp rx=0 tx=0).stop 2023-10-06T03:31:06.056 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:06.052+0000 7f9de7eb2700 1 -- 172.21.15.110:0/2009007456 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9de0159140 msgr2=0x7f9de015a4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:31:06.056 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:06.052+0000 7f9de7eb2700 1 --2- 172.21.15.110:0/2009007456 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9de0159140 0x7f9de015a4c0 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f9dd800ccf0 tx=0x7f9dd8007d20 comp rx=0 tx=0).stop 2023-10-06T03:31:06.057 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:06.052+0000 7f9de7eb2700 1 -- 172.21.15.110:0/2009007456 shutdown_connections 2023-10-06T03:31:06.057 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:06.052+0000 7f9de7eb2700 1 --2- 172.21.15.110:0/2009007456 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f9dc40606a0 0x7f9dc4062b50 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:06.057 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:06.052+0000 7f9de7eb2700 1 --2- 172.21.15.110:0/2009007456 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9de0159140 0x7f9de015a4c0 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:06.057 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:06.052+0000 7f9de7eb2700 1 --2- 172.21.15.110:0/2009007456 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9de0158d70 0x7f9de0159f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:06.057 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:06.052+0000 7f9de7eb2700 1 --2- 172.21.15.110:0/2009007456 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9de00cfc00 0x7f9de015cce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:06.057 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:06.052+0000 7f9de7eb2700 1 -- 172.21.15.110:0/2009007456 >> 172.21.15.110:0/2009007456 conn(0x7f9de0068590 msgr2=0x7f9de0157f60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:31:06.057 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:06.052+0000 7f9de7eb2700 1 -- 172.21.15.110:0/2009007456 shutdown_connections 2023-10-06T03:31:06.057 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:06.052+0000 7f9de7eb2700 1 -- 172.21.15.110:0/2009007456 wait complete. 2023-10-06T03:31:06.069 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/dump_on_update 2023-10-06T03:31:06.179 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:06.172+0000 7f1b049ac700 1 Processor -- start 2023-10-06T03:31:06.179 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:06.172+0000 7f1b049ac700 1 -- start start 2023-10-06T03:31:06.180 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:06.176+0000 7f1b049ac700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f1b000b11c0 0x7f1b000c06b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:31:06.180 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:06.176+0000 7f1b049ac700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1b000cb7b0 0x7f1b000cbc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:31:06.180 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:06.176+0000 7f1b049ac700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b0006a7a0 con 0x7f1b00156c50 2023-10-06T03:31:06.180 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:06.176+0000 7f1b049ac700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1b000adcb0 con 0x7f1b000b11c0 2023-10-06T03:31:06.180 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:06.176+0000 7f1b049ac700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b000cc170 con 0x7f1b000cb7b0 2023-10-06T03:31:06.181 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:06.176+0000 7f1afeffd700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f1b000b11c0 0x7f1b000c06b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:31:06.181 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:06.176+0000 7f1afeffd700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f1b000b11c0 0x7f1b000c06b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.110:55188/0 (socket says 172.21.15.110:55188) 2023-10-06T03:31:06.181 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:06.176+0000 7f1afeffd700 1 -- 172.21.15.110:0/837528856 learned_addr learned my addr 172.21.15.110:0/837528856 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:31:06.181 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:06.176+0000 7f1afffff700 1 --2- 172.21.15.110:0/837528856 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1b000cb7b0 0x7f1b000cbc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:31:06.182 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:06.176+0000 7f1afeffd700 1 -- 172.21.15.110:0/837528856 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1b000cb7b0 msgr2=0x7f1b000cbc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:31:06.182 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:06.176+0000 7f1afeffd700 1 --2- 172.21.15.110:0/837528856 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1b000cb7b0 0x7f1b000cbc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:06.182 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:06.176+0000 7f1afeffd700 1 -- 172.21.15.110:0/837528856 >> v1:172.21.15.192:6789/0 conn(0x7f1b00156c50 legacy=0x7f1b00157020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:31:06.182 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:06.176+0000 7f1afeffd700 1 -- 172.21.15.110:0/837528856 --> v2:172.21.15.192:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b000adec0 con 0x7f1b000b11c0 2023-10-06T03:31:06.182 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:06.176+0000 7f1afeffd700 1 --2- 172.21.15.110:0/837528856 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1b000b11c0 0x7f1b000c06b0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f1af00099b0 tx=0x7f1af0009250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=890e7a2f86b7f85c server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:31:06.183 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:06.176+0000 7f1afe7fc700 1 -- 172.21.15.110:0/837528856 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1af0009ea0 con 0x7f1b000b11c0 2023-10-06T03:31:06.183 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:06.176+0000 7f1afe7fc700 1 -- 172.21.15.110:0/837528856 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1af000b860 con 0x7f1b000b11c0 2023-10-06T03:31:06.183 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:06.176+0000 7f1afe7fc700 1 -- 172.21.15.110:0/837528856 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1af000bc00 con 0x7f1b000b11c0 2023-10-06T03:31:06.183 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:06.176+0000 7f1b049ac700 1 -- 172.21.15.110:0/837528856 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1b000b11c0 msgr2=0x7f1b000c06b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:31:06.183 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:06.176+0000 7f1b049ac700 1 --2- 172.21.15.110:0/837528856 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1b000b11c0 0x7f1b000c06b0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f1af00099b0 tx=0x7f1af0009250 comp rx=0 tx=0).stop 2023-10-06T03:31:06.183 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:06.176+0000 7f1b049ac700 1 -- 172.21.15.110:0/837528856 shutdown_connections 2023-10-06T03:31:06.184 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:06.176+0000 7f1b049ac700 1 --2- 172.21.15.110:0/837528856 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1b000cb7b0 0x7f1b000cbc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:06.184 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:06.176+0000 7f1b049ac700 1 --2- 172.21.15.110:0/837528856 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1b000b11c0 0x7f1b000c06b0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:06.184 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:06.176+0000 7f1b049ac700 1 -- 172.21.15.110:0/837528856 >> 172.21.15.110:0/837528856 conn(0x7f1b0006c760 msgr2=0x7f1b0006cb60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:31:06.184 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:06.176+0000 7f1b049ac700 1 -- 172.21.15.110:0/837528856 shutdown_connections 2023-10-06T03:31:06.184 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:06.176+0000 7f1b049ac700 1 -- 172.21.15.110:0/837528856 wait complete. 2023-10-06T03:31:06.185 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:06.180+0000 7f1b049ac700 1 Processor -- start 2023-10-06T03:31:06.185 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:06.180+0000 7f1b049ac700 1 -- start start 2023-10-06T03:31:06.185 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:06.180+0000 7f1b049ac700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1b000b11c0 0x7f1b000c65a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:31:06.186 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:06.180+0000 7f1b049ac700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1b000cb7b0 0x7f1b00145ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:31:06.186 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:06.180+0000 7f1b049ac700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1b00156c50 0x7f1b00146530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:31:06.186 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:06.180+0000 7f1b049ac700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b000c9950 con 0x7f1b00156c50 2023-10-06T03:31:06.186 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:06.180+0000 7f1b049ac700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b000c9ac0 con 0x7f1b000cb7b0 2023-10-06T03:31:06.186 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:06.180+0000 7f1b049ac700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b0014a640 con 0x7f1b000b11c0 2023-10-06T03:31:06.186 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:06.180+0000 7f1afffff700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1b00156c50 0x7f1b00146530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:31:06.186 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:06.180+0000 7f1afffff700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1b00156c50 0x7f1b00146530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:38850/0 (socket says 172.21.15.110:38850) 2023-10-06T03:31:06.186 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:06.180+0000 7f1afffff700 1 -- 172.21.15.110:0/612182494 learned_addr learned my addr 172.21.15.110:0/612182494 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:31:06.186 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:06.180+0000 7f1afeffd700 1 --2- 172.21.15.110:0/612182494 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1b000cb7b0 0x7f1b00145ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:31:06.186 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:06.180+0000 7f1aff7fe700 1 --2- 172.21.15.110:0/612182494 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1b000b11c0 0x7f1b000c65a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:31:06.188 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:06.180+0000 7f1afffff700 1 -- 172.21.15.110:0/612182494 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1b000b11c0 msgr2=0x7f1b000c65a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:31:06.188 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:06.180+0000 7f1afffff700 1 --2- 172.21.15.110:0/612182494 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1b000b11c0 0x7f1b000c65a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:06.188 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:06.180+0000 7f1afffff700 1 -- 172.21.15.110:0/612182494 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1b000cb7b0 msgr2=0x7f1b00145ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:31:06.188 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:06.180+0000 7f1afffff700 1 --2- 172.21.15.110:0/612182494 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1b000cb7b0 0x7f1b00145ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:06.188 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:06.180+0000 7f1afffff700 1 -- 172.21.15.110:0/612182494 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1af00094f0 con 0x7f1b00156c50 2023-10-06T03:31:06.188 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:06.180+0000 7f1aff7fe700 1 --2- 172.21.15.110:0/612182494 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1b000b11c0 0x7f1b000c65a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-06T03:31:06.188 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:06.180+0000 7f1afffff700 1 --2- 172.21.15.110:0/612182494 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1b00156c50 0x7f1b00146530 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f1af8009e90 tx=0x7f1af800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:31:06.188 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:06.180+0000 7f1ae77fe700 1 -- 172.21.15.110:0/612182494 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1af800cc20 con 0x7f1b00156c50 2023-10-06T03:31:06.189 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:06.180+0000 7f1ae77fe700 1 -- 172.21.15.110:0/612182494 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1af800f040 con 0x7f1b00156c50 2023-10-06T03:31:06.189 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:06.180+0000 7f1b049ac700 1 -- 172.21.15.110:0/612182494 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b0014a920 con 0x7f1b00156c50 2023-10-06T03:31:06.189 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:06.180+0000 7f1ae77fe700 1 -- 172.21.15.110:0/612182494 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1af8014480 con 0x7f1b00156c50 2023-10-06T03:31:06.189 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:06.180+0000 7f1b049ac700 1 -- 172.21.15.110:0/612182494 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b0014aee0 con 0x7f1b00156c50 2023-10-06T03:31:06.189 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:06.184+0000 7f1b049ac700 1 -- 172.21.15.110:0/612182494 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b000c9c30 con 0x7f1b00156c50 2023-10-06T03:31:06.189 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:06.184+0000 7f1ae77fe700 1 -- 172.21.15.110:0/612182494 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 13) v1 ==== 80061+0+0 (secure 0 0 0) 0x7f1af80147f0 con 0x7f1b00156c50 2023-10-06T03:31:06.190 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:06.184+0000 7f1ae77fe700 1 --2- 172.21.15.110:0/612182494 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f1aec0606a0 0x7f1aec062b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:31:06.190 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:06.184+0000 7f1ae77fe700 1 -- 172.21.15.110:0/612182494 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 6171+0+0 (secure 0 0 0) 0x7f1af807d6d0 con 0x7f1b00156c50 2023-10-06T03:31:06.190 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:06.184+0000 7f1aff7fe700 1 --2- 172.21.15.110:0/612182494 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f1aec0606a0 0x7f1aec062b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:31:06.190 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:06.184+0000 7f1aff7fe700 1 --2- 172.21.15.110:0/612182494 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f1aec0606a0 0x7f1aec062b50 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f1af4000c00 tx=0x7f1af4009040 comp rx=0 tx=0).ready entity=mgr.5505 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:31:06.193 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:06.188+0000 7f1ae77fe700 1 -- 172.21.15.110:0/612182494 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f1af8051360 con 0x7f1b00156c50 2023-10-06T03:31:06.353 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:06.348+0000 7f1b049ac700 1 -- 172.21.15.110:0/612182494 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"} v 0) v1 -- 0x7f1b00071020 con 0x7f1b00156c50 2023-10-06T03:31:06.361 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:06.356+0000 7f1ae77fe700 1 -- 172.21.15.110:0/612182494 <== mon.0 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"}]=0 v15) v1 ==== 115+0+0 (secure 0 0 0) 0x7f1af8050ef0 con 0x7f1b00156c50 2023-10-06T03:31:06.365 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:06.360+0000 7f1b049ac700 1 -- 172.21.15.110:0/612182494 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f1aec0606a0 msgr2=0x7f1aec062b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:31:06.365 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:06.360+0000 7f1b049ac700 1 --2- 172.21.15.110:0/612182494 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f1aec0606a0 0x7f1aec062b50 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f1af4000c00 tx=0x7f1af4009040 comp rx=0 tx=0).stop 2023-10-06T03:31:06.365 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:06.360+0000 7f1b049ac700 1 -- 172.21.15.110:0/612182494 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1b00156c50 msgr2=0x7f1b00146530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:31:06.366 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:06.360+0000 7f1b049ac700 1 --2- 172.21.15.110:0/612182494 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1b00156c50 0x7f1b00146530 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f1af8009e90 tx=0x7f1af800b040 comp rx=0 tx=0).stop 2023-10-06T03:31:06.366 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:06.360+0000 7f1b049ac700 1 -- 172.21.15.110:0/612182494 shutdown_connections 2023-10-06T03:31:06.366 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:06.360+0000 7f1b049ac700 1 --2- 172.21.15.110:0/612182494 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f1aec0606a0 0x7f1aec062b50 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:06.366 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:06.360+0000 7f1b049ac700 1 --2- 172.21.15.110:0/612182494 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1b00156c50 0x7f1b00146530 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:06.366 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:06.360+0000 7f1b049ac700 1 --2- 172.21.15.110:0/612182494 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1b000cb7b0 0x7f1b00145ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:06.366 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:06.360+0000 7f1b049ac700 1 --2- 172.21.15.110:0/612182494 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1b000b11c0 0x7f1b000c65a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:06.366 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:06.360+0000 7f1b049ac700 1 -- 172.21.15.110:0/612182494 >> 172.21.15.110:0/612182494 conn(0x7f1b0006c760 msgr2=0x7f1b0014cbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:31:06.366 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:06.360+0000 7f1b049ac700 1 -- 172.21.15.110:0/612182494 shutdown_connections 2023-10-06T03:31:06.366 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:06.360+0000 7f1b049ac700 1 -- 172.21.15.110:0/612182494 wait complete. 2023-10-06T03:31:06.379 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedulesSnapdir.test_snap_dir_name' 2023-10-06T03:31:06.486 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:06.480+0000 7f43a359e700 1 Processor -- start 2023-10-06T03:31:06.487 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:06.480+0000 7f43a359e700 1 -- start start 2023-10-06T03:31:06.488 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:06.484+0000 7f43a359e700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f43a41592d0 0x7f43a415cfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:31:06.488 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:06.484+0000 7f43a359e700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f43a415d520 0x7f43a415a820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:31:06.488 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:06.484+0000 7f43a359e700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43a4159750 con 0x7f43a414fca0 2023-10-06T03:31:06.488 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:06.484+0000 7f43a359e700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f43a415a2a0 con 0x7f43a41592d0 2023-10-06T03:31:06.488 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:06.484+0000 7f43a359e700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f43a4159a00 con 0x7f43a415d520 2023-10-06T03:31:06.488 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:06.484+0000 7f4399d9b700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f43a41592d0 0x7f43a415cfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:31:06.488 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:06.484+0000 7f43a259c700 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f43a414fca0 0x7f43a4067f00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:60332/0 (socket says 172.21.15.110:60332) 2023-10-06T03:31:06.488 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:06.484+0000 7f43a259c700 1 -- 172.21.15.110:0/3402542099 learned_addr learned my addr 172.21.15.110:0/3402542099 (peer_addr_for_me v1:172.21.15.110:0/0) 2023-10-06T03:31:06.488 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:06.484+0000 7f43a2d9d700 1 --2- 172.21.15.110:0/3402542099 >> v2:172.21.15.192:3300/0 conn(0x7f43a415d520 0x7f43a415a820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:31:06.490 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:06.484+0000 7f43a1d9b700 1 -- 172.21.15.110:0/3402542099 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 359173856 0 0) 0x7f43a4159750 con 0x7f43a414fca0 2023-10-06T03:31:06.490 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:06.484+0000 7f43a1d9b700 1 -- 172.21.15.110:0/3402542099 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4390003570 con 0x7f43a414fca0 2023-10-06T03:31:06.490 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:06.484+0000 7f4399d9b700 1 -- 172.21.15.110:0/3402542099 >> v2:172.21.15.192:3300/0 conn(0x7f43a415d520 msgr2=0x7f43a415a820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:31:06.491 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:06.484+0000 7f4399d9b700 1 --2- 172.21.15.110:0/3402542099 >> v2:172.21.15.192:3300/0 conn(0x7f43a415d520 0x7f43a415a820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:06.491 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:06.484+0000 7f4399d9b700 1 -- 172.21.15.110:0/3402542099 >> v1:172.21.15.110:6789/0 conn(0x7f43a414fca0 legacy=0x7f43a4067f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:31:06.491 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:06.484+0000 7f4399d9b700 1 -- 172.21.15.110:0/3402542099 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43a4159ce0 con 0x7f43a41592d0 2023-10-06T03:31:06.491 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:06.484+0000 7f4399d9b700 1 --2- 172.21.15.110:0/3402542099 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f43a41592d0 0x7f43a415cfe0 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f43a40abee0 tx=0x7f438c0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b4cae57e1633428 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:31:06.491 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:06.484+0000 7f43a1d9b700 1 -- 172.21.15.110:0/3402542099 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f438c011630 con 0x7f43a41592d0 2023-10-06T03:31:06.491 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:06.484+0000 7f43a1d9b700 1 -- 172.21.15.110:0/3402542099 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f438c011790 con 0x7f43a41592d0 2023-10-06T03:31:06.491 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:06.484+0000 7f43a1d9b700 1 -- 172.21.15.110:0/3402542099 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f438c011b30 con 0x7f43a41592d0 2023-10-06T03:31:06.491 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:06.484+0000 7f43a359e700 1 -- 172.21.15.110:0/3402542099 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f43a41592d0 msgr2=0x7f43a415cfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:31:06.491 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:06.484+0000 7f43a359e700 1 --2- 172.21.15.110:0/3402542099 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f43a41592d0 0x7f43a415cfe0 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f43a40abee0 tx=0x7f438c0091a0 comp rx=0 tx=0).stop 2023-10-06T03:31:06.491 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:06.484+0000 7f43a359e700 1 -- 172.21.15.110:0/3402542099 shutdown_connections 2023-10-06T03:31:06.491 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:06.484+0000 7f43a359e700 1 --2- 172.21.15.110:0/3402542099 >> v2:172.21.15.192:3300/0 conn(0x7f43a415d520 0x7f43a415a820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:06.492 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:06.484+0000 7f43a359e700 1 --2- 172.21.15.110:0/3402542099 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f43a41592d0 0x7f43a415cfe0 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:06.492 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:06.484+0000 7f43a359e700 1 -- 172.21.15.110:0/3402542099 >> 172.21.15.110:0/3402542099 conn(0x7f43a406b4c0 msgr2=0x7f43a406b8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:31:06.492 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:06.484+0000 7f43a359e700 1 -- 172.21.15.110:0/3402542099 shutdown_connections 2023-10-06T03:31:06.492 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:06.484+0000 7f43a359e700 1 -- 172.21.15.110:0/3402542099 wait complete. 2023-10-06T03:31:06.492 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:06.488+0000 7f43a359e700 1 Processor -- start 2023-10-06T03:31:06.492 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:06.488+0000 7f43a359e700 1 -- start start 2023-10-06T03:31:06.494 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:06.488+0000 7f43a359e700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f43a414fca0 0x7f43a40ab4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:31:06.494 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:06.488+0000 7f43a359e700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f43a41592d0 0x7f43a40c8cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:31:06.494 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:06.488+0000 7f43a359e700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f43a415d520 0x7f43a416c670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:31:06.494 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:06.488+0000 7f43a359e700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43a40c9210 con 0x7f43a415d520 2023-10-06T03:31:06.495 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:06.488+0000 7f43a359e700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43a40c9490 con 0x7f43a41592d0 2023-10-06T03:31:06.495 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:06.488+0000 7f43a359e700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43a40c9770 con 0x7f43a414fca0 2023-10-06T03:31:06.495 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:06.488+0000 7f43a2d9d700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f43a415d520 0x7f43a416c670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:31:06.495 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:06.488+0000 7f43a2d9d700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f43a415d520 0x7f43a416c670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:38860/0 (socket says 172.21.15.110:38860) 2023-10-06T03:31:06.495 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:06.488+0000 7f43a2d9d700 1 -- 172.21.15.110:0/1948571891 learned_addr learned my addr 172.21.15.110:0/1948571891 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:31:06.495 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:06.488+0000 7f4399d9b700 1 --2- 172.21.15.110:0/1948571891 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f43a41592d0 0x7f43a40c8cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:31:06.495 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:06.488+0000 7f43a2d9d700 1 -- 172.21.15.110:0/1948571891 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f43a414fca0 msgr2=0x7f43a40ab4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:31:06.495 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:06.488+0000 7f43a2d9d700 1 --2- 172.21.15.110:0/1948571891 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f43a414fca0 0x7f43a40ab4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:06.495 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:06.488+0000 7f43a2d9d700 1 -- 172.21.15.110:0/1948571891 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f43a41592d0 msgr2=0x7f43a40c8cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:31:06.495 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:06.488+0000 7f43a2d9d700 1 --2- 172.21.15.110:0/1948571891 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f43a41592d0 0x7f43a40c8cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:06.496 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:06.488+0000 7f43a2d9d700 1 -- 172.21.15.110:0/1948571891 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f438c009420 con 0x7f43a415d520 2023-10-06T03:31:06.496 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:06.488+0000 7f43a2d9d700 1 --2- 172.21.15.110:0/1948571891 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f43a415d520 0x7f43a416c670 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f439c009df0 tx=0x7f439c00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:31:06.497 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:06.488+0000 7f439affd700 1 -- 172.21.15.110:0/1948571891 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f439c00cb50 con 0x7f43a415d520 2023-10-06T03:31:06.497 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:06.488+0000 7f439affd700 1 -- 172.21.15.110:0/1948571891 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f439c00f040 con 0x7f43a415d520 2023-10-06T03:31:06.497 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:06.488+0000 7f43a359e700 1 -- 172.21.15.110:0/1948571891 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43a40c9a50 con 0x7f43a415d520 2023-10-06T03:31:06.497 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:06.488+0000 7f439affd700 1 -- 172.21.15.110:0/1948571891 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f439c014480 con 0x7f43a415d520 2023-10-06T03:31:06.497 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:06.488+0000 7f43a359e700 1 -- 172.21.15.110:0/1948571891 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43a414e7c0 con 0x7f43a415d520 2023-10-06T03:31:06.497 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:06.492+0000 7f439affd700 1 -- 172.21.15.110:0/1948571891 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 13) v1 ==== 80061+0+0 (secure 0 0 0) 0x7f439c014830 con 0x7f43a415d520 2023-10-06T03:31:06.497 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:06.492+0000 7f43a359e700 1 -- 172.21.15.110:0/1948571891 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f43a40c9c90 con 0x7f43a415d520 2023-10-06T03:31:06.498 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:06.492+0000 7f439affd700 1 --2- 172.21.15.110:0/1948571891 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f4384060670 0x7f4384062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:31:06.498 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:06.492+0000 7f43a259c700 1 --2- 172.21.15.110:0/1948571891 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f4384060670 0x7f4384062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:31:06.498 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:06.492+0000 7f439affd700 1 -- 172.21.15.110:0/1948571891 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 6171+0+0 (secure 0 0 0) 0x7f439c07d5f0 con 0x7f43a415d520 2023-10-06T03:31:06.498 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:06.492+0000 7f43a259c700 1 --2- 172.21.15.110:0/1948571891 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f4384060670 0x7f4384062b20 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f43940010e0 tx=0x7f43940079d0 comp rx=0 tx=0).ready entity=mgr.5505 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:31:06.502 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:06.496+0000 7f439affd700 1 -- 172.21.15.110:0/1948571891 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f43a40c9c90 con 0x7f43a415d520 2023-10-06T03:31:06.721 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:06.716+0000 7f43a359e700 1 -- 172.21.15.110:0/1948571891 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedulesSnapdir.test_snap_dir_name"]} v 0) v1 -- 0x7f43a40afa60 con 0x7f43a415d520 2023-10-06T03:31:07.408 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:07.404+0000 7f439affd700 1 -- 172.21.15.110:0/1948571891 <== mon.0 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedulesSnapdir.test_snap_dir_name"]}]=0 v105) v1 ==== 155+0+0 (secure 0 0 0) 0x7f43a40afa60 con 0x7f43a415d520 2023-10-06T03:31:07.413 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:07.408+0000 7f43a359e700 1 -- 172.21.15.110:0/1948571891 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f4384060670 msgr2=0x7f4384062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:31:07.413 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:07.408+0000 7f43a359e700 1 --2- 172.21.15.110:0/1948571891 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f4384060670 0x7f4384062b20 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f43940010e0 tx=0x7f43940079d0 comp rx=0 tx=0).stop 2023-10-06T03:31:07.413 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:07.408+0000 7f43a359e700 1 -- 172.21.15.110:0/1948571891 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f43a415d520 msgr2=0x7f43a416c670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:31:07.413 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:07.408+0000 7f43a359e700 1 --2- 172.21.15.110:0/1948571891 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f43a415d520 0x7f43a416c670 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f439c009df0 tx=0x7f439c00b040 comp rx=0 tx=0).stop 2023-10-06T03:31:07.413 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:07.408+0000 7f43a359e700 1 -- 172.21.15.110:0/1948571891 shutdown_connections 2023-10-06T03:31:07.413 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:07.408+0000 7f43a359e700 1 --2- 172.21.15.110:0/1948571891 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f4384060670 0x7f4384062b20 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:07.413 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:07.408+0000 7f43a359e700 1 --2- 172.21.15.110:0/1948571891 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f43a415d520 0x7f43a416c670 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:07.414 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:07.408+0000 7f43a359e700 1 --2- 172.21.15.110:0/1948571891 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f43a41592d0 0x7f43a40c8cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:07.414 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:07.408+0000 7f43a359e700 1 --2- 172.21.15.110:0/1948571891 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f43a414fca0 0x7f43a40ab4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:07.414 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:07.408+0000 7f43a359e700 1 -- 172.21.15.110:0/1948571891 >> 172.21.15.110:0/1948571891 conn(0x7f43a406b4c0 msgr2=0x7f43a40c2120 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:31:07.414 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:07.408+0000 7f43a359e700 1 -- 172.21.15.110:0/1948571891 shutdown_connections 2023-10-06T03:31:07.414 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:07.408+0000 7f43a359e700 1 -- 172.21.15.110:0/1948571891 wait complete. 2023-10-06T03:31:07.429 INFO:tasks.cephfs_test_runner:Test the correctness of snap directory name ... ok 2023-10-06T03:31:07.429 INFO:tasks.cephfs_test_runner: 2023-10-06T03:31:07.429 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2023-10-06T03:31:07.429 INFO:tasks.cephfs_test_runner:Ran 1 test in 51.165s 2023-10-06T03:31:07.429 INFO:tasks.cephfs_test_runner: 2023-10-06T03:31:07.430 INFO:tasks.cephfs_test_runner:OK 2023-10-06T03:31:07.430 DEBUG:teuthology.run_tasks:Unwinding manager cephfs_test_runner 2023-10-06T03:31:07.439 DEBUG:teuthology.run_tasks:Unwinding manager ceph-fuse 2023-10-06T03:31:07.447 INFO:tasks.ceph_fuse:Unmounting ceph-fuse clients... 2023-10-06T03:31:07.447 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-06T03:31:07.447 DEBUG:teuthology.orchestra.run.smithi110:> ip netns list 2023-10-06T03:31:07.455 INFO:teuthology.orchestra.run.smithi110.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2023-10-06T03:31:07.455 INFO:teuthology.orchestra.run.smithi110.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2023-10-06T03:31:07.456 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-06T03:31:07.456 DEBUG:teuthology.orchestra.run.smithi110:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.2 2023-10-06T03:31:07.473 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-06T03:31:07.473 DEBUG:teuthology.orchestra.run.smithi110:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.0 2023-10-06T03:31:07.488 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-06T03:31:07.489 DEBUG:teuthology.orchestra.run.smithi110:> sudo ip link delete ceph-brx 2023-10-06T03:31:07.657 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-06T03:31:07.658 DEBUG:teuthology.orchestra.run.smithi192:> ip netns list 2023-10-06T03:31:07.665 INFO:teuthology.orchestra.run.smithi192.stdout:ceph-ns--home-ubuntu-cephtest-mnt.3 (id: 1) 2023-10-06T03:31:07.666 INFO:teuthology.orchestra.run.smithi192.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2023-10-06T03:31:07.666 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-06T03:31:07.666 DEBUG:teuthology.orchestra.run.smithi192:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.3 2023-10-06T03:31:07.680 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-06T03:31:07.680 DEBUG:teuthology.orchestra.run.smithi192:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.1 2023-10-06T03:31:07.691 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-06T03:31:07.691 DEBUG:teuthology.orchestra.run.smithi192:> sudo ip link delete ceph-brx 2023-10-06T03:31:07.848 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2023-10-06T03:31:07.857 INFO:tasks.ceph.ceph_manager.ceph:Canceling any pending splits or merges... 2023-10-06T03:31:07.857 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-06T03:31:07.951 DEBUG:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-06T03:31:07.953 DEBUG:teuthology.orchestra.run.smithi192:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-06T03:31:07.974 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:07.968+0000 7fc987fff700 1 Processor -- start 2023-10-06T03:31:07.974 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:07.968+0000 7fc987fff700 1 -- start start 2023-10-06T03:31:07.975 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:07.968+0000 7fc987fff700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc98815cf70 0x7fc98806bbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:31:07.975 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:07.968+0000 7fc987fff700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fc98815d340 0x7fc9880cc030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:31:07.975 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:07.968+0000 7fc987fff700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc98806b750 con 0x7fc9880c9e00 2023-10-06T03:31:07.975 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:07.968+0000 7fc987fff700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc98806ab40 con 0x7fc98815d340 2023-10-06T03:31:07.975 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:07.968+0000 7fc987fff700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc98806ae20 con 0x7fc98815cf70 2023-10-06T03:31:07.975 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:07.968+0000 7fc9877fe700 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7fc9880c9e00 0x7fc9880c6420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:60334/0 (socket says 172.21.15.110:60334) 2023-10-06T03:31:07.975 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:07.968+0000 7fc9877fe700 1 -- 172.21.15.110:0/3388907138 learned_addr learned my addr 172.21.15.110:0/3388907138 (peer_addr_for_me v1:172.21.15.110:0/0) 2023-10-06T03:31:07.977 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:07.972+0000 7fc9867fc700 1 --2- 172.21.15.110:0/3388907138 >> v2:172.21.15.192:3300/0 conn(0x7fc98815d340 0x7fc9880cc030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:31:07.977 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:07.972+0000 7fc986ffd700 1 --2- 172.21.15.110:0/3388907138 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc98815cf70 0x7fc98806bbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:31:07.977 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:07.972+0000 7fc985ffb700 1 -- 172.21.15.110:0/3388907138 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3263818127 0 0) 0x7fc98806b750 con 0x7fc9880c9e00 2023-10-06T03:31:07.977 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:07.972+0000 7fc985ffb700 1 -- 172.21.15.110:0/3388907138 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc970003570 con 0x7fc9880c9e00 2023-10-06T03:31:07.977 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:07.972+0000 7fc9867fc700 1 -- 172.21.15.110:0/3388907138 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc98815cf70 msgr2=0x7fc98806bbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:31:07.977 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:07.972+0000 7fc9867fc700 1 --2- 172.21.15.110:0/3388907138 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc98815cf70 0x7fc98806bbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:07.977 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:07.972+0000 7fc9867fc700 1 -- 172.21.15.110:0/3388907138 >> v1:172.21.15.110:6789/0 conn(0x7fc9880c9e00 legacy=0x7fc9880c6420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:31:07.977 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:07.972+0000 7fc9867fc700 1 -- 172.21.15.110:0/3388907138 --> v2:172.21.15.192:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc98806b100 con 0x7fc98815d340 2023-10-06T03:31:07.978 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:07.972+0000 7fc9867fc700 1 --2- 172.21.15.110:0/3388907138 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc98815d340 0x7fc9880cc030 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fc97c0098e0 tx=0x7fc97c0091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f99dc9166586762f server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:31:07.978 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:07.972+0000 7fc985ffb700 1 -- 172.21.15.110:0/3388907138 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc97c011630 con 0x7fc98815d340 2023-10-06T03:31:07.978 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:07.972+0000 7fc985ffb700 1 -- 172.21.15.110:0/3388907138 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc97c011790 con 0x7fc98815d340 2023-10-06T03:31:07.978 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:07.972+0000 7fc985ffb700 1 -- 172.21.15.110:0/3388907138 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc97c011b10 con 0x7fc98815d340 2023-10-06T03:31:07.979 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:07.972+0000 7fc987fff700 1 -- 172.21.15.110:0/3388907138 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc98815d340 msgr2=0x7fc9880cc030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:31:07.979 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:07.972+0000 7fc987fff700 1 --2- 172.21.15.110:0/3388907138 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc98815d340 0x7fc9880cc030 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fc97c0098e0 tx=0x7fc97c0091a0 comp rx=0 tx=0).stop 2023-10-06T03:31:07.979 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:07.972+0000 7fc987fff700 1 -- 172.21.15.110:0/3388907138 shutdown_connections 2023-10-06T03:31:07.979 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:07.972+0000 7fc987fff700 1 --2- 172.21.15.110:0/3388907138 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc98815d340 0x7fc9880cc030 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:07.979 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:07.972+0000 7fc987fff700 1 --2- 172.21.15.110:0/3388907138 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc98815cf70 0x7fc98806bbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:07.979 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:07.972+0000 7fc987fff700 1 -- 172.21.15.110:0/3388907138 >> 172.21.15.110:0/3388907138 conn(0x7fc98806c760 msgr2=0x7fc98806cb60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:31:07.979 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:07.972+0000 7fc987fff700 1 -- 172.21.15.110:0/3388907138 shutdown_connections 2023-10-06T03:31:07.979 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:07.972+0000 7fc987fff700 1 -- 172.21.15.110:0/3388907138 wait complete. 2023-10-06T03:31:07.980 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:07.972+0000 7fc987fff700 1 Processor -- start 2023-10-06T03:31:07.980 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:07.976+0000 7fc987fff700 1 -- start start 2023-10-06T03:31:07.982 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:07.976+0000 7fc987fff700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc9880c9e00 0x7fc988156240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:31:07.982 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:07.976+0000 7fc987fff700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc98815cf70 0x7fc9880c40d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:31:07.982 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:07.976+0000 7fc987fff700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc98815d340 0x7fc98816cea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:31:07.982 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:07.976+0000 7fc987fff700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc988156780 con 0x7fc98815d340 2023-10-06T03:31:07.983 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:07.976+0000 7fc987fff700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc98816d9b0 con 0x7fc9880c9e00 2023-10-06T03:31:07.983 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:07.976+0000 7fc987fff700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc98816dc90 con 0x7fc98815cf70 2023-10-06T03:31:07.983 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:07.976+0000 7fc9877fe700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc98815d340 0x7fc98816cea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:31:07.983 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:07.976+0000 7fc9877fe700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc98815d340 0x7fc98816cea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:38864/0 (socket says 172.21.15.110:38864) 2023-10-06T03:31:07.983 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:07.976+0000 7fc9877fe700 1 -- 172.21.15.110:0/1258166409 learned_addr learned my addr 172.21.15.110:0/1258166409 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:31:07.983 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:07.976+0000 7fc986ffd700 1 --2- 172.21.15.110:0/1258166409 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc9880c9e00 0x7fc988156240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:31:07.983 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:07.976+0000 7fc9867fc700 1 --2- 172.21.15.110:0/1258166409 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc98815cf70 0x7fc9880c40d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:31:07.983 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:07.976+0000 7fc9877fe700 1 -- 172.21.15.110:0/1258166409 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc98815cf70 msgr2=0x7fc9880c40d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:31:07.983 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:07.976+0000 7fc9877fe700 1 --2- 172.21.15.110:0/1258166409 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc98815cf70 0x7fc9880c40d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:07.983 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:07.976+0000 7fc9877fe700 1 -- 172.21.15.110:0/1258166409 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc9880c9e00 msgr2=0x7fc988156240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:31:07.983 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:07.976+0000 7fc9877fe700 1 --2- 172.21.15.110:0/1258166409 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc9880c9e00 0x7fc988156240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:07.984 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:07.976+0000 7fc9877fe700 1 -- 172.21.15.110:0/1258166409 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc97c009420 con 0x7fc98815d340 2023-10-06T03:31:07.984 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:07.976+0000 7fc9877fe700 1 --2- 172.21.15.110:0/1258166409 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc98815d340 0x7fc98816cea0 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7fc980001460 tx=0x7fc980001ae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:31:07.984 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:07.976+0000 7fc96effd700 1 -- 172.21.15.110:0/1258166409 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc98000a890 con 0x7fc98815d340 2023-10-06T03:31:07.984 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:07.976+0000 7fc96effd700 1 -- 172.21.15.110:0/1258166409 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc98000a9f0 con 0x7fc98815d340 2023-10-06T03:31:07.984 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:07.976+0000 7fc987fff700 1 -- 172.21.15.110:0/1258166409 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9880c38d0 con 0x7fc98815d340 2023-10-06T03:31:07.984 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:07.976+0000 7fc96effd700 1 -- 172.21.15.110:0/1258166409 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc98000ad70 con 0x7fc98815d340 2023-10-06T03:31:07.984 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:07.976+0000 7fc987fff700 1 -- 172.21.15.110:0/1258166409 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9880c54b0 con 0x7fc98815d340 2023-10-06T03:31:07.985 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:07.976+0000 7fc987fff700 1 -- 172.21.15.110:0/1258166409 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc9880c3dd0 con 0x7fc98815d340 2023-10-06T03:31:07.985 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:07.980+0000 7fc96effd700 1 -- 172.21.15.110:0/1258166409 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 13) v1 ==== 80061+0+0 (secure 0 0 0) 0x7fc980007300 con 0x7fc98815d340 2023-10-06T03:31:07.985 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:07.980+0000 7fc96effd700 1 --2- 172.21.15.110:0/1258166409 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fc974060670 0x7fc974062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:31:07.985 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:07.980+0000 7fc96effd700 1 -- 172.21.15.110:0/1258166409 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 6171+0+0 (secure 0 0 0) 0x7fc98007cee0 con 0x7fc98815d340 2023-10-06T03:31:07.985 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:07.980+0000 7fc986ffd700 1 --2- 172.21.15.110:0/1258166409 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fc974060670 0x7fc974062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:31:07.985 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:07.980+0000 7fc986ffd700 1 --2- 172.21.15.110:0/1258166409 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fc974060670 0x7fc974062b20 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fc978009db0 tx=0x7fc9780091a0 comp rx=0 tx=0).ready entity=mgr.5505 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:31:07.991 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:07.988+0000 7fc96effd700 1 -- 172.21.15.110:0/1258166409 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fc98004d1e0 con 0x7fc98815d340 2023-10-06T03:31:08.157 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:08.152+0000 7fc987fff700 1 -- 172.21.15.110:0/1258166409 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc988064770 con 0x7fc98815d340 2023-10-06T03:31:08.159 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:08.152+0000 7fc96effd700 1 -- 172.21.15.110:0/1258166409 <== mon.0 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v47) v1 ==== 74+0+14848 (secure 0 0 0) 0x7fc980050800 con 0x7fc98815d340 2023-10-06T03:31:08.159 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-10-06T03:31:08.160 INFO:teuthology.orchestra.run.smithi110.stdout:{"epoch":47,"fsid":"93b8f6ac-4c92-4b45-b8e4-b59617024aa2","created":"2023-10-06T03:29:10.519583+0000","modified":"2023-10-06T03:31:05.388828+0000","last_up_change":"2023-10-06T03:29:21.960511+0000","last_in_change":"2023-10-06T03:29:12.560436+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-06T03:29:20.703743+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2023-10-06T03:29:28.307281+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"5866eb6c-2adc-46e4-be7a-e5bcda0f4065","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6824","nonce":33394},{"type":"v1","addr":"172.21.15.110:6825","nonce":33394}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6826","nonce":33394},{"type":"v1","addr":"172.21.15.110:6827","nonce":33394}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6830","nonce":33394},{"type":"v1","addr":"172.21.15.110:6831","nonce":33394}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6828","nonce":33394},{"type":"v1","addr":"172.21.15.110:6829","nonce":33394}]},"public_addr":"172.21.15.110:6825/33394","cluster_addr":"172.21.15.110:6827/33394","heartbeat_back_addr":"172.21.15.110:6831/33394","heartbeat_front_addr":"172.21.15.110:6829/33394","state":["exists","up"]},{"osd":1,"uuid":"a38de118-062b-4a05-9782-2ceddf6ff4df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6800","nonce":33396},{"type":"v1","addr":"172.21.15.110:6801","nonce":33396}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6802","nonce":33396},{"type":"v1","addr":"172.21.15.110:6803","nonce":33396}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6806","nonce":33396},{"type":"v1","addr":"172.21.15.110:6807","nonce":33396}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6804","nonce":33396},{"type":"v1","addr":"172.21.15.110:6805","nonce":33396}]},"public_addr":"172.21.15.110:6801/33396","cluster_addr":"172.21.15.110:6803/33396","heartbeat_back_addr":"172.21.15.110:6807/33396","heartbeat_front_addr":"172.21.15.110:6805/33396","state":["exists","up"]},{"osd":2,"uuid":"56e39be2-3407-4b69-877e-cf16617b06d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6816","nonce":33397},{"type":"v1","addr":"172.21.15.110:6817","nonce":33397}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6818","nonce":33397},{"type":"v1","addr":"172.21.15.110:6819","nonce":33397}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6822","nonce":33397},{"type":"v1","addr":"172.21.15.110:6823","nonce":33397}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6820","nonce":33397},{"type":"v1","addr":"172.21.15.110:6821","nonce":33397}]},"public_addr":"172.21.15.110:6817/33397","cluster_addr":"172.21.15.110:6819/33397","heartbeat_back_addr":"172.21.15.110:6823/33397","heartbeat_front_addr":"172.21.15.110:6821/33397","state":["exists","up"]},{"osd":3,"uuid":"01c82d13-c1d4-4bde-b55e-3e0089b40115","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6808","nonce":33395},{"type":"v1","addr":"172.21.15.110:6809","nonce":33395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6810","nonce":33395},{"type":"v1","addr":"172.21.15.110:6811","nonce":33395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6814","nonce":33395},{"type":"v1","addr":"172.21.15.110:6815","nonce":33395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6812","nonce":33395},{"type":"v1","addr":"172.21.15.110:6813","nonce":33395}]},"public_addr":"172.21.15.110:6809/33395","cluster_addr":"172.21.15.110:6811/33395","heartbeat_back_addr":"172.21.15.110:6815/33395","heartbeat_front_addr":"172.21.15.110:6813/33395","state":["exists","up"]},{"osd":4,"uuid":"86983430-5fa3-4d17-998f-03e2cd843237","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":33683},{"type":"v1","addr":"172.21.15.192:6817","nonce":33683}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":33683},{"type":"v1","addr":"172.21.15.192:6819","nonce":33683}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":33683},{"type":"v1","addr":"172.21.15.192:6823","nonce":33683}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6820","nonce":33683},{"type":"v1","addr":"172.21.15.192:6821","nonce":33683}]},"public_addr":"172.21.15.192:6817/33683","cluster_addr":"172.21.15.192:6819/33683","heartbeat_back_addr":"172.21.15.192:6823/33683","heartbeat_front_addr":"172.21.15.192:6821/33683","state":["exists","up"]},{"osd":5,"uuid":"6945257f-821d-4619-a63f-ebe857eba6f4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":33684},{"type":"v1","addr":"172.21.15.192:6801","nonce":33684}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":33684},{"type":"v1","addr":"172.21.15.192:6803","nonce":33684}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":33684},{"type":"v1","addr":"172.21.15.192:6807","nonce":33684}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6804","nonce":33684},{"type":"v1","addr":"172.21.15.192:6805","nonce":33684}]},"public_addr":"172.21.15.192:6801/33684","cluster_addr":"172.21.15.192:6803/33684","heartbeat_back_addr":"172.21.15.192:6807/33684","heartbeat_front_addr":"172.21.15.192:6805/33684","state":["exists","up"]},{"osd":6,"uuid":"7854d1ae-7060-4780-8f97-73cea5e7a12f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6824","nonce":33686},{"type":"v1","addr":"172.21.15.192:6825","nonce":33686}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":33686},{"type":"v1","addr":"172.21.15.192:6827","nonce":33686}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":33686},{"type":"v1","addr":"172.21.15.192:6831","nonce":33686}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6828","nonce":33686},{"type":"v1","addr":"172.21.15.192:6829","nonce":33686}]},"public_addr":"172.21.15.192:6825/33686","cluster_addr":"172.21.15.192:6827/33686","heartbeat_back_addr":"172.21.15.192:6831/33686","heartbeat_front_addr":"172.21.15.192:6829/33686","state":["exists","up"]},{"osd":7,"uuid":"f1ebdfd5-b859-482b-9f97-c18c4a26f445","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6808","nonce":33685},{"type":"v1","addr":"172.21.15.192:6809","nonce":33685}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":33685},{"type":"v1","addr":"172.21.15.192:6811","nonce":33685}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":33685},{"type":"v1","addr":"172.21.15.192:6815","nonce":33685}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6812","nonce":33685},{"type":"v1","addr":"172.21.15.192:6813","nonce":33685}]},"public_addr":"172.21.15.192:6809/33685","cluster_addr":"172.21.15.192:6811/33685","heartbeat_back_addr":"172.21.15.192:6815/33685","heartbeat_front_addr":"172.21.15.192:6813/33685","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T03:29:18.002048+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T03:29:17.478576+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T03:29:18.034944+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T03:29:17.628513+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T03:29:18.047338+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T03:29:17.772349+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T03:29:17.883958+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T03:29:18.151838+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.110:6836/1379510962":"2023-10-07T03:31:02.317023+0000","172.21.15.110:0/3461792981":"2023-10-07T03:31:01.923482+0000","172.21.15.110:0/1315925152":"2023-10-07T03:31:01.923482+0000","172.21.15.110:0/3495425172":"2023-10-07T03:31:01.923482+0000","172.21.15.110:0/3149105774":"2023-10-07T03:31:01.923482+0000","172.21.15.110:0/3193317457":"2023-10-07T03:31:01.923482+0000","172.21.15.110:0/2107091982":"2023-10-07T03:31:01.923482+0000","172.21.15.110:0/2944476429":"2023-10-07T03:31:01.923482+0000","172.21.15.110:0/1857995858":"2023-10-07T03:30:46.697197+0000","172.21.15.110:0/1318624357":"2023-10-07T03:30:46.697197+0000","172.21.15.110:0/1139019390":"2023-10-07T03:30:46.697197+0000","172.21.15.110:6837/1379510962":"2023-10-07T03:31:02.317023+0000","172.21.15.110:6833/33268":"2023-10-07T03:30:46.697197+0000","172.21.15.110:0/463196587":"2023-10-07T03:30:46.697197+0000","172.21.15.110:6832/33268":"2023-10-07T03:30:46.697197+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-06T03:31:08.161 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:08.156+0000 7fc987fff700 1 -- 172.21.15.110:0/1258166409 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fc974060670 msgr2=0x7fc974062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:31:08.162 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:08.156+0000 7fc987fff700 1 --2- 172.21.15.110:0/1258166409 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fc974060670 0x7fc974062b20 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fc978009db0 tx=0x7fc9780091a0 comp rx=0 tx=0).stop 2023-10-06T03:31:08.162 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:08.156+0000 7fc987fff700 1 -- 172.21.15.110:0/1258166409 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc98815d340 msgr2=0x7fc98816cea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:31:08.162 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:08.156+0000 7fc987fff700 1 --2- 172.21.15.110:0/1258166409 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc98815d340 0x7fc98816cea0 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7fc980001460 tx=0x7fc980001ae0 comp rx=0 tx=0).stop 2023-10-06T03:31:08.162 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:08.156+0000 7fc987fff700 1 -- 172.21.15.110:0/1258166409 shutdown_connections 2023-10-06T03:31:08.162 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:08.156+0000 7fc987fff700 1 --2- 172.21.15.110:0/1258166409 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fc974060670 0x7fc974062b20 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:08.162 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:08.156+0000 7fc987fff700 1 --2- 172.21.15.110:0/1258166409 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc98815d340 0x7fc98816cea0 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:08.162 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:08.156+0000 7fc987fff700 1 --2- 172.21.15.110:0/1258166409 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc98815cf70 0x7fc9880c40d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:08.162 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:08.156+0000 7fc987fff700 1 --2- 172.21.15.110:0/1258166409 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc9880c9e00 0x7fc988156240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:08.162 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:08.156+0000 7fc987fff700 1 -- 172.21.15.110:0/1258166409 >> 172.21.15.110:0/1258166409 conn(0x7fc98806c760 msgr2=0x7fc9880c87d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:31:08.163 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:08.156+0000 7fc987fff700 1 -- 172.21.15.110:0/1258166409 shutdown_connections 2023-10-06T03:31:08.163 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:08.156+0000 7fc987fff700 1 -- 172.21.15.110:0/1258166409 wait complete. 2023-10-06T03:31:08.173 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2023-10-06T03:31:08.174 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-06T03:31:08.278 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:08.272+0000 7fd4e5469700 1 Processor -- start 2023-10-06T03:31:08.278 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:08.272+0000 7fd4e5469700 1 -- start start 2023-10-06T03:31:08.279 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:08.272+0000 7fd4e5469700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fd4e00aaca0 0x7fd4e00cb8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:31:08.279 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:08.272+0000 7fd4e5469700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fd4e00cbdf0 0x7fd4e00c7430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:31:08.279 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:08.272+0000 7fd4e5469700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4e0156c30 con 0x7fd4e00c7970 2023-10-06T03:31:08.280 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:08.272+0000 7fd4e5469700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd4e00cc220 con 0x7fd4e00aaca0 2023-10-06T03:31:08.280 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:08.272+0000 7fd4e5469700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd4e0156300 con 0x7fd4e00cbdf0 2023-10-06T03:31:08.280 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:08.272+0000 7fd4e4c68700 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7fd4e00c7970 0x7fd4e0159680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:60350/0 (socket says 172.21.15.110:60350) 2023-10-06T03:31:08.280 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:08.272+0000 7fd4e4c68700 1 -- 172.21.15.110:0/1613000849 learned_addr learned my addr 172.21.15.110:0/1613000849 (peer_addr_for_me v1:172.21.15.110:0/0) 2023-10-06T03:31:08.280 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:08.272+0000 7fd4dffff700 1 --2- 172.21.15.110:0/1613000849 >> v2:172.21.15.110:3300/0 conn(0x7fd4e00aaca0 0x7fd4e00cb8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:31:08.280 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:08.272+0000 7fd4df7fe700 1 --2- 172.21.15.110:0/1613000849 >> v2:172.21.15.192:3300/0 conn(0x7fd4e00cbdf0 0x7fd4e00c7430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:31:08.280 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:08.276+0000 7fd4deffd700 1 -- 172.21.15.110:0/1613000849 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3485912889 0 0) 0x7fd4e0156c30 con 0x7fd4e00c7970 2023-10-06T03:31:08.280 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:08.276+0000 7fd4deffd700 1 -- 172.21.15.110:0/1613000849 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd4cc003570 con 0x7fd4e00c7970 2023-10-06T03:31:08.282 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:08.276+0000 7fd4dffff700 1 -- 172.21.15.110:0/1613000849 >> v2:172.21.15.192:3300/0 conn(0x7fd4e00cbdf0 msgr2=0x7fd4e00c7430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:31:08.282 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:08.276+0000 7fd4dffff700 1 --2- 172.21.15.110:0/1613000849 >> v2:172.21.15.192:3300/0 conn(0x7fd4e00cbdf0 0x7fd4e00c7430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:08.282 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:08.276+0000 7fd4dffff700 1 -- 172.21.15.110:0/1613000849 >> v1:172.21.15.110:6789/0 conn(0x7fd4e00c7970 legacy=0x7fd4e0159680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:31:08.282 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:08.276+0000 7fd4dffff700 1 -- 172.21.15.110:0/1613000849 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4e01565e0 con 0x7fd4e00aaca0 2023-10-06T03:31:08.282 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:08.276+0000 7fd4df7fe700 1 --2- 172.21.15.110:0/1613000849 >> v2:172.21.15.192:3300/0 conn(0x7fd4e00cbdf0 0x7fd4e00c7430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-06T03:31:08.282 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:08.276+0000 7fd4dffff700 1 --2- 172.21.15.110:0/1613000849 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd4e00aaca0 0x7fd4e00cb8b0 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7fd4e0069ae0 tx=0x7fd4d00091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e80d780178513519 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:31:08.282 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:08.276+0000 7fd4deffd700 1 -- 172.21.15.110:0/1613000849 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4d0011630 con 0x7fd4e00aaca0 2023-10-06T03:31:08.282 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:08.276+0000 7fd4deffd700 1 -- 172.21.15.110:0/1613000849 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4d00117b0 con 0x7fd4e00aaca0 2023-10-06T03:31:08.283 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:08.276+0000 7fd4deffd700 1 -- 172.21.15.110:0/1613000849 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4d0011b70 con 0x7fd4e00aaca0 2023-10-06T03:31:08.283 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:08.276+0000 7fd4e5469700 1 -- 172.21.15.110:0/1613000849 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd4e00aaca0 msgr2=0x7fd4e00cb8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:31:08.283 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:08.276+0000 7fd4e5469700 1 --2- 172.21.15.110:0/1613000849 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd4e00aaca0 0x7fd4e00cb8b0 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7fd4e0069ae0 tx=0x7fd4d00091a0 comp rx=0 tx=0).stop 2023-10-06T03:31:08.283 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:08.276+0000 7fd4e5469700 1 -- 172.21.15.110:0/1613000849 shutdown_connections 2023-10-06T03:31:08.283 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:08.276+0000 7fd4e5469700 1 --2- 172.21.15.110:0/1613000849 >> v2:172.21.15.192:3300/0 conn(0x7fd4e00cbdf0 0x7fd4e00c7430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:08.283 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:08.276+0000 7fd4e5469700 1 --2- 172.21.15.110:0/1613000849 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd4e00aaca0 0x7fd4e00cb8b0 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:08.283 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:08.276+0000 7fd4e5469700 1 -- 172.21.15.110:0/1613000849 >> 172.21.15.110:0/1613000849 conn(0x7fd4e006c760 msgr2=0x7fd4e006cb60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:31:08.283 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:08.276+0000 7fd4e5469700 1 -- 172.21.15.110:0/1613000849 shutdown_connections 2023-10-06T03:31:08.283 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:08.276+0000 7fd4e5469700 1 -- 172.21.15.110:0/1613000849 wait complete. 2023-10-06T03:31:08.284 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:08.276+0000 7fd4e5469700 1 Processor -- start 2023-10-06T03:31:08.284 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:08.280+0000 7fd4e5469700 1 -- start start 2023-10-06T03:31:08.287 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:08.280+0000 7fd4e5469700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd4e00aaca0 0x7fd4e0149730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:31:08.287 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:08.280+0000 7fd4e5469700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd4e00c7970 0x7fd4e00ac850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:31:08.287 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:08.280+0000 7fd4e5469700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd4e00cbdf0 0x7fd4e00acd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:31:08.287 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:08.280+0000 7fd4e5469700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4e00ab3a0 con 0x7fd4e00aaca0 2023-10-06T03:31:08.287 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:08.280+0000 7fd4e5469700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4e00aff70 con 0x7fd4e00c7970 2023-10-06T03:31:08.287 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:08.280+0000 7fd4e5469700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4e00ad6e0 con 0x7fd4e00cbdf0 2023-10-06T03:31:08.287 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:08.280+0000 7fd4dffff700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd4e00aaca0 0x7fd4e0149730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:31:08.287 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:08.280+0000 7fd4dffff700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd4e00aaca0 0x7fd4e0149730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:38886/0 (socket says 172.21.15.110:38886) 2023-10-06T03:31:08.288 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:08.280+0000 7fd4dffff700 1 -- 172.21.15.110:0/1879965826 learned_addr learned my addr 172.21.15.110:0/1879965826 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:31:08.288 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:08.280+0000 7fd4df7fe700 1 --2- 172.21.15.110:0/1879965826 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd4e00c7970 0x7fd4e00ac850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:31:08.288 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:08.280+0000 7fd4e4c68700 1 --2- 172.21.15.110:0/1879965826 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd4e00cbdf0 0x7fd4e00acd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:31:08.288 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:08.280+0000 7fd4dffff700 1 -- 172.21.15.110:0/1879965826 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd4e00cbdf0 msgr2=0x7fd4e00acd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:31:08.288 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:08.280+0000 7fd4dffff700 1 --2- 172.21.15.110:0/1879965826 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd4e00cbdf0 0x7fd4e00acd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:08.288 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:08.280+0000 7fd4dffff700 1 -- 172.21.15.110:0/1879965826 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd4e00c7970 msgr2=0x7fd4e00ac850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:31:08.288 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:08.280+0000 7fd4dffff700 1 --2- 172.21.15.110:0/1879965826 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd4e00c7970 0x7fd4e00ac850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:08.288 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:08.280+0000 7fd4dffff700 1 -- 172.21.15.110:0/1879965826 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4c8009a20 con 0x7fd4e00aaca0 2023-10-06T03:31:08.288 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:08.280+0000 7fd4df7fe700 1 --2- 172.21.15.110:0/1879965826 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd4e00c7970 0x7fd4e00ac850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-06T03:31:08.288 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:08.280+0000 7fd4e4c68700 1 --2- 172.21.15.110:0/1879965826 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd4e00cbdf0 0x7fd4e00acd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-06T03:31:08.288 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:08.280+0000 7fd4dffff700 1 --2- 172.21.15.110:0/1879965826 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd4e00aaca0 0x7fd4e0149730 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7fd4d0013000 tx=0x7fd4d0017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:31:08.289 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:08.280+0000 7fd4d7fff700 1 -- 172.21.15.110:0/1879965826 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4d001f070 con 0x7fd4e00aaca0 2023-10-06T03:31:08.289 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:08.280+0000 7fd4d7fff700 1 -- 172.21.15.110:0/1879965826 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4d0000bc0 con 0x7fd4e00aaca0 2023-10-06T03:31:08.289 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:08.280+0000 7fd4d7fff700 1 -- 172.21.15.110:0/1879965826 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4d0011b70 con 0x7fd4e00aaca0 2023-10-06T03:31:08.289 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:08.280+0000 7fd4e5469700 1 -- 172.21.15.110:0/1879965826 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4d0009420 con 0x7fd4e00aaca0 2023-10-06T03:31:08.289 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:08.280+0000 7fd4e5469700 1 -- 172.21.15.110:0/1879965826 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4e00adca0 con 0x7fd4e00aaca0 2023-10-06T03:31:08.290 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:08.284+0000 7fd4e5469700 1 -- 172.21.15.110:0/1879965826 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd4e00adec0 con 0x7fd4e00aaca0 2023-10-06T03:31:08.290 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:08.284+0000 7fd4d7fff700 1 -- 172.21.15.110:0/1879965826 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 13) v1 ==== 80061+0+0 (secure 0 0 0) 0x7fd4d0023070 con 0x7fd4e00aaca0 2023-10-06T03:31:08.290 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:08.284+0000 7fd4d7fff700 1 --2- 172.21.15.110:0/1879965826 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fd4c00606a0 0x7fd4c0062b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:31:08.291 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:08.284+0000 7fd4df7fe700 1 --2- 172.21.15.110:0/1879965826 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fd4c00606a0 0x7fd4c0062b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:31:08.291 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:08.284+0000 7fd4d7fff700 1 -- 172.21.15.110:0/1879965826 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 6171+0+0 (secure 0 0 0) 0x7fd4d007dd70 con 0x7fd4e00aaca0 2023-10-06T03:31:08.291 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:08.284+0000 7fd4df7fe700 1 --2- 172.21.15.110:0/1879965826 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fd4c00606a0 0x7fd4c0062b50 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fd4c80099f0 tx=0x7fd4c800b040 comp rx=0 tx=0).ready entity=mgr.5505 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:31:08.297 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:08.292+0000 7fd4d7fff700 1 -- 172.21.15.110:0/1879965826 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fd4d004e080 con 0x7fd4e00aaca0 2023-10-06T03:31:08.453 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:08.448+0000 7fd4e5469700 1 -- 172.21.15.110:0/1879965826 --> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fd4e0064770 con 0x7fd4c00606a0 2023-10-06T03:31:08.460 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:08.456+0000 7fd4d7fff700 1 -- 172.21.15.110:0/1879965826 <== mgr.5505 v2:172.21.15.110:6832/33268 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+71219 (secure 0 0 0) 0x7fd4e0064770 con 0x7fd4c00606a0 2023-10-06T03:31:08.461 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-10-06T03:31:08.462 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:08.456+0000 7fd4e5469700 1 -- 172.21.15.110:0/1879965826 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fd4c00606a0 msgr2=0x7fd4c0062b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:31:08.462 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:08.456+0000 7fd4e5469700 1 --2- 172.21.15.110:0/1879965826 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fd4c00606a0 0x7fd4c0062b50 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fd4c80099f0 tx=0x7fd4c800b040 comp rx=0 tx=0).stop 2023-10-06T03:31:08.462 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:08.456+0000 7fd4e5469700 1 -- 172.21.15.110:0/1879965826 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd4e00aaca0 msgr2=0x7fd4e0149730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:31:08.462 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:08.456+0000 7fd4e5469700 1 --2- 172.21.15.110:0/1879965826 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd4e00aaca0 0x7fd4e0149730 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7fd4d0013000 tx=0x7fd4d0017000 comp rx=0 tx=0).stop 2023-10-06T03:31:08.463 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:08.456+0000 7fd4e5469700 1 -- 172.21.15.110:0/1879965826 shutdown_connections 2023-10-06T03:31:08.463 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:08.456+0000 7fd4e5469700 1 --2- 172.21.15.110:0/1879965826 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fd4c00606a0 0x7fd4c0062b50 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:08.463 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:08.456+0000 7fd4e5469700 1 --2- 172.21.15.110:0/1879965826 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fd4e00cbdf0 0x7fd4e00acd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:08.463 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:08.456+0000 7fd4e5469700 1 --2- 172.21.15.110:0/1879965826 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd4e00c7970 0x7fd4e00ac850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:08.463 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:08.456+0000 7fd4e5469700 1 --2- 172.21.15.110:0/1879965826 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd4e00aaca0 0x7fd4e0149730 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:08.463 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:08.456+0000 7fd4e5469700 1 -- 172.21.15.110:0/1879965826 >> 172.21.15.110:0/1879965826 conn(0x7fd4e006c760 msgr2=0x7fd4e00c0770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:31:08.463 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:08.456+0000 7fd4e5469700 1 -- 172.21.15.110:0/1879965826 shutdown_connections 2023-10-06T03:31:08.463 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:08.456+0000 7fd4e5469700 1 -- 172.21.15.110:0/1879965826 wait complete. 2023-10-06T03:31:08.464 INFO:teuthology.orchestra.run.smithi110.stderr:dumped all 2023-10-06T03:31:08.476 INFO:teuthology.orchestra.run.smithi110.stdout:{"pg_ready":true,"pg_map":{"version":9,"stamp":"2023-10-06T03:31:07.946720+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":308,"num_read_kb":512,"num_write":199,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":139,"ondisk_log_size":139,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":210,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2401152,"kb_used_data":6464,"kb_used_omap":0,"kb_used_meta":2394624,"kb_avail":752573568,"statfs":{"total":773094113280,"available":770635333632,"internally_reserved":0,"allocated":6619136,"data_stored":4346957,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2452094976},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-667624,"num_objects":-25,"num_object_clones":0,"num_object_copies":-50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-25,"num_whiteouts":0,"num_read":-317,"num_read_kb":-281,"num_write":-285,"num_write_kb":-2055,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.991630"},"pg_stats":[{"pgid":"1.0","version":"44'139","reported_seq":261,"reported_epoch":44,"state":"active+clean","last_fresh":"2023-10-06T03:31:02.032960+0000","last_change":"2023-10-06T03:29:21.978311+0000","last_active":"2023-10-06T03:31:02.032960+0000","last_peered":"2023-10-06T03:31:02.032960+0000","last_clean":"2023-10-06T03:31:02.032960+0000","last_became_active":"2023-10-06T03:29:21.978017+0000","last_became_peered":"2023-10-06T03:29:21.978017+0000","last_unstale":"2023-10-06T03:31:02.032960+0000","last_undegraded":"2023-10-06T03:31:02.032960+0000","last_fullsized":"2023-10-06T03:31:02.032960+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:20.957955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:20.957955+0000","last_clean_scrub_stamp":"2023-10-06T03:29:20.957955+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T07:33:10.429084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":308,"num_read_kb":512,"num_write":199,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":31,"reported_epoch":44,"state":"active+clean","last_fresh":"2023-10-06T03:31:01.980717+0000","last_change":"2023-10-06T03:29:29.386120+0000","last_active":"2023-10-06T03:31:01.980717+0000","last_peered":"2023-10-06T03:31:01.980717+0000","last_clean":"2023-10-06T03:31:01.980717+0000","last_became_active":"2023-10-06T03:29:29.385692+0000","last_became_peered":"2023-10-06T03:29:29.385692+0000","last_unstale":"2023-10-06T03:31:01.980717+0000","last_undegraded":"2023-10-06T03:31:01.980717+0000","last_fullsized":"2023-10-06T03:31:01.980717+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:28.357506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:28.357506+0000","last_clean_scrub_stamp":"2023-10-06T03:29:28.357506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T10:21:35.864895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":30,"reported_epoch":43,"state":"active+clean","last_fresh":"2023-10-06T03:30:46.743121+0000","last_change":"2023-10-06T03:29:29.384353+0000","last_active":"2023-10-06T03:30:46.743121+0000","last_peered":"2023-10-06T03:30:46.743121+0000","last_clean":"2023-10-06T03:30:46.743121+0000","last_became_active":"2023-10-06T03:29:29.383884+0000","last_became_peered":"2023-10-06T03:29:29.383884+0000","last_unstale":"2023-10-06T03:30:46.743121+0000","last_undegraded":"2023-10-06T03:30:46.743121+0000","last_fullsized":"2023-10-06T03:30:46.743121+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:28.357506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:28.357506+0000","last_clean_scrub_stamp":"2023-10-06T03:29:28.357506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T14:43:09.107140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":32,"reported_epoch":44,"state":"active+clean","last_fresh":"2023-10-06T03:31:01.933050+0000","last_change":"2023-10-06T03:29:29.606195+0000","last_active":"2023-10-06T03:31:01.933050+0000","last_peered":"2023-10-06T03:31:01.933050+0000","last_clean":"2023-10-06T03:31:01.933050+0000","last_became_active":"2023-10-06T03:29:29.605489+0000","last_became_peered":"2023-10-06T03:29:29.605489+0000","last_unstale":"2023-10-06T03:31:01.933050+0000","last_undegraded":"2023-10-06T03:31:01.933050+0000","last_fullsized":"2023-10-06T03:31:01.933050+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:28.357506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:28.357506+0000","last_clean_scrub_stamp":"2023-10-06T03:29:28.357506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T10:21:35.864895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":31,"reported_epoch":44,"state":"active+clean","last_fresh":"2023-10-06T03:31:01.980497+0000","last_change":"2023-10-06T03:29:29.384415+0000","last_active":"2023-10-06T03:31:01.980497+0000","last_peered":"2023-10-06T03:31:01.980497+0000","last_clean":"2023-10-06T03:31:01.980497+0000","last_became_active":"2023-10-06T03:29:29.383894+0000","last_became_peered":"2023-10-06T03:29:29.383894+0000","last_unstale":"2023-10-06T03:31:01.980497+0000","last_undegraded":"2023-10-06T03:31:01.980497+0000","last_fullsized":"2023-10-06T03:31:01.980497+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:28.357506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:28.357506+0000","last_clean_scrub_stamp":"2023-10-06T03:29:28.357506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T13:09:00.657107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":30,"reported_epoch":43,"state":"active+clean","last_fresh":"2023-10-06T03:30:46.714052+0000","last_change":"2023-10-06T03:29:29.382183+0000","last_active":"2023-10-06T03:30:46.714052+0000","last_peered":"2023-10-06T03:30:46.714052+0000","last_clean":"2023-10-06T03:30:46.714052+0000","last_became_active":"2023-10-06T03:29:29.381918+0000","last_became_peered":"2023-10-06T03:29:29.381918+0000","last_unstale":"2023-10-06T03:30:46.714052+0000","last_undegraded":"2023-10-06T03:30:46.714052+0000","last_fullsized":"2023-10-06T03:30:46.714052+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:28.357506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:28.357506+0000","last_clean_scrub_stamp":"2023-10-06T03:29:28.357506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T11:56:26.013716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":30,"reported_epoch":43,"state":"active+clean","last_fresh":"2023-10-06T03:30:46.743697+0000","last_change":"2023-10-06T03:29:29.384279+0000","last_active":"2023-10-06T03:30:46.743697+0000","last_peered":"2023-10-06T03:30:46.743697+0000","last_clean":"2023-10-06T03:30:46.743697+0000","last_became_active":"2023-10-06T03:29:29.383533+0000","last_became_peered":"2023-10-06T03:29:29.383533+0000","last_unstale":"2023-10-06T03:30:46.743697+0000","last_undegraded":"2023-10-06T03:30:46.743697+0000","last_fullsized":"2023-10-06T03:30:46.743697+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:28.357506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:28.357506+0000","last_clean_scrub_stamp":"2023-10-06T03:29:28.357506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T08:34:19.641525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":30,"reported_epoch":43,"state":"active+clean","last_fresh":"2023-10-06T03:30:46.713940+0000","last_change":"2023-10-06T03:29:29.386238+0000","last_active":"2023-10-06T03:30:46.713940+0000","last_peered":"2023-10-06T03:30:46.713940+0000","last_clean":"2023-10-06T03:30:46.713940+0000","last_became_active":"2023-10-06T03:29:29.385865+0000","last_became_peered":"2023-10-06T03:29:29.385865+0000","last_unstale":"2023-10-06T03:30:46.713940+0000","last_undegraded":"2023-10-06T03:30:46.713940+0000","last_fullsized":"2023-10-06T03:30:46.713940+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:28.357506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:28.357506+0000","last_clean_scrub_stamp":"2023-10-06T03:29:28.357506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T12:43:34.714730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":30,"reported_epoch":44,"state":"active+clean","last_fresh":"2023-10-06T03:31:02.100499+0000","last_change":"2023-10-06T03:29:29.387402+0000","last_active":"2023-10-06T03:31:02.100499+0000","last_peered":"2023-10-06T03:31:02.100499+0000","last_clean":"2023-10-06T03:31:02.100499+0000","last_became_active":"2023-10-06T03:29:29.386812+0000","last_became_peered":"2023-10-06T03:29:29.386812+0000","last_unstale":"2023-10-06T03:31:02.100499+0000","last_undegraded":"2023-10-06T03:31:02.100499+0000","last_fullsized":"2023-10-06T03:31:02.100499+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:28.357506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:28.357506+0000","last_clean_scrub_stamp":"2023-10-06T03:29:28.357506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T12:09:43.895810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":308,"num_read_kb":512,"num_write":199,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":139,"ondisk_log_size":139,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509464,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298360,"kb_used_data":1072,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94073480,"statfs":{"total":96636764160,"available":96331243520,"internally_reserved":0,"allocated":1097728,"data_stored":817631,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Oct 6 03:30:26 2023","interfaces":[{"interface":"back","average":{"1min":0.878,"5min":0.878,"15min":0.878},"min":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"max":{"1min":1.228,"5min":1.228,"15min":1.228},"last":0.69899999999999995},{"interface":"front","average":{"1min":0.878,"5min":0.878,"15min":0.878},"min":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":1.181,"5min":1.181,"15min":1.181},"last":0.95399999999999996}]},{"osd":1,"last update":"Fri Oct 6 03:30:26 2023","interfaces":[{"interface":"back","average":{"1min":0.93000000000000005,"5min":0.93000000000000005,"15min":0.93000000000000005},"min":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"max":{"1min":1.2889999999999999,"5min":1.2889999999999999,"15min":1.2889999999999999},"last":0.58799999999999997},{"interface":"front","average":{"1min":0.94299999999999995,"5min":0.94299999999999995,"15min":0.94299999999999995},"min":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"max":{"1min":1.341,"5min":1.341,"15min":1.341},"last":0.61799999999999999}]},{"osd":2,"last update":"Fri Oct 6 03:30:26 2023","interfaces":[{"interface":"back","average":{"1min":0.95299999999999996,"5min":0.95299999999999996,"15min":0.95299999999999996},"min":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.72599999999999998},"max":{"1min":1.2050000000000001,"5min":1.2050000000000001,"15min":1.2050000000000001},"last":0.76800000000000002},{"interface":"front","average":{"1min":1.006,"5min":1.006,"15min":1.006},"min":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"max":{"1min":1.409,"5min":1.409,"15min":1.409},"last":0.73199999999999998}]},{"osd":3,"last update":"Fri Oct 6 03:30:26 2023","interfaces":[{"interface":"back","average":{"1min":0.86299999999999999,"5min":0.86299999999999999,"15min":0.86299999999999999},"min":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"max":{"1min":1.1339999999999999,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.85999999999999999},{"interface":"front","average":{"1min":0.96899999999999997,"5min":0.96899999999999997,"15min":0.96899999999999997},"min":{"1min":0.63400000000000001,"5min":0.63400000000000001,"15min":0.63400000000000001},"max":{"1min":1.256,"5min":1.256,"15min":1.256},"last":0.80800000000000005}]},{"osd":4,"last update":"Fri Oct 6 03:30:26 2023","interfaces":[{"interface":"back","average":{"1min":0.91900000000000004,"5min":0.91900000000000004,"15min":0.91900000000000004},"min":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"max":{"1min":1.2230000000000001,"5min":1.2230000000000001,"15min":1.2230000000000001},"last":0.83699999999999997},{"interface":"front","average":{"1min":1.0529999999999999,"5min":1.0529999999999999,"15min":1.0529999999999999},"min":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"max":{"1min":1.4750000000000001,"5min":1.4750000000000001,"15min":1.4750000000000001},"last":1.0529999999999999}]},{"osd":5,"last update":"Fri Oct 6 03:30:26 2023","interfaces":[{"interface":"back","average":{"1min":1.0229999999999999,"5min":1.0229999999999999,"15min":1.0229999999999999},"min":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":0.77900000000000003},"max":{"1min":1.431,"5min":1.431,"15min":1.431},"last":1.0309999999999999},{"interface":"front","average":{"1min":1.105,"5min":1.105,"15min":1.105},"min":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"max":{"1min":1.407,"5min":1.407,"15min":1.407},"last":0.91200000000000003}]},{"osd":6,"last update":"Fri Oct 6 03:30:26 2023","interfaces":[{"interface":"back","average":{"1min":1.1399999999999999,"5min":1.1399999999999999,"15min":1.1399999999999999},"min":{"1min":0.86799999999999999,"5min":0.86799999999999999,"15min":0.86799999999999999},"max":{"1min":1.526,"5min":1.526,"15min":1.526},"last":1.1259999999999999},{"interface":"front","average":{"1min":1.198,"5min":1.198,"15min":1.198},"min":{"1min":0.98399999999999999,"5min":0.98399999999999999,"15min":0.98399999999999999},"max":{"1min":1.6719999999999999,"5min":1.6719999999999999,"15min":1.6719999999999999},"last":1.1719999999999999}]}]},{"osd":6,"up_from":16,"seq":68719476761,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297776,"kb_used_data":488,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074064,"statfs":{"total":96636764160,"available":96331841536,"internally_reserved":0,"allocated":499712,"data_stored":220402,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Oct 6 03:30:26 2023","interfaces":[{"interface":"back","average":{"1min":0.876,"5min":0.876,"15min":0.876},"min":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"max":{"1min":1.149,"5min":1.149,"15min":1.149},"last":0.77000000000000002},{"interface":"front","average":{"1min":0.90400000000000003,"5min":0.90400000000000003,"15min":0.90400000000000003},"min":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"max":{"1min":1.254,"5min":1.254,"15min":1.254},"last":0.70599999999999996}]},{"osd":1,"last update":"Fri Oct 6 03:30:26 2023","interfaces":[{"interface":"back","average":{"1min":0.81299999999999994,"5min":0.81299999999999994,"15min":0.81299999999999994},"min":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"max":{"1min":1.4730000000000001,"5min":1.4730000000000001,"15min":1.4730000000000001},"last":1.1459999999999999},{"interface":"front","average":{"1min":0.97099999999999997,"5min":0.97099999999999997,"15min":0.97099999999999997},"min":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"max":{"1min":1.3440000000000001,"5min":1.3440000000000001,"15min":1.3440000000000001},"last":1.127}]},{"osd":2,"last update":"Fri Oct 6 03:30:26 2023","interfaces":[{"interface":"back","average":{"1min":0.94099999999999995,"5min":0.94099999999999995,"15min":0.94099999999999995},"min":{"1min":0.629,"5min":0.629,"15min":0.629},"max":{"1min":1.236,"5min":1.236,"15min":1.236},"last":0.90400000000000003},{"interface":"front","average":{"1min":0.95599999999999996,"5min":0.95599999999999996,"15min":0.95599999999999996},"min":{"1min":0.623,"5min":0.623,"15min":0.623},"max":{"1min":1.3660000000000001,"5min":1.3660000000000001,"15min":1.3660000000000001},"last":0.82799999999999996}]},{"osd":3,"last update":"Fri Oct 6 03:30:26 2023","interfaces":[{"interface":"back","average":{"1min":0.99299999999999999,"5min":0.99299999999999999,"15min":0.99299999999999999},"min":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":1.3660000000000001,"5min":1.3660000000000001,"15min":1.3660000000000001},"last":0.98099999999999998},{"interface":"front","average":{"1min":0.95599999999999996,"5min":0.95599999999999996,"15min":0.95599999999999996},"min":{"1min":0.76300000000000001,"5min":0.76300000000000001,"15min":0.76300000000000001},"max":{"1min":1.2609999999999999,"5min":1.2609999999999999,"15min":1.2609999999999999},"last":1.008}]},{"osd":4,"last update":"Fri Oct 6 03:30:26 2023","interfaces":[{"interface":"back","average":{"1min":1.1040000000000001,"5min":1.1040000000000001,"15min":1.1040000000000001},"min":{"1min":0.875,"5min":0.875,"15min":0.875},"max":{"1min":1.2869999999999999,"5min":1.2869999999999999,"15min":1.2869999999999999},"last":1.0700000000000001},{"interface":"front","average":{"1min":1.1140000000000001,"5min":1.1140000000000001,"15min":1.1140000000000001},"min":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.73399999999999999},"max":{"1min":1.4450000000000001,"5min":1.4450000000000001,"15min":1.4450000000000001},"last":1.194}]},{"osd":5,"last update":"Fri Oct 6 03:30:26 2023","interfaces":[{"interface":"back","average":{"1min":1.105,"5min":1.105,"15min":1.105},"min":{"1min":0.86799999999999999,"5min":0.86799999999999999,"15min":0.86799999999999999},"max":{"1min":1.613,"5min":1.613,"15min":1.613},"last":1.5620000000000001},{"interface":"front","average":{"1min":1.1699999999999999,"5min":1.1699999999999999,"15min":1.1699999999999999},"min":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"max":{"1min":1.351,"5min":1.351,"15min":1.351},"last":1.171}]},{"osd":7,"last update":"Fri Oct 6 03:30:26 2023","interfaces":[{"interface":"back","average":{"1min":1.2370000000000001,"5min":1.2370000000000001,"15min":1.2370000000000001},"min":{"1min":1.012,"5min":1.012,"15min":1.012},"max":{"1min":1.5409999999999999,"5min":1.5409999999999999,"15min":1.5409999999999999},"last":1.0980000000000001},{"interface":"front","average":{"1min":1.212,"5min":1.212,"15min":1.212},"min":{"1min":0.96099999999999997,"5min":0.96099999999999997,"15min":0.96099999999999997},"max":{"1min":1.704,"5min":1.704,"15min":1.704},"last":1.6539999999999999}]}]},{"osd":5,"up_from":16,"seq":68719476761,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298116,"kb_used_data":828,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94073724,"statfs":{"total":96636764160,"available":96331493376,"internally_reserved":0,"allocated":847872,"data_stored":562776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Oct 6 03:30:23 2023","interfaces":[{"interface":"back","average":{"1min":0.86399999999999999,"5min":0.86399999999999999,"15min":0.86399999999999999},"min":{"1min":0.505,"5min":0.505,"15min":0.505},"max":{"1min":1.4079999999999999,"5min":1.4079999999999999,"15min":1.4079999999999999},"last":1.141},{"interface":"front","average":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":0.89800000000000002},"min":{"1min":0.64500000000000002,"5min":0.64500000000000002,"15min":0.64500000000000002},"max":{"1min":1.3779999999999999,"5min":1.3779999999999999,"15min":1.3779999999999999},"last":0.93700000000000006}]},{"osd":1,"last update":"Fri Oct 6 03:30:23 2023","interfaces":[{"interface":"back","average":{"1min":0.97799999999999998,"5min":0.97799999999999998,"15min":0.97799999999999998},"min":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":1.2829999999999999,"5min":1.2829999999999999,"15min":1.2829999999999999},"last":0.66200000000000003},{"interface":"front","average":{"1min":0.85999999999999999,"5min":0.85999999999999999,"15min":0.85999999999999999},"min":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"max":{"1min":1.581,"5min":1.581,"15min":1.581},"last":0.72599999999999998}]},{"osd":2,"last update":"Fri Oct 6 03:30:23 2023","interfaces":[{"interface":"back","average":{"1min":1,"5min":1,"15min":1},"min":{"1min":0.67300000000000004,"5min":0.67300000000000004,"15min":0.67300000000000004},"max":{"1min":1.258,"5min":1.258,"15min":1.258},"last":1.1919999999999999},{"interface":"front","average":{"1min":0.90600000000000003,"5min":0.90600000000000003,"15min":0.90600000000000003},"min":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.67100000000000004},"max":{"1min":1.367,"5min":1.367,"15min":1.367},"last":0.84099999999999997}]},{"osd":3,"last update":"Fri Oct 6 03:30:23 2023","interfaces":[{"interface":"back","average":{"1min":1.034,"5min":1.034,"15min":1.034},"min":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.77200000000000002},"max":{"1min":1.4079999999999999,"5min":1.4079999999999999,"15min":1.4079999999999999},"last":1.0920000000000001},{"interface":"front","average":{"1min":1.034,"5min":1.034,"15min":1.034},"min":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"max":{"1min":1.244,"5min":1.244,"15min":1.244},"last":0.999}]},{"osd":4,"last update":"Fri Oct 6 03:30:23 2023","interfaces":[{"interface":"back","average":{"1min":1.0920000000000001,"5min":1.0920000000000001,"15min":1.0920000000000001},"min":{"1min":0.752,"5min":0.752,"15min":0.752},"max":{"1min":2.2050000000000001,"5min":2.2050000000000001,"15min":2.2050000000000001},"last":1.2310000000000001},{"interface":"front","average":{"1min":1.1870000000000001,"5min":1.1870000000000001,"15min":1.1870000000000001},"min":{"1min":0.90900000000000003,"5min":0.90900000000000003,"15min":0.90900000000000003},"max":{"1min":1.518,"5min":1.518,"15min":1.518},"last":1.2629999999999999}]},{"osd":6,"last update":"Fri Oct 6 03:30:23 2023","interfaces":[{"interface":"back","average":{"1min":1.1040000000000001,"5min":1.1040000000000001,"15min":1.1040000000000001},"min":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"max":{"1min":1.3169999999999999,"5min":1.3169999999999999,"15min":1.3169999999999999},"last":1.4259999999999999},{"interface":"front","average":{"1min":1.2370000000000001,"5min":1.2370000000000001,"15min":1.2370000000000001},"min":{"1min":0.94599999999999995,"5min":0.94599999999999995,"15min":0.94599999999999995},"max":{"1min":2.3809999999999998,"5min":2.3809999999999998,"15min":2.3809999999999998},"last":1.3859999999999999}]},{"osd":7,"last update":"Fri Oct 6 03:30:23 2023","interfaces":[{"interface":"back","average":{"1min":1.2370000000000001,"5min":1.2370000000000001,"15min":1.2370000000000001},"min":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.79500000000000004},"max":{"1min":1.637,"5min":1.637,"15min":1.637},"last":1.6060000000000001},{"interface":"front","average":{"1min":1.2929999999999999,"5min":1.2929999999999999,"15min":1.2929999999999999},"min":{"1min":1.0429999999999999,"5min":1.0429999999999999,"15min":1.0429999999999999},"max":{"1min":2.3330000000000002,"5min":2.3330000000000002,"15min":2.3330000000000002},"last":1.339}]}]},{"osd":4,"up_from":16,"seq":68719476760,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298116,"kb_used_data":828,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94073724,"statfs":{"total":96636764160,"available":96331493376,"internally_reserved":0,"allocated":847872,"data_stored":564896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Oct 6 03:30:25 2023","interfaces":[{"interface":"back","average":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"min":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"max":{"1min":1.129,"5min":1.129,"15min":1.129},"last":0.71999999999999997},{"interface":"front","average":{"1min":0.96299999999999997,"5min":0.96299999999999997,"15min":0.96299999999999997},"min":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"max":{"1min":1.629,"5min":1.629,"15min":1.629},"last":0.55800000000000005}]},{"osd":1,"last update":"Fri Oct 6 03:30:25 2023","interfaces":[{"interface":"back","average":{"1min":0.89300000000000002,"5min":0.89300000000000002,"15min":0.89300000000000002},"min":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"max":{"1min":1.23,"5min":1.23,"15min":1.23},"last":1.0680000000000001},{"interface":"front","average":{"1min":0.86799999999999999,"5min":0.86799999999999999,"15min":0.86799999999999999},"min":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"max":{"1min":1.3680000000000001,"5min":1.3680000000000001,"15min":1.3680000000000001},"last":0.59899999999999998}]},{"osd":2,"last update":"Fri Oct 6 03:30:25 2023","interfaces":[{"interface":"back","average":{"1min":0.98999999999999999,"5min":0.98999999999999999,"15min":0.98999999999999999},"min":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":1.5549999999999999,"5min":1.5549999999999999,"15min":1.5549999999999999},"last":0.91300000000000003},{"interface":"front","average":{"1min":0.92100000000000004,"5min":0.92100000000000004,"15min":0.92100000000000004},"min":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.60799999999999998},"max":{"1min":1.306,"5min":1.306,"15min":1.306},"last":0.63400000000000001}]},{"osd":3,"last update":"Fri Oct 6 03:30:25 2023","interfaces":[{"interface":"back","average":{"1min":1.1040000000000001,"5min":1.1040000000000001,"15min":1.1040000000000001},"min":{"1min":0.84499999999999997,"5min":0.84499999999999997,"15min":0.84499999999999997},"max":{"1min":1.5580000000000001,"5min":1.5580000000000001,"15min":1.5580000000000001},"last":1.0229999999999999},{"interface":"front","average":{"1min":1.04,"5min":1.04,"15min":1.04},"min":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.67500000000000004},"max":{"1min":1.6699999999999999,"5min":1.6699999999999999,"15min":1.6699999999999999},"last":0.98699999999999999}]},{"osd":5,"last update":"Fri Oct 6 03:30:25 2023","interfaces":[{"interface":"back","average":{"1min":1.044,"5min":1.044,"15min":1.044},"min":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.73399999999999999},"max":{"1min":1.5169999999999999,"5min":1.5169999999999999,"15min":1.5169999999999999},"last":0.94799999999999995},{"interface":"front","average":{"1min":1.266,"5min":1.266,"15min":1.266},"min":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":0.82799999999999996},"max":{"1min":2.2029999999999998,"5min":2.2029999999999998,"15min":2.2029999999999998},"last":0.79500000000000004}]},{"osd":6,"last update":"Fri Oct 6 03:30:25 2023","interfaces":[{"interface":"back","average":{"1min":1.181,"5min":1.181,"15min":1.181},"min":{"1min":0.79900000000000004,"5min":0.79900000000000004,"15min":0.79900000000000004},"max":{"1min":2.0209999999999999,"5min":2.0209999999999999,"15min":2.0209999999999999},"last":0.76600000000000001},{"interface":"front","average":{"1min":1.179,"5min":1.179,"15min":1.179},"min":{"1min":0.71399999999999997,"5min":0.71399999999999997,"15min":0.71399999999999997},"max":{"1min":2.1440000000000001,"5min":2.1440000000000001,"15min":2.1440000000000001},"last":0.85399999999999998}]},{"osd":7,"last update":"Fri Oct 6 03:30:25 2023","interfaces":[{"interface":"back","average":{"1min":1.337,"5min":1.337,"15min":1.337},"min":{"1min":0.94399999999999995,"5min":0.94399999999999995,"15min":0.94399999999999995},"max":{"1min":2.0939999999999999,"5min":2.0939999999999999,"15min":2.0939999999999999},"last":1.1080000000000001},{"interface":"front","average":{"1min":1.3089999999999999,"5min":1.3089999999999999,"15min":1.3089999999999999},"min":{"1min":0.95899999999999996,"5min":0.95899999999999996,"15min":0.95899999999999996},"max":{"1min":1.968,"5min":1.968,"15min":1.968},"last":1.147}]}]},{"osd":3,"up_from":14,"seq":60129542168,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302204,"kb_used_data":820,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94069636,"statfs":{"total":96636764160,"available":96327307264,"internally_reserved":0,"allocated":839680,"data_stored":555463,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Oct 6 03:30:26 2023","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.81899999999999995,"5min":0.81899999999999995,"15min":0.81899999999999995},"last":0.86899999999999999},{"interface":"front","average":{"1min":0.754,"5min":0.754,"15min":0.754},"min":{"1min":0.505,"5min":0.505,"15min":0.505},"max":{"1min":1.0920000000000001,"5min":1.0920000000000001,"15min":1.0920000000000001},"last":0.54800000000000004}]},{"osd":1,"last update":"Fri Oct 6 03:30:20 2023","interfaces":[{"interface":"back","average":{"1min":0.83199999999999996,"5min":0.83199999999999996,"15min":0.83199999999999996},"min":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":1.2210000000000001,"5min":1.2210000000000001,"15min":1.2210000000000001},"last":0.79600000000000004},{"interface":"front","average":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":1.321,"5min":1.321,"15min":1.321},"last":0.60999999999999999}]},{"osd":2,"last update":"Fri Oct 6 03:30:20 2023","interfaces":[{"interface":"back","average":{"1min":0.86799999999999999,"5min":0.86799999999999999,"15min":0.86799999999999999},"min":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.61199999999999999},"max":{"1min":1.228,"5min":1.228,"15min":1.228},"last":0.92900000000000005},{"interface":"front","average":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"min":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":1.3680000000000001,"5min":1.3680000000000001,"15min":1.3680000000000001},"last":0.47699999999999998}]},{"osd":4,"last update":"Fri Oct 6 03:30:26 2023","interfaces":[{"interface":"back","average":{"1min":1.0129999999999999,"5min":1.0129999999999999,"15min":1.0129999999999999},"min":{"1min":0.76800000000000002,"5min":0.76800000000000002,"15min":0.76800000000000002},"max":{"1min":1.6359999999999999,"5min":1.6359999999999999,"15min":1.6359999999999999},"last":0.98599999999999999},{"interface":"front","average":{"1min":0.96699999999999997,"5min":0.96699999999999997,"15min":0.96699999999999997},"min":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"max":{"1min":1.409,"5min":1.409,"15min":1.409},"last":0.91000000000000003}]},{"osd":5,"last update":"Fri Oct 6 03:30:26 2023","interfaces":[{"interface":"back","average":{"1min":0.98999999999999999,"5min":0.98999999999999999,"15min":0.98999999999999999},"min":{"1min":0.746,"5min":0.746,"15min":0.746},"max":{"1min":1.387,"5min":1.387,"15min":1.387},"last":0.96599999999999997},{"interface":"front","average":{"1min":1.1359999999999999,"5min":1.1359999999999999,"15min":1.1359999999999999},"min":{"1min":0.89000000000000001,"5min":0.89000000000000001,"15min":0.89000000000000001},"max":{"1min":1.615,"5min":1.615,"15min":1.615},"last":1.0269999999999999}]},{"osd":6,"last update":"Fri Oct 6 03:30:26 2023","interfaces":[{"interface":"back","average":{"1min":1.079,"5min":1.079,"15min":1.079},"min":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"max":{"1min":1.643,"5min":1.643,"15min":1.643},"last":0.66800000000000004},{"interface":"front","average":{"1min":1.105,"5min":1.105,"15min":1.105},"min":{"1min":0.74399999999999999,"5min":0.74399999999999999,"15min":0.74399999999999999},"max":{"1min":1.6819999999999999,"5min":1.6819999999999999,"15min":1.6819999999999999},"last":1.0509999999999999}]},{"osd":7,"last update":"Fri Oct 6 03:30:26 2023","interfaces":[{"interface":"back","average":{"1min":1.1850000000000001,"5min":1.1850000000000001,"15min":1.1850000000000001},"min":{"1min":0.92000000000000004,"5min":0.92000000000000004,"15min":0.92000000000000004},"max":{"1min":1.597,"5min":1.597,"15min":1.597},"last":0.70799999999999996},{"interface":"front","average":{"1min":1.111,"5min":1.111,"15min":1.111},"min":{"1min":0.86599999999999999,"5min":0.86599999999999999,"15min":0.86599999999999999},"max":{"1min":1.446,"5min":1.446,"15min":1.446},"last":0.83399999999999996}]}]},{"osd":2,"up_from":14,"seq":60129542167,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":301908,"kb_used_data":524,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94069932,"statfs":{"total":96636764160,"available":96327610368,"internally_reserved":0,"allocated":536576,"data_stored":245337,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Oct 6 03:30:21 2023","interfaces":[{"interface":"back","average":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":1.1240000000000001,"5min":1.1240000000000001,"15min":1.1240000000000001},"last":0.76700000000000002},{"interface":"front","average":{"1min":0.70399999999999996,"5min":0.70399999999999996,"15min":0.70399999999999996},"min":{"1min":0.495,"5min":0.495,"15min":0.495},"max":{"1min":1.1559999999999999,"5min":1.1559999999999999,"15min":1.1559999999999999},"last":0.495}]},{"osd":1,"last update":"Fri Oct 6 03:30:21 2023","interfaces":[{"interface":"back","average":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"min":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":1.1060000000000001,"5min":1.1060000000000001,"15min":1.1060000000000001},"last":1.0449999999999999},{"interface":"front","average":{"1min":0.73499999999999999,"5min":0.73499999999999999,"15min":0.73499999999999999},"min":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":1.2390000000000001,"5min":1.2390000000000001,"15min":1.2390000000000001},"last":0.52700000000000002}]},{"osd":3,"last update":"Fri Oct 6 03:30:21 2023","interfaces":[{"interface":"back","average":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":0.78500000000000003},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":1.165,"5min":1.165,"15min":1.165},"last":0.66300000000000003},{"interface":"front","average":{"1min":0.873,"5min":0.873,"15min":0.873},"min":{"1min":0.626,"5min":0.626,"15min":0.626},"max":{"1min":1.2949999999999999,"5min":1.2949999999999999,"15min":1.2949999999999999},"last":0.45300000000000001}]},{"osd":4,"last update":"Fri Oct 6 03:30:23 2023","interfaces":[{"interface":"back","average":{"1min":1.095,"5min":1.095,"15min":1.095},"min":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"max":{"1min":2.3450000000000002,"5min":2.3450000000000002,"15min":2.3450000000000002},"last":0.86599999999999999},{"interface":"front","average":{"1min":0.95599999999999996,"5min":0.95599999999999996,"15min":0.95599999999999996},"min":{"1min":0.74399999999999999,"5min":0.74399999999999999,"15min":0.74399999999999999},"max":{"1min":1.1950000000000001,"5min":1.1950000000000001,"15min":1.1950000000000001},"last":0.83399999999999996}]},{"osd":5,"last update":"Fri Oct 6 03:30:23 2023","interfaces":[{"interface":"back","average":{"1min":1.139,"5min":1.139,"15min":1.139},"min":{"1min":0.79900000000000004,"5min":0.79900000000000004,"15min":0.79900000000000004},"max":{"1min":2.5169999999999999,"5min":2.5169999999999999,"15min":2.5169999999999999},"last":0.93600000000000005},{"interface":"front","average":{"1min":1.107,"5min":1.107,"15min":1.107},"min":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":0.81799999999999995},"max":{"1min":1.7250000000000001,"5min":1.7250000000000001,"15min":1.7250000000000001},"last":0.81299999999999994}]},{"osd":6,"last update":"Fri Oct 6 03:30:23 2023","interfaces":[{"interface":"back","average":{"1min":1.014,"5min":1.014,"15min":1.014},"min":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"max":{"1min":1.218,"5min":1.218,"15min":1.218},"last":0.55200000000000005},{"interface":"front","average":{"1min":1.1899999999999999,"5min":1.1899999999999999,"15min":1.1899999999999999},"min":{"1min":0.85899999999999999,"5min":0.85899999999999999,"15min":0.85899999999999999},"max":{"1min":2.2450000000000001,"5min":2.2450000000000001,"15min":2.2450000000000001},"last":1.0029999999999999}]},{"osd":7,"last update":"Fri Oct 6 03:30:21 2023","interfaces":[{"interface":"back","average":{"1min":1.1639999999999999,"5min":1.1639999999999999,"15min":1.1639999999999999},"min":{"1min":0.81599999999999995,"5min":0.81599999999999995,"15min":0.81599999999999995},"max":{"1min":1.8140000000000001,"5min":1.8140000000000001,"15min":1.8140000000000001},"last":0.70299999999999996},{"interface":"front","average":{"1min":1.0860000000000001,"5min":1.0860000000000001,"15min":1.0860000000000001},"min":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.79500000000000004},"max":{"1min":1.2749999999999999,"5min":1.2749999999999999,"15min":1.2749999999999999},"last":0.71599999999999997}]}]},{"osd":1,"up_from":14,"seq":60129542167,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302204,"kb_used_data":820,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94069636,"statfs":{"total":96636764160,"available":96327307264,"internally_reserved":0,"allocated":839680,"data_stored":555576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Oct 6 03:30:26 2023","interfaces":[{"interface":"back","average":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.60799999999999998},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.48599999999999999},{"interface":"front","average":{"1min":0.71599999999999997,"5min":0.71599999999999997,"15min":0.71599999999999997},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":1.5329999999999999,"5min":1.5329999999999999,"15min":1.5329999999999999},"last":0.59199999999999997}]},{"osd":2,"last update":"Fri Oct 6 03:30:20 2023","interfaces":[{"interface":"back","average":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.93899999999999995,"5min":0.93899999999999995,"15min":0.93899999999999995},"last":1.018},{"interface":"front","average":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"min":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"max":{"1min":1.278,"5min":1.278,"15min":1.278},"last":0.66600000000000004}]},{"osd":3,"last update":"Fri Oct 6 03:30:20 2023","interfaces":[{"interface":"back","average":{"1min":0.72899999999999998,"5min":0.72899999999999998,"15min":0.72899999999999998},"min":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"max":{"1min":0.90500000000000003,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.53700000000000003},{"interface":"front","average":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.3700000000000001,"5min":1.3700000000000001,"15min":1.3700000000000001},"last":0.80600000000000005}]},{"osd":4,"last update":"Fri Oct 6 03:30:26 2023","interfaces":[{"interface":"back","average":{"1min":0.94299999999999995,"5min":0.94299999999999995,"15min":0.94299999999999995},"min":{"1min":0.66400000000000003,"5min":0.66400000000000003,"15min":0.66400000000000003},"max":{"1min":1.7549999999999999,"5min":1.7549999999999999,"15min":1.7549999999999999},"last":0.96799999999999997},{"interface":"front","average":{"1min":0.97499999999999998,"5min":0.97499999999999998,"15min":0.97499999999999998},"min":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":1.415,"5min":1.415,"15min":1.415},"last":1.2250000000000001}]},{"osd":5,"last update":"Fri Oct 6 03:30:26 2023","interfaces":[{"interface":"back","average":{"1min":0.93999999999999995,"5min":0.93999999999999995,"15min":0.93999999999999995},"min":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"max":{"1min":1.4910000000000001,"5min":1.4910000000000001,"15min":1.4910000000000001},"last":1.0980000000000001},{"interface":"front","average":{"1min":1.0049999999999999,"5min":1.0049999999999999,"15min":1.0049999999999999},"min":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":0.70499999999999996},"max":{"1min":1.4370000000000001,"5min":1.4370000000000001,"15min":1.4370000000000001},"last":1.01}]},{"osd":6,"last update":"Fri Oct 6 03:30:26 2023","interfaces":[{"interface":"back","average":{"1min":1.022,"5min":1.022,"15min":1.022},"min":{"1min":0.69199999999999995,"5min":0.69199999999999995,"15min":0.69199999999999995},"max":{"1min":1.603,"5min":1.603,"15min":1.603},"last":1.3},{"interface":"front","average":{"1min":1.1080000000000001,"5min":1.1080000000000001,"15min":1.1080000000000001},"min":{"1min":0.69199999999999995,"5min":0.69199999999999995,"15min":0.69199999999999995},"max":{"1min":1.5069999999999999,"5min":1.5069999999999999,"15min":1.5069999999999999},"last":1.365}]},{"osd":7,"last update":"Fri Oct 6 03:30:26 2023","interfaces":[{"interface":"back","average":{"1min":1.04,"5min":1.04,"15min":1.04},"min":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.60799999999999998},"max":{"1min":1.6160000000000001,"5min":1.6160000000000001,"15min":1.6160000000000001},"last":0.98299999999999998},{"interface":"front","average":{"1min":1.071,"5min":1.071,"15min":1.071},"min":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.67500000000000004},"max":{"1min":1.5700000000000001,"5min":1.5700000000000001,"15min":1.5700000000000001},"last":1.1299999999999999}]}]},{"osd":0,"up_from":15,"seq":64424509465,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302468,"kb_used_data":1084,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94069372,"statfs":{"total":96636764160,"available":96327036928,"internally_reserved":0,"allocated":1110016,"data_stored":824876,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Oct 6 03:30:22 2023","interfaces":[{"interface":"back","average":{"1min":0.629,"5min":0.629,"15min":0.629},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":1.0029999999999999,"5min":1.0029999999999999,"15min":1.0029999999999999},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.74199999999999999},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":1.179,"5min":1.179,"15min":1.179},"last":0.63300000000000001}]},{"osd":2,"last update":"Fri Oct 6 03:30:22 2023","interfaces":[{"interface":"back","average":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":1.1879999999999999,"5min":1.1879999999999999,"15min":1.1879999999999999},"last":0.439},{"interface":"front","average":{"1min":0.72499999999999998,"5min":0.72499999999999998,"15min":0.72499999999999998},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":1.3879999999999999,"5min":1.3879999999999999,"15min":1.3879999999999999},"last":0.52300000000000002}]},{"osd":3,"last update":"Fri Oct 6 03:30:22 2023","interfaces":[{"interface":"back","average":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":1.1930000000000001,"5min":1.1930000000000001,"15min":1.1930000000000001},"last":0.67500000000000004},{"interface":"front","average":{"1min":0.86399999999999999,"5min":0.86399999999999999,"15min":0.86399999999999999},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":1.1970000000000001,"5min":1.1970000000000001,"15min":1.1970000000000001},"last":0.71399999999999997}]},{"osd":4,"last update":"Fri Oct 6 03:30:22 2023","interfaces":[{"interface":"back","average":{"1min":1.0189999999999999,"5min":1.0189999999999999,"15min":1.0189999999999999},"min":{"1min":0.72899999999999998,"5min":0.72899999999999998,"15min":0.72899999999999998},"max":{"1min":1.55,"5min":1.55,"15min":1.55},"last":0.98499999999999999},{"interface":"front","average":{"1min":0.99099999999999999,"5min":0.99099999999999999,"15min":0.99099999999999999},"min":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"max":{"1min":1.5269999999999999,"5min":1.5269999999999999,"15min":1.5269999999999999},"last":1.2969999999999999}]},{"osd":5,"last update":"Fri Oct 6 03:30:22 2023","interfaces":[{"interface":"back","average":{"1min":0.98499999999999999,"5min":0.98499999999999999,"15min":0.98499999999999999},"min":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"max":{"1min":1.5409999999999999,"5min":1.5409999999999999,"15min":1.5409999999999999},"last":0.91100000000000003},{"interface":"front","average":{"1min":1.087,"5min":1.087,"15min":1.087},"min":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"max":{"1min":1.353,"5min":1.353,"15min":1.353},"last":1.24}]},{"osd":6,"last update":"Fri Oct 6 03:30:22 2023","interfaces":[{"interface":"back","average":{"1min":1.145,"5min":1.145,"15min":1.145},"min":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"max":{"1min":1.502,"5min":1.502,"15min":1.502},"last":1.1859999999999999},{"interface":"front","average":{"1min":1.1299999999999999,"5min":1.1299999999999999,"15min":1.1299999999999999},"min":{"1min":0.64900000000000002,"5min":0.64900000000000002,"15min":0.64900000000000002},"max":{"1min":1.6439999999999999,"5min":1.6439999999999999,"15min":1.6439999999999999},"last":1.085}]},{"osd":7,"last update":"Fri Oct 6 03:30:22 2023","interfaces":[{"interface":"back","average":{"1min":1.099,"5min":1.099,"15min":1.099},"min":{"1min":0.874,"5min":0.874,"15min":0.874},"max":{"1min":1.369,"5min":1.369,"15min":1.369},"last":1.046},{"interface":"front","average":{"1min":1.2030000000000001,"5min":1.2030000000000001,"15min":1.2030000000000001},"min":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"max":{"1min":1.573,"5min":1.573,"15min":1.573},"last":1.0209999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-06T03:31:08.478 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-06T03:31:08.583 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:08.576+0000 7f4b16103700 1 Processor -- start 2023-10-06T03:31:08.583 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:08.576+0000 7f4b16103700 1 -- start start 2023-10-06T03:31:08.584 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:08.580+0000 7f4b16103700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4b100cbfc0 0x7f4b100c7430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:31:08.584 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:08.580+0000 7f4b16103700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b10156c30 con 0x7f4b100aaca0 2023-10-06T03:31:08.584 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:08.580+0000 7f4b16103700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b10156300 con 0x7f4b100c7970 2023-10-06T03:31:08.584 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:08.580+0000 7f4b16103700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b101565e0 con 0x7f4b100cbfc0 2023-10-06T03:31:08.584 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:08.580+0000 7f4b15101700 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f4b100aaca0 0x7f4b100cb8b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:60366/0 (socket says 172.21.15.110:60366) 2023-10-06T03:31:08.585 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:08.580+0000 7f4b15101700 1 -- 172.21.15.110:0/2082592223 learned_addr learned my addr 172.21.15.110:0/2082592223 (peer_addr_for_me v1:172.21.15.110:0/0) 2023-10-06T03:31:08.585 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:08.580+0000 7f4b14900700 1 --2- 172.21.15.110:0/2082592223 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4b100cbfc0 0x7f4b100c7430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:31:08.586 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:08.580+0000 7f4b07fff700 1 -- 172.21.15.110:0/2082592223 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3080673161 0 0) 0x7f4b10156c30 con 0x7f4b100aaca0 2023-10-06T03:31:08.586 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:08.580+0000 7f4b07fff700 1 -- 172.21.15.110:0/2082592223 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4afc003570 con 0x7f4b100aaca0 2023-10-06T03:31:08.586 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:08.580+0000 7f4b07fff700 1 -- 172.21.15.110:0/2082592223 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 410338767 0 0) 0x7f4b10156300 con 0x7f4b100c7970 2023-10-06T03:31:08.586 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:08.580+0000 7f4b07fff700 1 -- 172.21.15.110:0/2082592223 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4b10156c30 con 0x7f4b100c7970 2023-10-06T03:31:08.586 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:08.580+0000 7f4b14900700 1 -- 172.21.15.110:0/2082592223 >> v1:172.21.15.192:6789/0 conn(0x7f4b100c7970 legacy=0x7f4b1006a2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:31:08.587 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:08.580+0000 7f4b14900700 1 -- 172.21.15.110:0/2082592223 >> v1:172.21.15.110:6789/0 conn(0x7f4b100aaca0 legacy=0x7f4b100cb8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:31:08.587 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:08.580+0000 7f4b07fff700 1 -- 172.21.15.110:0/2082592223 <== mon.0 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2414679169 0 0) 0x7f4afc003570 con 0x7f4b100aaca0 2023-10-06T03:31:08.587 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:08.580+0000 7f4b14900700 1 -- 172.21.15.110:0/2082592223 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b101568c0 con 0x7f4b100cbfc0 2023-10-06T03:31:08.587 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:08.580+0000 7f4b14900700 1 --2- 172.21.15.110:0/2082592223 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4b100cbfc0 0x7f4b100c7430 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f4b0c009910 tx=0x7f4b0c0091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2edf86476e541a8c server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:31:08.587 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:08.580+0000 7f4b07fff700 1 -- 172.21.15.110:0/2082592223 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b00001470 con 0x7f4b100cbfc0 2023-10-06T03:31:08.587 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:08.580+0000 7f4b07fff700 1 -- 172.21.15.110:0/2082592223 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b0c011420 con 0x7f4b100cbfc0 2023-10-06T03:31:08.587 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:08.580+0000 7f4b07fff700 1 -- 172.21.15.110:0/2082592223 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b0c0117c0 con 0x7f4b100cbfc0 2023-10-06T03:31:08.588 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:08.580+0000 7f4b16103700 1 -- 172.21.15.110:0/2082592223 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4b100cbfc0 msgr2=0x7f4b100c7430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:31:08.588 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:08.580+0000 7f4b16103700 1 --2- 172.21.15.110:0/2082592223 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4b100cbfc0 0x7f4b100c7430 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f4b0c009910 tx=0x7f4b0c0091a0 comp rx=0 tx=0).stop 2023-10-06T03:31:08.588 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:08.580+0000 7f4b16103700 1 -- 172.21.15.110:0/2082592223 shutdown_connections 2023-10-06T03:31:08.588 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:08.580+0000 7f4b16103700 1 --2- 172.21.15.110:0/2082592223 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4b100cbfc0 0x7f4b100c7430 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:08.588 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:08.580+0000 7f4b16103700 1 -- 172.21.15.110:0/2082592223 >> 172.21.15.110:0/2082592223 conn(0x7f4b1006c760 msgr2=0x7f4b1006cb60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:31:08.588 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:08.580+0000 7f4b16103700 1 -- 172.21.15.110:0/2082592223 shutdown_connections 2023-10-06T03:31:08.588 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:08.580+0000 7f4b16103700 1 -- 172.21.15.110:0/2082592223 wait complete. 2023-10-06T03:31:08.589 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:08.584+0000 7f4b16103700 1 Processor -- start 2023-10-06T03:31:08.589 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:08.584+0000 7f4b16103700 1 -- start start 2023-10-06T03:31:08.591 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:08.584+0000 7f4b16103700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4b100aaca0 0x7f4b100c1c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:31:08.591 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:08.584+0000 7f4b16103700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4b100c7970 0x7f4b100c31d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:31:08.591 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:08.584+0000 7f4b16103700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4b100cbfc0 0x7f4b10154220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:31:08.591 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:08.584+0000 7f4b16103700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b100ac060 con 0x7f4b100c7970 2023-10-06T03:31:08.592 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:08.584+0000 7f4b16103700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b100ab450 con 0x7f4b100cbfc0 2023-10-06T03:31:08.592 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:08.584+0000 7f4b16103700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b100ab730 con 0x7f4b100aaca0 2023-10-06T03:31:08.592 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:08.584+0000 7f4b14900700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4b100c7970 0x7f4b100c31d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:31:08.592 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:08.584+0000 7f4b14900700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4b100c7970 0x7f4b100c31d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:38902/0 (socket says 172.21.15.110:38902) 2023-10-06T03:31:08.592 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:08.584+0000 7f4b14900700 1 -- 172.21.15.110:0/511624643 learned_addr learned my addr 172.21.15.110:0/511624643 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:31:08.592 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:08.584+0000 7f4b15101700 1 --2- 172.21.15.110:0/511624643 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4b100aaca0 0x7f4b100c1c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:31:08.592 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:08.584+0000 7f4b15902700 1 --2- 172.21.15.110:0/511624643 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4b100cbfc0 0x7f4b10154220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:31:08.592 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:08.584+0000 7f4b14900700 1 -- 172.21.15.110:0/511624643 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4b100aaca0 msgr2=0x7f4b100c1c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:31:08.592 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:08.584+0000 7f4b14900700 1 --2- 172.21.15.110:0/511624643 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4b100aaca0 0x7f4b100c1c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:08.592 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:08.584+0000 7f4b14900700 1 -- 172.21.15.110:0/511624643 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4b100cbfc0 msgr2=0x7f4b10154220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:31:08.592 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:08.584+0000 7f4b14900700 1 --2- 172.21.15.110:0/511624643 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4b100cbfc0 0x7f4b10154220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:08.592 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:08.584+0000 7f4b14900700 1 -- 172.21.15.110:0/511624643 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b0c009420 con 0x7f4b100c7970 2023-10-06T03:31:08.593 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:08.584+0000 7f4b15902700 1 --2- 172.21.15.110:0/511624643 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4b100cbfc0 0x7f4b10154220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-06T03:31:08.593 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:08.584+0000 7f4b14900700 1 --2- 172.21.15.110:0/511624643 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4b100c7970 0x7f4b100c31d0 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f4b0c009860 tx=0x7f4b0c016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:31:08.593 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:08.584+0000 7f4b057fa700 1 -- 172.21.15.110:0/511624643 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b0c01cc90 con 0x7f4b100c7970 2023-10-06T03:31:08.593 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:08.584+0000 7f4b057fa700 1 -- 172.21.15.110:0/511624643 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b0c01cdf0 con 0x7f4b100c7970 2023-10-06T03:31:08.593 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:08.584+0000 7f4b16103700 1 -- 172.21.15.110:0/511624643 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b100aba10 con 0x7f4b100c7970 2023-10-06T03:31:08.593 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:08.584+0000 7f4b057fa700 1 -- 172.21.15.110:0/511624643 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b0c01a6a0 con 0x7f4b100c7970 2023-10-06T03:31:08.593 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:08.584+0000 7f4b16103700 1 -- 172.21.15.110:0/511624643 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b100ab070 con 0x7f4b100c7970 2023-10-06T03:31:08.594 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:08.588+0000 7f4b16103700 1 -- 172.21.15.110:0/511624643 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b10064770 con 0x7f4b100c7970 2023-10-06T03:31:08.594 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:08.588+0000 7f4b057fa700 1 -- 172.21.15.110:0/511624643 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 13) v1 ==== 80061+0+0 (secure 0 0 0) 0x7f4b0c01aad0 con 0x7f4b100c7970 2023-10-06T03:31:08.594 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:08.588+0000 7f4b057fa700 1 --2- 172.21.15.110:0/511624643 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f4af4060670 0x7f4af4062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:31:08.594 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:08.588+0000 7f4b057fa700 1 -- 172.21.15.110:0/511624643 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 6171+0+0 (secure 0 0 0) 0x7f4b0c07ea80 con 0x7f4b100c7970 2023-10-06T03:31:08.594 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:08.588+0000 7f4b15101700 1 --2- 172.21.15.110:0/511624643 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f4af4060670 0x7f4af4062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:31:08.595 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:08.588+0000 7f4b15101700 1 --2- 172.21.15.110:0/511624643 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f4af4060670 0x7f4af4062b20 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f4b000015a0 tx=0x7f4b00000bc0 comp rx=0 tx=0).ready entity=mgr.5505 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:31:08.600 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:08.596+0000 7f4b057fa700 1 -- 172.21.15.110:0/511624643 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f4b0c04ed90 con 0x7f4b100c7970 2023-10-06T03:31:08.758 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:08.752+0000 7f4b16103700 1 -- 172.21.15.110:0/511624643 --> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f4b10155fe0 con 0x7f4af4060670 2023-10-06T03:31:08.765 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:08.760+0000 7f4b057fa700 1 -- 172.21.15.110:0/511624643 <== mgr.5505 v2:172.21.15.110:6832/33268 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+71219 (secure 0 0 0) 0x7f4b10155fe0 con 0x7f4af4060670 2023-10-06T03:31:08.766 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-10-06T03:31:08.768 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:08.764+0000 7f4b16103700 1 -- 172.21.15.110:0/511624643 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f4af4060670 msgr2=0x7f4af4062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:31:08.768 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:08.764+0000 7f4b16103700 1 --2- 172.21.15.110:0/511624643 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f4af4060670 0x7f4af4062b20 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f4b000015a0 tx=0x7f4b00000bc0 comp rx=0 tx=0).stop 2023-10-06T03:31:08.768 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:08.764+0000 7f4b16103700 1 -- 172.21.15.110:0/511624643 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4b100c7970 msgr2=0x7f4b100c31d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:31:08.768 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:08.764+0000 7f4b16103700 1 --2- 172.21.15.110:0/511624643 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4b100c7970 0x7f4b100c31d0 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f4b0c009860 tx=0x7f4b0c016000 comp rx=0 tx=0).stop 2023-10-06T03:31:08.768 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:08.764+0000 7f4b16103700 1 -- 172.21.15.110:0/511624643 shutdown_connections 2023-10-06T03:31:08.768 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:08.764+0000 7f4b16103700 1 --2- 172.21.15.110:0/511624643 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f4af4060670 0x7f4af4062b20 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:08.768 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:08.764+0000 7f4b16103700 1 --2- 172.21.15.110:0/511624643 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4b100cbfc0 0x7f4b10154220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:08.768 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:08.764+0000 7f4b16103700 1 --2- 172.21.15.110:0/511624643 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4b100c7970 0x7f4b100c31d0 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:08.768 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:08.764+0000 7f4b16103700 1 --2- 172.21.15.110:0/511624643 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4b100aaca0 0x7f4b100c1c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:08.768 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:08.764+0000 7f4b16103700 1 -- 172.21.15.110:0/511624643 >> 172.21.15.110:0/511624643 conn(0x7f4b1006c760 msgr2=0x7f4b100c9c70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:31:08.769 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:08.764+0000 7f4b16103700 1 -- 172.21.15.110:0/511624643 shutdown_connections 2023-10-06T03:31:08.769 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:08.764+0000 7f4b16103700 1 -- 172.21.15.110:0/511624643 wait complete. 2023-10-06T03:31:08.769 INFO:teuthology.orchestra.run.smithi110.stderr:dumped all 2023-10-06T03:31:08.781 INFO:teuthology.orchestra.run.smithi110.stdout:{"pg_ready":true,"pg_map":{"version":9,"stamp":"2023-10-06T03:31:07.946720+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":308,"num_read_kb":512,"num_write":199,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":139,"ondisk_log_size":139,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":210,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2401152,"kb_used_data":6464,"kb_used_omap":0,"kb_used_meta":2394624,"kb_avail":752573568,"statfs":{"total":773094113280,"available":770635333632,"internally_reserved":0,"allocated":6619136,"data_stored":4346957,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2452094976},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-667624,"num_objects":-25,"num_object_clones":0,"num_object_copies":-50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-25,"num_whiteouts":0,"num_read":-317,"num_read_kb":-281,"num_write":-285,"num_write_kb":-2055,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.991630"},"pg_stats":[{"pgid":"1.0","version":"44'139","reported_seq":261,"reported_epoch":44,"state":"active+clean","last_fresh":"2023-10-06T03:31:02.032960+0000","last_change":"2023-10-06T03:29:21.978311+0000","last_active":"2023-10-06T03:31:02.032960+0000","last_peered":"2023-10-06T03:31:02.032960+0000","last_clean":"2023-10-06T03:31:02.032960+0000","last_became_active":"2023-10-06T03:29:21.978017+0000","last_became_peered":"2023-10-06T03:29:21.978017+0000","last_unstale":"2023-10-06T03:31:02.032960+0000","last_undegraded":"2023-10-06T03:31:02.032960+0000","last_fullsized":"2023-10-06T03:31:02.032960+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:20.957955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:20.957955+0000","last_clean_scrub_stamp":"2023-10-06T03:29:20.957955+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T07:33:10.429084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":308,"num_read_kb":512,"num_write":199,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":31,"reported_epoch":44,"state":"active+clean","last_fresh":"2023-10-06T03:31:01.980717+0000","last_change":"2023-10-06T03:29:29.386120+0000","last_active":"2023-10-06T03:31:01.980717+0000","last_peered":"2023-10-06T03:31:01.980717+0000","last_clean":"2023-10-06T03:31:01.980717+0000","last_became_active":"2023-10-06T03:29:29.385692+0000","last_became_peered":"2023-10-06T03:29:29.385692+0000","last_unstale":"2023-10-06T03:31:01.980717+0000","last_undegraded":"2023-10-06T03:31:01.980717+0000","last_fullsized":"2023-10-06T03:31:01.980717+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:28.357506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:28.357506+0000","last_clean_scrub_stamp":"2023-10-06T03:29:28.357506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T10:21:35.864895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":30,"reported_epoch":43,"state":"active+clean","last_fresh":"2023-10-06T03:30:46.743121+0000","last_change":"2023-10-06T03:29:29.384353+0000","last_active":"2023-10-06T03:30:46.743121+0000","last_peered":"2023-10-06T03:30:46.743121+0000","last_clean":"2023-10-06T03:30:46.743121+0000","last_became_active":"2023-10-06T03:29:29.383884+0000","last_became_peered":"2023-10-06T03:29:29.383884+0000","last_unstale":"2023-10-06T03:30:46.743121+0000","last_undegraded":"2023-10-06T03:30:46.743121+0000","last_fullsized":"2023-10-06T03:30:46.743121+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:28.357506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:28.357506+0000","last_clean_scrub_stamp":"2023-10-06T03:29:28.357506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T14:43:09.107140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":32,"reported_epoch":44,"state":"active+clean","last_fresh":"2023-10-06T03:31:01.933050+0000","last_change":"2023-10-06T03:29:29.606195+0000","last_active":"2023-10-06T03:31:01.933050+0000","last_peered":"2023-10-06T03:31:01.933050+0000","last_clean":"2023-10-06T03:31:01.933050+0000","last_became_active":"2023-10-06T03:29:29.605489+0000","last_became_peered":"2023-10-06T03:29:29.605489+0000","last_unstale":"2023-10-06T03:31:01.933050+0000","last_undegraded":"2023-10-06T03:31:01.933050+0000","last_fullsized":"2023-10-06T03:31:01.933050+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:28.357506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:28.357506+0000","last_clean_scrub_stamp":"2023-10-06T03:29:28.357506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T10:21:35.864895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":31,"reported_epoch":44,"state":"active+clean","last_fresh":"2023-10-06T03:31:01.980497+0000","last_change":"2023-10-06T03:29:29.384415+0000","last_active":"2023-10-06T03:31:01.980497+0000","last_peered":"2023-10-06T03:31:01.980497+0000","last_clean":"2023-10-06T03:31:01.980497+0000","last_became_active":"2023-10-06T03:29:29.383894+0000","last_became_peered":"2023-10-06T03:29:29.383894+0000","last_unstale":"2023-10-06T03:31:01.980497+0000","last_undegraded":"2023-10-06T03:31:01.980497+0000","last_fullsized":"2023-10-06T03:31:01.980497+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:28.357506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:28.357506+0000","last_clean_scrub_stamp":"2023-10-06T03:29:28.357506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T13:09:00.657107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":30,"reported_epoch":43,"state":"active+clean","last_fresh":"2023-10-06T03:30:46.714052+0000","last_change":"2023-10-06T03:29:29.382183+0000","last_active":"2023-10-06T03:30:46.714052+0000","last_peered":"2023-10-06T03:30:46.714052+0000","last_clean":"2023-10-06T03:30:46.714052+0000","last_became_active":"2023-10-06T03:29:29.381918+0000","last_became_peered":"2023-10-06T03:29:29.381918+0000","last_unstale":"2023-10-06T03:30:46.714052+0000","last_undegraded":"2023-10-06T03:30:46.714052+0000","last_fullsized":"2023-10-06T03:30:46.714052+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:28.357506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:28.357506+0000","last_clean_scrub_stamp":"2023-10-06T03:29:28.357506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T11:56:26.013716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":30,"reported_epoch":43,"state":"active+clean","last_fresh":"2023-10-06T03:30:46.743697+0000","last_change":"2023-10-06T03:29:29.384279+0000","last_active":"2023-10-06T03:30:46.743697+0000","last_peered":"2023-10-06T03:30:46.743697+0000","last_clean":"2023-10-06T03:30:46.743697+0000","last_became_active":"2023-10-06T03:29:29.383533+0000","last_became_peered":"2023-10-06T03:29:29.383533+0000","last_unstale":"2023-10-06T03:30:46.743697+0000","last_undegraded":"2023-10-06T03:30:46.743697+0000","last_fullsized":"2023-10-06T03:30:46.743697+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:28.357506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:28.357506+0000","last_clean_scrub_stamp":"2023-10-06T03:29:28.357506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T08:34:19.641525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":30,"reported_epoch":43,"state":"active+clean","last_fresh":"2023-10-06T03:30:46.713940+0000","last_change":"2023-10-06T03:29:29.386238+0000","last_active":"2023-10-06T03:30:46.713940+0000","last_peered":"2023-10-06T03:30:46.713940+0000","last_clean":"2023-10-06T03:30:46.713940+0000","last_became_active":"2023-10-06T03:29:29.385865+0000","last_became_peered":"2023-10-06T03:29:29.385865+0000","last_unstale":"2023-10-06T03:30:46.713940+0000","last_undegraded":"2023-10-06T03:30:46.713940+0000","last_fullsized":"2023-10-06T03:30:46.713940+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:28.357506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:28.357506+0000","last_clean_scrub_stamp":"2023-10-06T03:29:28.357506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T12:43:34.714730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":30,"reported_epoch":44,"state":"active+clean","last_fresh":"2023-10-06T03:31:02.100499+0000","last_change":"2023-10-06T03:29:29.387402+0000","last_active":"2023-10-06T03:31:02.100499+0000","last_peered":"2023-10-06T03:31:02.100499+0000","last_clean":"2023-10-06T03:31:02.100499+0000","last_became_active":"2023-10-06T03:29:29.386812+0000","last_became_peered":"2023-10-06T03:29:29.386812+0000","last_unstale":"2023-10-06T03:31:02.100499+0000","last_undegraded":"2023-10-06T03:31:02.100499+0000","last_fullsized":"2023-10-06T03:31:02.100499+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:28.357506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:28.357506+0000","last_clean_scrub_stamp":"2023-10-06T03:29:28.357506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T12:09:43.895810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":308,"num_read_kb":512,"num_write":199,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":139,"ondisk_log_size":139,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509464,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298360,"kb_used_data":1072,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94073480,"statfs":{"total":96636764160,"available":96331243520,"internally_reserved":0,"allocated":1097728,"data_stored":817631,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Oct 6 03:30:26 2023","interfaces":[{"interface":"back","average":{"1min":0.878,"5min":0.878,"15min":0.878},"min":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"max":{"1min":1.228,"5min":1.228,"15min":1.228},"last":0.69899999999999995},{"interface":"front","average":{"1min":0.878,"5min":0.878,"15min":0.878},"min":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":1.181,"5min":1.181,"15min":1.181},"last":0.95399999999999996}]},{"osd":1,"last update":"Fri Oct 6 03:30:26 2023","interfaces":[{"interface":"back","average":{"1min":0.93000000000000005,"5min":0.93000000000000005,"15min":0.93000000000000005},"min":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"max":{"1min":1.2889999999999999,"5min":1.2889999999999999,"15min":1.2889999999999999},"last":0.58799999999999997},{"interface":"front","average":{"1min":0.94299999999999995,"5min":0.94299999999999995,"15min":0.94299999999999995},"min":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"max":{"1min":1.341,"5min":1.341,"15min":1.341},"last":0.61799999999999999}]},{"osd":2,"last update":"Fri Oct 6 03:30:26 2023","interfaces":[{"interface":"back","average":{"1min":0.95299999999999996,"5min":0.95299999999999996,"15min":0.95299999999999996},"min":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.72599999999999998},"max":{"1min":1.2050000000000001,"5min":1.2050000000000001,"15min":1.2050000000000001},"last":0.76800000000000002},{"interface":"front","average":{"1min":1.006,"5min":1.006,"15min":1.006},"min":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"max":{"1min":1.409,"5min":1.409,"15min":1.409},"last":0.73199999999999998}]},{"osd":3,"last update":"Fri Oct 6 03:30:26 2023","interfaces":[{"interface":"back","average":{"1min":0.86299999999999999,"5min":0.86299999999999999,"15min":0.86299999999999999},"min":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"max":{"1min":1.1339999999999999,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.85999999999999999},{"interface":"front","average":{"1min":0.96899999999999997,"5min":0.96899999999999997,"15min":0.96899999999999997},"min":{"1min":0.63400000000000001,"5min":0.63400000000000001,"15min":0.63400000000000001},"max":{"1min":1.256,"5min":1.256,"15min":1.256},"last":0.80800000000000005}]},{"osd":4,"last update":"Fri Oct 6 03:30:26 2023","interfaces":[{"interface":"back","average":{"1min":0.91900000000000004,"5min":0.91900000000000004,"15min":0.91900000000000004},"min":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"max":{"1min":1.2230000000000001,"5min":1.2230000000000001,"15min":1.2230000000000001},"last":0.83699999999999997},{"interface":"front","average":{"1min":1.0529999999999999,"5min":1.0529999999999999,"15min":1.0529999999999999},"min":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"max":{"1min":1.4750000000000001,"5min":1.4750000000000001,"15min":1.4750000000000001},"last":1.0529999999999999}]},{"osd":5,"last update":"Fri Oct 6 03:30:26 2023","interfaces":[{"interface":"back","average":{"1min":1.0229999999999999,"5min":1.0229999999999999,"15min":1.0229999999999999},"min":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":0.77900000000000003},"max":{"1min":1.431,"5min":1.431,"15min":1.431},"last":1.0309999999999999},{"interface":"front","average":{"1min":1.105,"5min":1.105,"15min":1.105},"min":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"max":{"1min":1.407,"5min":1.407,"15min":1.407},"last":0.91200000000000003}]},{"osd":6,"last update":"Fri Oct 6 03:30:26 2023","interfaces":[{"interface":"back","average":{"1min":1.1399999999999999,"5min":1.1399999999999999,"15min":1.1399999999999999},"min":{"1min":0.86799999999999999,"5min":0.86799999999999999,"15min":0.86799999999999999},"max":{"1min":1.526,"5min":1.526,"15min":1.526},"last":1.1259999999999999},{"interface":"front","average":{"1min":1.198,"5min":1.198,"15min":1.198},"min":{"1min":0.98399999999999999,"5min":0.98399999999999999,"15min":0.98399999999999999},"max":{"1min":1.6719999999999999,"5min":1.6719999999999999,"15min":1.6719999999999999},"last":1.1719999999999999}]}]},{"osd":6,"up_from":16,"seq":68719476761,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297776,"kb_used_data":488,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074064,"statfs":{"total":96636764160,"available":96331841536,"internally_reserved":0,"allocated":499712,"data_stored":220402,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Oct 6 03:30:26 2023","interfaces":[{"interface":"back","average":{"1min":0.876,"5min":0.876,"15min":0.876},"min":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"max":{"1min":1.149,"5min":1.149,"15min":1.149},"last":0.77000000000000002},{"interface":"front","average":{"1min":0.90400000000000003,"5min":0.90400000000000003,"15min":0.90400000000000003},"min":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"max":{"1min":1.254,"5min":1.254,"15min":1.254},"last":0.70599999999999996}]},{"osd":1,"last update":"Fri Oct 6 03:30:26 2023","interfaces":[{"interface":"back","average":{"1min":0.81299999999999994,"5min":0.81299999999999994,"15min":0.81299999999999994},"min":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"max":{"1min":1.4730000000000001,"5min":1.4730000000000001,"15min":1.4730000000000001},"last":1.1459999999999999},{"interface":"front","average":{"1min":0.97099999999999997,"5min":0.97099999999999997,"15min":0.97099999999999997},"min":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"max":{"1min":1.3440000000000001,"5min":1.3440000000000001,"15min":1.3440000000000001},"last":1.127}]},{"osd":2,"last update":"Fri Oct 6 03:30:26 2023","interfaces":[{"interface":"back","average":{"1min":0.94099999999999995,"5min":0.94099999999999995,"15min":0.94099999999999995},"min":{"1min":0.629,"5min":0.629,"15min":0.629},"max":{"1min":1.236,"5min":1.236,"15min":1.236},"last":0.90400000000000003},{"interface":"front","average":{"1min":0.95599999999999996,"5min":0.95599999999999996,"15min":0.95599999999999996},"min":{"1min":0.623,"5min":0.623,"15min":0.623},"max":{"1min":1.3660000000000001,"5min":1.3660000000000001,"15min":1.3660000000000001},"last":0.82799999999999996}]},{"osd":3,"last update":"Fri Oct 6 03:30:26 2023","interfaces":[{"interface":"back","average":{"1min":0.99299999999999999,"5min":0.99299999999999999,"15min":0.99299999999999999},"min":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":1.3660000000000001,"5min":1.3660000000000001,"15min":1.3660000000000001},"last":0.98099999999999998},{"interface":"front","average":{"1min":0.95599999999999996,"5min":0.95599999999999996,"15min":0.95599999999999996},"min":{"1min":0.76300000000000001,"5min":0.76300000000000001,"15min":0.76300000000000001},"max":{"1min":1.2609999999999999,"5min":1.2609999999999999,"15min":1.2609999999999999},"last":1.008}]},{"osd":4,"last update":"Fri Oct 6 03:30:26 2023","interfaces":[{"interface":"back","average":{"1min":1.1040000000000001,"5min":1.1040000000000001,"15min":1.1040000000000001},"min":{"1min":0.875,"5min":0.875,"15min":0.875},"max":{"1min":1.2869999999999999,"5min":1.2869999999999999,"15min":1.2869999999999999},"last":1.0700000000000001},{"interface":"front","average":{"1min":1.1140000000000001,"5min":1.1140000000000001,"15min":1.1140000000000001},"min":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.73399999999999999},"max":{"1min":1.4450000000000001,"5min":1.4450000000000001,"15min":1.4450000000000001},"last":1.194}]},{"osd":5,"last update":"Fri Oct 6 03:30:26 2023","interfaces":[{"interface":"back","average":{"1min":1.105,"5min":1.105,"15min":1.105},"min":{"1min":0.86799999999999999,"5min":0.86799999999999999,"15min":0.86799999999999999},"max":{"1min":1.613,"5min":1.613,"15min":1.613},"last":1.5620000000000001},{"interface":"front","average":{"1min":1.1699999999999999,"5min":1.1699999999999999,"15min":1.1699999999999999},"min":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"max":{"1min":1.351,"5min":1.351,"15min":1.351},"last":1.171}]},{"osd":7,"last update":"Fri Oct 6 03:30:26 2023","interfaces":[{"interface":"back","average":{"1min":1.2370000000000001,"5min":1.2370000000000001,"15min":1.2370000000000001},"min":{"1min":1.012,"5min":1.012,"15min":1.012},"max":{"1min":1.5409999999999999,"5min":1.5409999999999999,"15min":1.5409999999999999},"last":1.0980000000000001},{"interface":"front","average":{"1min":1.212,"5min":1.212,"15min":1.212},"min":{"1min":0.96099999999999997,"5min":0.96099999999999997,"15min":0.96099999999999997},"max":{"1min":1.704,"5min":1.704,"15min":1.704},"last":1.6539999999999999}]}]},{"osd":5,"up_from":16,"seq":68719476761,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298116,"kb_used_data":828,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94073724,"statfs":{"total":96636764160,"available":96331493376,"internally_reserved":0,"allocated":847872,"data_stored":562776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Oct 6 03:30:23 2023","interfaces":[{"interface":"back","average":{"1min":0.86399999999999999,"5min":0.86399999999999999,"15min":0.86399999999999999},"min":{"1min":0.505,"5min":0.505,"15min":0.505},"max":{"1min":1.4079999999999999,"5min":1.4079999999999999,"15min":1.4079999999999999},"last":1.141},{"interface":"front","average":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":0.89800000000000002},"min":{"1min":0.64500000000000002,"5min":0.64500000000000002,"15min":0.64500000000000002},"max":{"1min":1.3779999999999999,"5min":1.3779999999999999,"15min":1.3779999999999999},"last":0.93700000000000006}]},{"osd":1,"last update":"Fri Oct 6 03:30:23 2023","interfaces":[{"interface":"back","average":{"1min":0.97799999999999998,"5min":0.97799999999999998,"15min":0.97799999999999998},"min":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":1.2829999999999999,"5min":1.2829999999999999,"15min":1.2829999999999999},"last":0.66200000000000003},{"interface":"front","average":{"1min":0.85999999999999999,"5min":0.85999999999999999,"15min":0.85999999999999999},"min":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"max":{"1min":1.581,"5min":1.581,"15min":1.581},"last":0.72599999999999998}]},{"osd":2,"last update":"Fri Oct 6 03:30:23 2023","interfaces":[{"interface":"back","average":{"1min":1,"5min":1,"15min":1},"min":{"1min":0.67300000000000004,"5min":0.67300000000000004,"15min":0.67300000000000004},"max":{"1min":1.258,"5min":1.258,"15min":1.258},"last":1.1919999999999999},{"interface":"front","average":{"1min":0.90600000000000003,"5min":0.90600000000000003,"15min":0.90600000000000003},"min":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.67100000000000004},"max":{"1min":1.367,"5min":1.367,"15min":1.367},"last":0.84099999999999997}]},{"osd":3,"last update":"Fri Oct 6 03:30:23 2023","interfaces":[{"interface":"back","average":{"1min":1.034,"5min":1.034,"15min":1.034},"min":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.77200000000000002},"max":{"1min":1.4079999999999999,"5min":1.4079999999999999,"15min":1.4079999999999999},"last":1.0920000000000001},{"interface":"front","average":{"1min":1.034,"5min":1.034,"15min":1.034},"min":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"max":{"1min":1.244,"5min":1.244,"15min":1.244},"last":0.999}]},{"osd":4,"last update":"Fri Oct 6 03:30:23 2023","interfaces":[{"interface":"back","average":{"1min":1.0920000000000001,"5min":1.0920000000000001,"15min":1.0920000000000001},"min":{"1min":0.752,"5min":0.752,"15min":0.752},"max":{"1min":2.2050000000000001,"5min":2.2050000000000001,"15min":2.2050000000000001},"last":1.2310000000000001},{"interface":"front","average":{"1min":1.1870000000000001,"5min":1.1870000000000001,"15min":1.1870000000000001},"min":{"1min":0.90900000000000003,"5min":0.90900000000000003,"15min":0.90900000000000003},"max":{"1min":1.518,"5min":1.518,"15min":1.518},"last":1.2629999999999999}]},{"osd":6,"last update":"Fri Oct 6 03:30:23 2023","interfaces":[{"interface":"back","average":{"1min":1.1040000000000001,"5min":1.1040000000000001,"15min":1.1040000000000001},"min":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"max":{"1min":1.3169999999999999,"5min":1.3169999999999999,"15min":1.3169999999999999},"last":1.4259999999999999},{"interface":"front","average":{"1min":1.2370000000000001,"5min":1.2370000000000001,"15min":1.2370000000000001},"min":{"1min":0.94599999999999995,"5min":0.94599999999999995,"15min":0.94599999999999995},"max":{"1min":2.3809999999999998,"5min":2.3809999999999998,"15min":2.3809999999999998},"last":1.3859999999999999}]},{"osd":7,"last update":"Fri Oct 6 03:30:23 2023","interfaces":[{"interface":"back","average":{"1min":1.2370000000000001,"5min":1.2370000000000001,"15min":1.2370000000000001},"min":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.79500000000000004},"max":{"1min":1.637,"5min":1.637,"15min":1.637},"last":1.6060000000000001},{"interface":"front","average":{"1min":1.2929999999999999,"5min":1.2929999999999999,"15min":1.2929999999999999},"min":{"1min":1.0429999999999999,"5min":1.0429999999999999,"15min":1.0429999999999999},"max":{"1min":2.3330000000000002,"5min":2.3330000000000002,"15min":2.3330000000000002},"last":1.339}]}]},{"osd":4,"up_from":16,"seq":68719476760,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298116,"kb_used_data":828,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94073724,"statfs":{"total":96636764160,"available":96331493376,"internally_reserved":0,"allocated":847872,"data_stored":564896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Oct 6 03:30:25 2023","interfaces":[{"interface":"back","average":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"min":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"max":{"1min":1.129,"5min":1.129,"15min":1.129},"last":0.71999999999999997},{"interface":"front","average":{"1min":0.96299999999999997,"5min":0.96299999999999997,"15min":0.96299999999999997},"min":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"max":{"1min":1.629,"5min":1.629,"15min":1.629},"last":0.55800000000000005}]},{"osd":1,"last update":"Fri Oct 6 03:30:25 2023","interfaces":[{"interface":"back","average":{"1min":0.89300000000000002,"5min":0.89300000000000002,"15min":0.89300000000000002},"min":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"max":{"1min":1.23,"5min":1.23,"15min":1.23},"last":1.0680000000000001},{"interface":"front","average":{"1min":0.86799999999999999,"5min":0.86799999999999999,"15min":0.86799999999999999},"min":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"max":{"1min":1.3680000000000001,"5min":1.3680000000000001,"15min":1.3680000000000001},"last":0.59899999999999998}]},{"osd":2,"last update":"Fri Oct 6 03:30:25 2023","interfaces":[{"interface":"back","average":{"1min":0.98999999999999999,"5min":0.98999999999999999,"15min":0.98999999999999999},"min":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":1.5549999999999999,"5min":1.5549999999999999,"15min":1.5549999999999999},"last":0.91300000000000003},{"interface":"front","average":{"1min":0.92100000000000004,"5min":0.92100000000000004,"15min":0.92100000000000004},"min":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.60799999999999998},"max":{"1min":1.306,"5min":1.306,"15min":1.306},"last":0.63400000000000001}]},{"osd":3,"last update":"Fri Oct 6 03:30:25 2023","interfaces":[{"interface":"back","average":{"1min":1.1040000000000001,"5min":1.1040000000000001,"15min":1.1040000000000001},"min":{"1min":0.84499999999999997,"5min":0.84499999999999997,"15min":0.84499999999999997},"max":{"1min":1.5580000000000001,"5min":1.5580000000000001,"15min":1.5580000000000001},"last":1.0229999999999999},{"interface":"front","average":{"1min":1.04,"5min":1.04,"15min":1.04},"min":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.67500000000000004},"max":{"1min":1.6699999999999999,"5min":1.6699999999999999,"15min":1.6699999999999999},"last":0.98699999999999999}]},{"osd":5,"last update":"Fri Oct 6 03:30:25 2023","interfaces":[{"interface":"back","average":{"1min":1.044,"5min":1.044,"15min":1.044},"min":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.73399999999999999},"max":{"1min":1.5169999999999999,"5min":1.5169999999999999,"15min":1.5169999999999999},"last":0.94799999999999995},{"interface":"front","average":{"1min":1.266,"5min":1.266,"15min":1.266},"min":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":0.82799999999999996},"max":{"1min":2.2029999999999998,"5min":2.2029999999999998,"15min":2.2029999999999998},"last":0.79500000000000004}]},{"osd":6,"last update":"Fri Oct 6 03:30:25 2023","interfaces":[{"interface":"back","average":{"1min":1.181,"5min":1.181,"15min":1.181},"min":{"1min":0.79900000000000004,"5min":0.79900000000000004,"15min":0.79900000000000004},"max":{"1min":2.0209999999999999,"5min":2.0209999999999999,"15min":2.0209999999999999},"last":0.76600000000000001},{"interface":"front","average":{"1min":1.179,"5min":1.179,"15min":1.179},"min":{"1min":0.71399999999999997,"5min":0.71399999999999997,"15min":0.71399999999999997},"max":{"1min":2.1440000000000001,"5min":2.1440000000000001,"15min":2.1440000000000001},"last":0.85399999999999998}]},{"osd":7,"last update":"Fri Oct 6 03:30:25 2023","interfaces":[{"interface":"back","average":{"1min":1.337,"5min":1.337,"15min":1.337},"min":{"1min":0.94399999999999995,"5min":0.94399999999999995,"15min":0.94399999999999995},"max":{"1min":2.0939999999999999,"5min":2.0939999999999999,"15min":2.0939999999999999},"last":1.1080000000000001},{"interface":"front","average":{"1min":1.3089999999999999,"5min":1.3089999999999999,"15min":1.3089999999999999},"min":{"1min":0.95899999999999996,"5min":0.95899999999999996,"15min":0.95899999999999996},"max":{"1min":1.968,"5min":1.968,"15min":1.968},"last":1.147}]}]},{"osd":3,"up_from":14,"seq":60129542168,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302204,"kb_used_data":820,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94069636,"statfs":{"total":96636764160,"available":96327307264,"internally_reserved":0,"allocated":839680,"data_stored":555463,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Oct 6 03:30:26 2023","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.81899999999999995,"5min":0.81899999999999995,"15min":0.81899999999999995},"last":0.86899999999999999},{"interface":"front","average":{"1min":0.754,"5min":0.754,"15min":0.754},"min":{"1min":0.505,"5min":0.505,"15min":0.505},"max":{"1min":1.0920000000000001,"5min":1.0920000000000001,"15min":1.0920000000000001},"last":0.54800000000000004}]},{"osd":1,"last update":"Fri Oct 6 03:30:20 2023","interfaces":[{"interface":"back","average":{"1min":0.83199999999999996,"5min":0.83199999999999996,"15min":0.83199999999999996},"min":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":1.2210000000000001,"5min":1.2210000000000001,"15min":1.2210000000000001},"last":0.79600000000000004},{"interface":"front","average":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":1.321,"5min":1.321,"15min":1.321},"last":0.60999999999999999}]},{"osd":2,"last update":"Fri Oct 6 03:30:20 2023","interfaces":[{"interface":"back","average":{"1min":0.86799999999999999,"5min":0.86799999999999999,"15min":0.86799999999999999},"min":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.61199999999999999},"max":{"1min":1.228,"5min":1.228,"15min":1.228},"last":0.92900000000000005},{"interface":"front","average":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"min":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":1.3680000000000001,"5min":1.3680000000000001,"15min":1.3680000000000001},"last":0.47699999999999998}]},{"osd":4,"last update":"Fri Oct 6 03:30:26 2023","interfaces":[{"interface":"back","average":{"1min":1.0129999999999999,"5min":1.0129999999999999,"15min":1.0129999999999999},"min":{"1min":0.76800000000000002,"5min":0.76800000000000002,"15min":0.76800000000000002},"max":{"1min":1.6359999999999999,"5min":1.6359999999999999,"15min":1.6359999999999999},"last":0.98599999999999999},{"interface":"front","average":{"1min":0.96699999999999997,"5min":0.96699999999999997,"15min":0.96699999999999997},"min":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"max":{"1min":1.409,"5min":1.409,"15min":1.409},"last":0.91000000000000003}]},{"osd":5,"last update":"Fri Oct 6 03:30:26 2023","interfaces":[{"interface":"back","average":{"1min":0.98999999999999999,"5min":0.98999999999999999,"15min":0.98999999999999999},"min":{"1min":0.746,"5min":0.746,"15min":0.746},"max":{"1min":1.387,"5min":1.387,"15min":1.387},"last":0.96599999999999997},{"interface":"front","average":{"1min":1.1359999999999999,"5min":1.1359999999999999,"15min":1.1359999999999999},"min":{"1min":0.89000000000000001,"5min":0.89000000000000001,"15min":0.89000000000000001},"max":{"1min":1.615,"5min":1.615,"15min":1.615},"last":1.0269999999999999}]},{"osd":6,"last update":"Fri Oct 6 03:30:26 2023","interfaces":[{"interface":"back","average":{"1min":1.079,"5min":1.079,"15min":1.079},"min":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"max":{"1min":1.643,"5min":1.643,"15min":1.643},"last":0.66800000000000004},{"interface":"front","average":{"1min":1.105,"5min":1.105,"15min":1.105},"min":{"1min":0.74399999999999999,"5min":0.74399999999999999,"15min":0.74399999999999999},"max":{"1min":1.6819999999999999,"5min":1.6819999999999999,"15min":1.6819999999999999},"last":1.0509999999999999}]},{"osd":7,"last update":"Fri Oct 6 03:30:26 2023","interfaces":[{"interface":"back","average":{"1min":1.1850000000000001,"5min":1.1850000000000001,"15min":1.1850000000000001},"min":{"1min":0.92000000000000004,"5min":0.92000000000000004,"15min":0.92000000000000004},"max":{"1min":1.597,"5min":1.597,"15min":1.597},"last":0.70799999999999996},{"interface":"front","average":{"1min":1.111,"5min":1.111,"15min":1.111},"min":{"1min":0.86599999999999999,"5min":0.86599999999999999,"15min":0.86599999999999999},"max":{"1min":1.446,"5min":1.446,"15min":1.446},"last":0.83399999999999996}]}]},{"osd":2,"up_from":14,"seq":60129542167,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":301908,"kb_used_data":524,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94069932,"statfs":{"total":96636764160,"available":96327610368,"internally_reserved":0,"allocated":536576,"data_stored":245337,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Oct 6 03:30:21 2023","interfaces":[{"interface":"back","average":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":1.1240000000000001,"5min":1.1240000000000001,"15min":1.1240000000000001},"last":0.76700000000000002},{"interface":"front","average":{"1min":0.70399999999999996,"5min":0.70399999999999996,"15min":0.70399999999999996},"min":{"1min":0.495,"5min":0.495,"15min":0.495},"max":{"1min":1.1559999999999999,"5min":1.1559999999999999,"15min":1.1559999999999999},"last":0.495}]},{"osd":1,"last update":"Fri Oct 6 03:30:21 2023","interfaces":[{"interface":"back","average":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"min":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":1.1060000000000001,"5min":1.1060000000000001,"15min":1.1060000000000001},"last":1.0449999999999999},{"interface":"front","average":{"1min":0.73499999999999999,"5min":0.73499999999999999,"15min":0.73499999999999999},"min":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":1.2390000000000001,"5min":1.2390000000000001,"15min":1.2390000000000001},"last":0.52700000000000002}]},{"osd":3,"last update":"Fri Oct 6 03:30:21 2023","interfaces":[{"interface":"back","average":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":0.78500000000000003},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":1.165,"5min":1.165,"15min":1.165},"last":0.66300000000000003},{"interface":"front","average":{"1min":0.873,"5min":0.873,"15min":0.873},"min":{"1min":0.626,"5min":0.626,"15min":0.626},"max":{"1min":1.2949999999999999,"5min":1.2949999999999999,"15min":1.2949999999999999},"last":0.45300000000000001}]},{"osd":4,"last update":"Fri Oct 6 03:30:23 2023","interfaces":[{"interface":"back","average":{"1min":1.095,"5min":1.095,"15min":1.095},"min":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"max":{"1min":2.3450000000000002,"5min":2.3450000000000002,"15min":2.3450000000000002},"last":0.86599999999999999},{"interface":"front","average":{"1min":0.95599999999999996,"5min":0.95599999999999996,"15min":0.95599999999999996},"min":{"1min":0.74399999999999999,"5min":0.74399999999999999,"15min":0.74399999999999999},"max":{"1min":1.1950000000000001,"5min":1.1950000000000001,"15min":1.1950000000000001},"last":0.83399999999999996}]},{"osd":5,"last update":"Fri Oct 6 03:30:23 2023","interfaces":[{"interface":"back","average":{"1min":1.139,"5min":1.139,"15min":1.139},"min":{"1min":0.79900000000000004,"5min":0.79900000000000004,"15min":0.79900000000000004},"max":{"1min":2.5169999999999999,"5min":2.5169999999999999,"15min":2.5169999999999999},"last":0.93600000000000005},{"interface":"front","average":{"1min":1.107,"5min":1.107,"15min":1.107},"min":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":0.81799999999999995},"max":{"1min":1.7250000000000001,"5min":1.7250000000000001,"15min":1.7250000000000001},"last":0.81299999999999994}]},{"osd":6,"last update":"Fri Oct 6 03:30:23 2023","interfaces":[{"interface":"back","average":{"1min":1.014,"5min":1.014,"15min":1.014},"min":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"max":{"1min":1.218,"5min":1.218,"15min":1.218},"last":0.55200000000000005},{"interface":"front","average":{"1min":1.1899999999999999,"5min":1.1899999999999999,"15min":1.1899999999999999},"min":{"1min":0.85899999999999999,"5min":0.85899999999999999,"15min":0.85899999999999999},"max":{"1min":2.2450000000000001,"5min":2.2450000000000001,"15min":2.2450000000000001},"last":1.0029999999999999}]},{"osd":7,"last update":"Fri Oct 6 03:30:21 2023","interfaces":[{"interface":"back","average":{"1min":1.1639999999999999,"5min":1.1639999999999999,"15min":1.1639999999999999},"min":{"1min":0.81599999999999995,"5min":0.81599999999999995,"15min":0.81599999999999995},"max":{"1min":1.8140000000000001,"5min":1.8140000000000001,"15min":1.8140000000000001},"last":0.70299999999999996},{"interface":"front","average":{"1min":1.0860000000000001,"5min":1.0860000000000001,"15min":1.0860000000000001},"min":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.79500000000000004},"max":{"1min":1.2749999999999999,"5min":1.2749999999999999,"15min":1.2749999999999999},"last":0.71599999999999997}]}]},{"osd":1,"up_from":14,"seq":60129542167,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302204,"kb_used_data":820,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94069636,"statfs":{"total":96636764160,"available":96327307264,"internally_reserved":0,"allocated":839680,"data_stored":555576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Oct 6 03:30:26 2023","interfaces":[{"interface":"back","average":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.60799999999999998},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.48599999999999999},{"interface":"front","average":{"1min":0.71599999999999997,"5min":0.71599999999999997,"15min":0.71599999999999997},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":1.5329999999999999,"5min":1.5329999999999999,"15min":1.5329999999999999},"last":0.59199999999999997}]},{"osd":2,"last update":"Fri Oct 6 03:30:20 2023","interfaces":[{"interface":"back","average":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.93899999999999995,"5min":0.93899999999999995,"15min":0.93899999999999995},"last":1.018},{"interface":"front","average":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"min":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"max":{"1min":1.278,"5min":1.278,"15min":1.278},"last":0.66600000000000004}]},{"osd":3,"last update":"Fri Oct 6 03:30:20 2023","interfaces":[{"interface":"back","average":{"1min":0.72899999999999998,"5min":0.72899999999999998,"15min":0.72899999999999998},"min":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"max":{"1min":0.90500000000000003,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.53700000000000003},{"interface":"front","average":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.3700000000000001,"5min":1.3700000000000001,"15min":1.3700000000000001},"last":0.80600000000000005}]},{"osd":4,"last update":"Fri Oct 6 03:30:26 2023","interfaces":[{"interface":"back","average":{"1min":0.94299999999999995,"5min":0.94299999999999995,"15min":0.94299999999999995},"min":{"1min":0.66400000000000003,"5min":0.66400000000000003,"15min":0.66400000000000003},"max":{"1min":1.7549999999999999,"5min":1.7549999999999999,"15min":1.7549999999999999},"last":0.96799999999999997},{"interface":"front","average":{"1min":0.97499999999999998,"5min":0.97499999999999998,"15min":0.97499999999999998},"min":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":1.415,"5min":1.415,"15min":1.415},"last":1.2250000000000001}]},{"osd":5,"last update":"Fri Oct 6 03:30:26 2023","interfaces":[{"interface":"back","average":{"1min":0.93999999999999995,"5min":0.93999999999999995,"15min":0.93999999999999995},"min":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"max":{"1min":1.4910000000000001,"5min":1.4910000000000001,"15min":1.4910000000000001},"last":1.0980000000000001},{"interface":"front","average":{"1min":1.0049999999999999,"5min":1.0049999999999999,"15min":1.0049999999999999},"min":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":0.70499999999999996},"max":{"1min":1.4370000000000001,"5min":1.4370000000000001,"15min":1.4370000000000001},"last":1.01}]},{"osd":6,"last update":"Fri Oct 6 03:30:26 2023","interfaces":[{"interface":"back","average":{"1min":1.022,"5min":1.022,"15min":1.022},"min":{"1min":0.69199999999999995,"5min":0.69199999999999995,"15min":0.69199999999999995},"max":{"1min":1.603,"5min":1.603,"15min":1.603},"last":1.3},{"interface":"front","average":{"1min":1.1080000000000001,"5min":1.1080000000000001,"15min":1.1080000000000001},"min":{"1min":0.69199999999999995,"5min":0.69199999999999995,"15min":0.69199999999999995},"max":{"1min":1.5069999999999999,"5min":1.5069999999999999,"15min":1.5069999999999999},"last":1.365}]},{"osd":7,"last update":"Fri Oct 6 03:30:26 2023","interfaces":[{"interface":"back","average":{"1min":1.04,"5min":1.04,"15min":1.04},"min":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.60799999999999998},"max":{"1min":1.6160000000000001,"5min":1.6160000000000001,"15min":1.6160000000000001},"last":0.98299999999999998},{"interface":"front","average":{"1min":1.071,"5min":1.071,"15min":1.071},"min":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.67500000000000004},"max":{"1min":1.5700000000000001,"5min":1.5700000000000001,"15min":1.5700000000000001},"last":1.1299999999999999}]}]},{"osd":0,"up_from":15,"seq":64424509465,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302468,"kb_used_data":1084,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94069372,"statfs":{"total":96636764160,"available":96327036928,"internally_reserved":0,"allocated":1110016,"data_stored":824876,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Oct 6 03:30:22 2023","interfaces":[{"interface":"back","average":{"1min":0.629,"5min":0.629,"15min":0.629},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":1.0029999999999999,"5min":1.0029999999999999,"15min":1.0029999999999999},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.74199999999999999},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":1.179,"5min":1.179,"15min":1.179},"last":0.63300000000000001}]},{"osd":2,"last update":"Fri Oct 6 03:30:22 2023","interfaces":[{"interface":"back","average":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":1.1879999999999999,"5min":1.1879999999999999,"15min":1.1879999999999999},"last":0.439},{"interface":"front","average":{"1min":0.72499999999999998,"5min":0.72499999999999998,"15min":0.72499999999999998},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":1.3879999999999999,"5min":1.3879999999999999,"15min":1.3879999999999999},"last":0.52300000000000002}]},{"osd":3,"last update":"Fri Oct 6 03:30:22 2023","interfaces":[{"interface":"back","average":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":1.1930000000000001,"5min":1.1930000000000001,"15min":1.1930000000000001},"last":0.67500000000000004},{"interface":"front","average":{"1min":0.86399999999999999,"5min":0.86399999999999999,"15min":0.86399999999999999},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":1.1970000000000001,"5min":1.1970000000000001,"15min":1.1970000000000001},"last":0.71399999999999997}]},{"osd":4,"last update":"Fri Oct 6 03:30:22 2023","interfaces":[{"interface":"back","average":{"1min":1.0189999999999999,"5min":1.0189999999999999,"15min":1.0189999999999999},"min":{"1min":0.72899999999999998,"5min":0.72899999999999998,"15min":0.72899999999999998},"max":{"1min":1.55,"5min":1.55,"15min":1.55},"last":0.98499999999999999},{"interface":"front","average":{"1min":0.99099999999999999,"5min":0.99099999999999999,"15min":0.99099999999999999},"min":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"max":{"1min":1.5269999999999999,"5min":1.5269999999999999,"15min":1.5269999999999999},"last":1.2969999999999999}]},{"osd":5,"last update":"Fri Oct 6 03:30:22 2023","interfaces":[{"interface":"back","average":{"1min":0.98499999999999999,"5min":0.98499999999999999,"15min":0.98499999999999999},"min":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"max":{"1min":1.5409999999999999,"5min":1.5409999999999999,"15min":1.5409999999999999},"last":0.91100000000000003},{"interface":"front","average":{"1min":1.087,"5min":1.087,"15min":1.087},"min":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"max":{"1min":1.353,"5min":1.353,"15min":1.353},"last":1.24}]},{"osd":6,"last update":"Fri Oct 6 03:30:22 2023","interfaces":[{"interface":"back","average":{"1min":1.145,"5min":1.145,"15min":1.145},"min":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"max":{"1min":1.502,"5min":1.502,"15min":1.502},"last":1.1859999999999999},{"interface":"front","average":{"1min":1.1299999999999999,"5min":1.1299999999999999,"15min":1.1299999999999999},"min":{"1min":0.64900000000000002,"5min":0.64900000000000002,"15min":0.64900000000000002},"max":{"1min":1.6439999999999999,"5min":1.6439999999999999,"15min":1.6439999999999999},"last":1.085}]},{"osd":7,"last update":"Fri Oct 6 03:30:22 2023","interfaces":[{"interface":"back","average":{"1min":1.099,"5min":1.099,"15min":1.099},"min":{"1min":0.874,"5min":0.874,"15min":0.874},"max":{"1min":1.369,"5min":1.369,"15min":1.369},"last":1.046},{"interface":"front","average":{"1min":1.2030000000000001,"5min":1.2030000000000001,"15min":1.2030000000000001},"min":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"max":{"1min":1.573,"5min":1.573,"15min":1.573},"last":1.0209999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-06T03:31:08.784 INFO:tasks.ceph.ceph_manager.ceph:clean! 2023-10-06T03:31:08.784 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-06T03:31:08.891 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:08.884+0000 7fc308629700 1 Processor -- start 2023-10-06T03:31:08.891 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:08.884+0000 7fc308629700 1 -- start start 2023-10-06T03:31:08.893 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:08.888+0000 7fc308629700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fc30006b4f0 0x7fc3000c8aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:31:08.893 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:08.888+0000 7fc308629700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fc30015d0a0 0x7fc30015d520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:31:08.893 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:08.888+0000 7fc308629700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3000ac320 con 0x7fc3000c86d0 2023-10-06T03:31:08.893 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:08.888+0000 7fc308629700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc3000c7000 con 0x7fc30006b4f0 2023-10-06T03:31:08.893 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:08.888+0000 7fc308629700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc3000abd10 con 0x7fc30015d0a0 2023-10-06T03:31:08.893 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:08.888+0000 7fc305bc4700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fc30006b4f0 0x7fc3000c8aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:31:08.893 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:08.888+0000 7fc305bc4700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fc30006b4f0 0x7fc3000c8aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:38916/0 (socket says 172.21.15.110:38916) 2023-10-06T03:31:08.893 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:08.888+0000 7fc305bc4700 1 -- 172.21.15.110:0/1972736373 learned_addr learned my addr 172.21.15.110:0/1972736373 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:31:08.893 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:08.888+0000 7fc306bc6700 1 --2- 172.21.15.110:0/1972736373 >> v2:172.21.15.192:3300/0 conn(0x7fc30015d0a0 0x7fc30015d520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:31:08.893 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:08.888+0000 7fc305bc4700 1 -- 172.21.15.110:0/1972736373 >> v2:172.21.15.192:3300/0 conn(0x7fc30015d0a0 msgr2=0x7fc30015d520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:31:08.894 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:08.888+0000 7fc305bc4700 1 --2- 172.21.15.110:0/1972736373 >> v2:172.21.15.192:3300/0 conn(0x7fc30015d0a0 0x7fc30015d520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:08.894 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:08.888+0000 7fc305bc4700 1 -- 172.21.15.110:0/1972736373 >> v1:172.21.15.110:6789/0 conn(0x7fc3000c86d0 legacy=0x7fc3000c68f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:31:08.894 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:08.888+0000 7fc3053c3700 1 -- 172.21.15.110:0/1972736373 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 512989480 0 0) 0x7fc3000ac320 con 0x7fc3000c86d0 2023-10-06T03:31:08.894 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:08.888+0000 7fc305bc4700 1 -- 172.21.15.110:0/1972736373 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3000c71f0 con 0x7fc30006b4f0 2023-10-06T03:31:08.894 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:08.888+0000 7fc306bc6700 1 --2- 172.21.15.110:0/1972736373 >> v2:172.21.15.192:3300/0 conn(0x7fc30015d0a0 0x7fc30015d520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-06T03:31:08.895 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:08.888+0000 7fc305bc4700 1 --2- 172.21.15.110:0/1972736373 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc30006b4f0 0x7fc3000c8aa0 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7fc30006a470 tx=0x7fc2f0009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=16c47b491fe2571b server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:31:08.895 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:08.888+0000 7fc3053c3700 1 -- 172.21.15.110:0/1972736373 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2f000e070 con 0x7fc30006b4f0 2023-10-06T03:31:08.895 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:08.888+0000 7fc3053c3700 1 -- 172.21.15.110:0/1972736373 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2f000ba50 con 0x7fc30006b4f0 2023-10-06T03:31:08.895 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:08.888+0000 7fc3053c3700 1 -- 172.21.15.110:0/1972736373 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2f00092e0 con 0x7fc30006b4f0 2023-10-06T03:31:08.895 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:08.888+0000 7fc308629700 1 -- 172.21.15.110:0/1972736373 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc30006b4f0 msgr2=0x7fc3000c8aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:31:08.896 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:08.888+0000 7fc308629700 1 --2- 172.21.15.110:0/1972736373 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc30006b4f0 0x7fc3000c8aa0 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7fc30006a470 tx=0x7fc2f0009840 comp rx=0 tx=0).stop 2023-10-06T03:31:08.896 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:08.888+0000 7fc308629700 1 -- 172.21.15.110:0/1972736373 shutdown_connections 2023-10-06T03:31:08.896 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:08.888+0000 7fc308629700 1 --2- 172.21.15.110:0/1972736373 >> v2:172.21.15.192:3300/0 conn(0x7fc30015d0a0 0x7fc30015d520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:08.896 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:08.888+0000 7fc308629700 1 --2- 172.21.15.110:0/1972736373 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc30006b4f0 0x7fc3000c8aa0 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:08.896 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:08.888+0000 7fc308629700 1 -- 172.21.15.110:0/1972736373 >> 172.21.15.110:0/1972736373 conn(0x7fc300067040 msgr2=0x7fc300067410 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:31:08.896 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:08.888+0000 7fc308629700 1 -- 172.21.15.110:0/1972736373 shutdown_connections 2023-10-06T03:31:08.896 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:08.888+0000 7fc308629700 1 -- 172.21.15.110:0/1972736373 wait complete. 2023-10-06T03:31:08.896 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:08.892+0000 7fc308629700 1 Processor -- start 2023-10-06T03:31:08.897 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:08.892+0000 7fc308629700 1 -- start start 2023-10-06T03:31:08.897 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:08.892+0000 7fc308629700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc30006b4f0 0x7fc30015b240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:31:08.898 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:08.892+0000 7fc308629700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc3000c86d0 0x7fc3000c1720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:31:08.898 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:08.892+0000 7fc308629700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc30015d0a0 0x7fc3000c4830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:31:08.898 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:08.892+0000 7fc308629700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc30015b780 con 0x7fc3000c86d0 2023-10-06T03:31:08.898 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:08.892+0000 7fc308629700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3000c3790 con 0x7fc30006b4f0 2023-10-06T03:31:08.898 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:08.892+0000 7fc308629700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3000c3a70 con 0x7fc30015d0a0 2023-10-06T03:31:08.898 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:08.892+0000 7fc305bc4700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc3000c86d0 0x7fc3000c1720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:31:08.898 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:08.892+0000 7fc305bc4700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc3000c86d0 0x7fc3000c1720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:38924/0 (socket says 172.21.15.110:38924) 2023-10-06T03:31:08.898 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:08.892+0000 7fc305bc4700 1 -- 172.21.15.110:0/2990255907 learned_addr learned my addr 172.21.15.110:0/2990255907 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:31:08.898 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:08.892+0000 7fc3063c5700 1 --2- 172.21.15.110:0/2990255907 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc30006b4f0 0x7fc30015b240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:31:08.898 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:08.892+0000 7fc306bc6700 1 --2- 172.21.15.110:0/2990255907 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc30015d0a0 0x7fc3000c4830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:31:08.900 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:08.892+0000 7fc3063c5700 1 -- 172.21.15.110:0/2990255907 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc30015d0a0 msgr2=0x7fc3000c4830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:31:08.900 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:08.892+0000 7fc3063c5700 1 --2- 172.21.15.110:0/2990255907 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc30015d0a0 0x7fc3000c4830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:08.900 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:08.892+0000 7fc3063c5700 1 -- 172.21.15.110:0/2990255907 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc3000c86d0 msgr2=0x7fc3000c1720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:31:08.900 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:08.892+0000 7fc3063c5700 1 --2- 172.21.15.110:0/2990255907 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc3000c86d0 0x7fc3000c1720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:08.900 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:08.892+0000 7fc3063c5700 1 -- 172.21.15.110:0/2990255907 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2f0009ae0 con 0x7fc30006b4f0 2023-10-06T03:31:08.900 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:08.892+0000 7fc305bc4700 1 --2- 172.21.15.110:0/2990255907 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc3000c86d0 0x7fc3000c1720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-06T03:31:08.900 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:08.892+0000 7fc3063c5700 1 --2- 172.21.15.110:0/2990255907 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc30006b4f0 0x7fc30015b240 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fc2f40011e0 tx=0x7fc2f4007d30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:31:08.900 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:08.892+0000 7fc2fe7fc700 1 -- 172.21.15.110:0/2990255907 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2f4011590 con 0x7fc30006b4f0 2023-10-06T03:31:08.900 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:08.892+0000 7fc2fe7fc700 1 -- 172.21.15.110:0/2990255907 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2f4011710 con 0x7fc30006b4f0 2023-10-06T03:31:08.900 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:08.892+0000 7fc308629700 1 -- 172.21.15.110:0/2990255907 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3000c3d50 con 0x7fc30006b4f0 2023-10-06T03:31:08.901 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:08.892+0000 7fc2fe7fc700 1 -- 172.21.15.110:0/2990255907 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2f4011ad0 con 0x7fc30006b4f0 2023-10-06T03:31:08.901 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:08.892+0000 7fc308629700 1 -- 172.21.15.110:0/2990255907 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3000c4310 con 0x7fc30006b4f0 2023-10-06T03:31:08.901 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:08.896+0000 7fc308629700 1 -- 172.21.15.110:0/2990255907 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc3000c4530 con 0x7fc30006b4f0 2023-10-06T03:31:08.901 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:08.896+0000 7fc2fe7fc700 1 -- 172.21.15.110:0/2990255907 <== mon.1 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 13) v1 ==== 80061+0+0 (secure 0 0 0) 0x7fc2f401b3b0 con 0x7fc30006b4f0 2023-10-06T03:31:08.902 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:08.896+0000 7fc2fe7fc700 1 --2- 172.21.15.110:0/2990255907 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fc2ec060670 0x7fc2ec062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:31:08.902 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:08.896+0000 7fc2fe7fc700 1 -- 172.21.15.110:0/2990255907 <== mon.1 v2:172.21.15.192:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 6171+0+0 (secure 0 0 0) 0x7fc2f407bfa0 con 0x7fc30006b4f0 2023-10-06T03:31:08.902 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:08.896+0000 7fc305bc4700 1 --2- 172.21.15.110:0/2990255907 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fc2ec060670 0x7fc2ec062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:31:08.902 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:08.896+0000 7fc305bc4700 1 --2- 172.21.15.110:0/2990255907 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fc2ec060670 0x7fc2ec062b20 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fc3000c3340 tx=0x7fc2f0018000 comp rx=0 tx=0).ready entity=mgr.5505 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:31:08.912 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:08.908+0000 7fc2fe7fc700 1 -- 172.21.15.110:0/2990255907 <== mon.1 v2:172.21.15.192:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fc2f40abbe0 con 0x7fc30006b4f0 2023-10-06T03:31:09.070 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:09.064+0000 7fc308629700 1 -- 172.21.15.110:0/2990255907 --> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fc300064770 con 0x7fc2ec060670 2023-10-06T03:31:09.072 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:09.068+0000 7fc2fe7fc700 1 -- 172.21.15.110:0/2990255907 <== mgr.5505 v2:172.21.15.110:6832/33268 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+71219 (secure 0 0 0) 0x7fc300064770 con 0x7fc2ec060670 2023-10-06T03:31:09.073 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-10-06T03:31:09.075 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:09.068+0000 7fc308629700 1 -- 172.21.15.110:0/2990255907 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fc2ec060670 msgr2=0x7fc2ec062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:31:09.075 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:09.068+0000 7fc308629700 1 --2- 172.21.15.110:0/2990255907 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fc2ec060670 0x7fc2ec062b20 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fc3000c3340 tx=0x7fc2f0018000 comp rx=0 tx=0).stop 2023-10-06T03:31:09.075 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:09.068+0000 7fc308629700 1 -- 172.21.15.110:0/2990255907 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc30006b4f0 msgr2=0x7fc30015b240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:31:09.075 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:09.068+0000 7fc308629700 1 --2- 172.21.15.110:0/2990255907 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc30006b4f0 0x7fc30015b240 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fc2f40011e0 tx=0x7fc2f4007d30 comp rx=0 tx=0).stop 2023-10-06T03:31:09.075 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:09.068+0000 7fc308629700 1 -- 172.21.15.110:0/2990255907 shutdown_connections 2023-10-06T03:31:09.075 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:09.068+0000 7fc308629700 1 --2- 172.21.15.110:0/2990255907 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fc2ec060670 0x7fc2ec062b20 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:09.075 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:09.068+0000 7fc308629700 1 --2- 172.21.15.110:0/2990255907 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc30015d0a0 0x7fc3000c4830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:09.075 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:09.068+0000 7fc308629700 1 --2- 172.21.15.110:0/2990255907 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc3000c86d0 0x7fc3000c1720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:09.075 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:09.068+0000 7fc308629700 1 --2- 172.21.15.110:0/2990255907 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc30006b4f0 0x7fc30015b240 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:09.076 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:09.068+0000 7fc308629700 1 -- 172.21.15.110:0/2990255907 >> 172.21.15.110:0/2990255907 conn(0x7fc300067040 msgr2=0x7fc3000ac7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:31:09.076 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:09.068+0000 7fc308629700 1 -- 172.21.15.110:0/2990255907 shutdown_connections 2023-10-06T03:31:09.076 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:09.068+0000 7fc308629700 1 -- 172.21.15.110:0/2990255907 wait complete. 2023-10-06T03:31:09.076 INFO:teuthology.orchestra.run.smithi110.stderr:dumped all 2023-10-06T03:31:09.092 INFO:teuthology.orchestra.run.smithi110.stdout:{"pg_ready":true,"pg_map":{"version":9,"stamp":"2023-10-06T03:31:07.946720+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":308,"num_read_kb":512,"num_write":199,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":139,"ondisk_log_size":139,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":210,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2401152,"kb_used_data":6464,"kb_used_omap":0,"kb_used_meta":2394624,"kb_avail":752573568,"statfs":{"total":773094113280,"available":770635333632,"internally_reserved":0,"allocated":6619136,"data_stored":4346957,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2452094976},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-667624,"num_objects":-25,"num_object_clones":0,"num_object_copies":-50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-25,"num_whiteouts":0,"num_read":-317,"num_read_kb":-281,"num_write":-285,"num_write_kb":-2055,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.991630"},"pg_stats":[{"pgid":"1.0","version":"44'139","reported_seq":261,"reported_epoch":44,"state":"active+clean","last_fresh":"2023-10-06T03:31:02.032960+0000","last_change":"2023-10-06T03:29:21.978311+0000","last_active":"2023-10-06T03:31:02.032960+0000","last_peered":"2023-10-06T03:31:02.032960+0000","last_clean":"2023-10-06T03:31:02.032960+0000","last_became_active":"2023-10-06T03:29:21.978017+0000","last_became_peered":"2023-10-06T03:29:21.978017+0000","last_unstale":"2023-10-06T03:31:02.032960+0000","last_undegraded":"2023-10-06T03:31:02.032960+0000","last_fullsized":"2023-10-06T03:31:02.032960+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:20.957955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:20.957955+0000","last_clean_scrub_stamp":"2023-10-06T03:29:20.957955+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T07:33:10.429084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":308,"num_read_kb":512,"num_write":199,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":31,"reported_epoch":44,"state":"active+clean","last_fresh":"2023-10-06T03:31:01.980717+0000","last_change":"2023-10-06T03:29:29.386120+0000","last_active":"2023-10-06T03:31:01.980717+0000","last_peered":"2023-10-06T03:31:01.980717+0000","last_clean":"2023-10-06T03:31:01.980717+0000","last_became_active":"2023-10-06T03:29:29.385692+0000","last_became_peered":"2023-10-06T03:29:29.385692+0000","last_unstale":"2023-10-06T03:31:01.980717+0000","last_undegraded":"2023-10-06T03:31:01.980717+0000","last_fullsized":"2023-10-06T03:31:01.980717+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:28.357506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:28.357506+0000","last_clean_scrub_stamp":"2023-10-06T03:29:28.357506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T10:21:35.864895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":30,"reported_epoch":43,"state":"active+clean","last_fresh":"2023-10-06T03:30:46.743121+0000","last_change":"2023-10-06T03:29:29.384353+0000","last_active":"2023-10-06T03:30:46.743121+0000","last_peered":"2023-10-06T03:30:46.743121+0000","last_clean":"2023-10-06T03:30:46.743121+0000","last_became_active":"2023-10-06T03:29:29.383884+0000","last_became_peered":"2023-10-06T03:29:29.383884+0000","last_unstale":"2023-10-06T03:30:46.743121+0000","last_undegraded":"2023-10-06T03:30:46.743121+0000","last_fullsized":"2023-10-06T03:30:46.743121+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:28.357506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:28.357506+0000","last_clean_scrub_stamp":"2023-10-06T03:29:28.357506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T14:43:09.107140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":32,"reported_epoch":44,"state":"active+clean","last_fresh":"2023-10-06T03:31:01.933050+0000","last_change":"2023-10-06T03:29:29.606195+0000","last_active":"2023-10-06T03:31:01.933050+0000","last_peered":"2023-10-06T03:31:01.933050+0000","last_clean":"2023-10-06T03:31:01.933050+0000","last_became_active":"2023-10-06T03:29:29.605489+0000","last_became_peered":"2023-10-06T03:29:29.605489+0000","last_unstale":"2023-10-06T03:31:01.933050+0000","last_undegraded":"2023-10-06T03:31:01.933050+0000","last_fullsized":"2023-10-06T03:31:01.933050+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:28.357506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:28.357506+0000","last_clean_scrub_stamp":"2023-10-06T03:29:28.357506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T10:21:35.864895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":31,"reported_epoch":44,"state":"active+clean","last_fresh":"2023-10-06T03:31:01.980497+0000","last_change":"2023-10-06T03:29:29.384415+0000","last_active":"2023-10-06T03:31:01.980497+0000","last_peered":"2023-10-06T03:31:01.980497+0000","last_clean":"2023-10-06T03:31:01.980497+0000","last_became_active":"2023-10-06T03:29:29.383894+0000","last_became_peered":"2023-10-06T03:29:29.383894+0000","last_unstale":"2023-10-06T03:31:01.980497+0000","last_undegraded":"2023-10-06T03:31:01.980497+0000","last_fullsized":"2023-10-06T03:31:01.980497+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:28.357506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:28.357506+0000","last_clean_scrub_stamp":"2023-10-06T03:29:28.357506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T13:09:00.657107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":30,"reported_epoch":43,"state":"active+clean","last_fresh":"2023-10-06T03:30:46.714052+0000","last_change":"2023-10-06T03:29:29.382183+0000","last_active":"2023-10-06T03:30:46.714052+0000","last_peered":"2023-10-06T03:30:46.714052+0000","last_clean":"2023-10-06T03:30:46.714052+0000","last_became_active":"2023-10-06T03:29:29.381918+0000","last_became_peered":"2023-10-06T03:29:29.381918+0000","last_unstale":"2023-10-06T03:30:46.714052+0000","last_undegraded":"2023-10-06T03:30:46.714052+0000","last_fullsized":"2023-10-06T03:30:46.714052+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:28.357506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:28.357506+0000","last_clean_scrub_stamp":"2023-10-06T03:29:28.357506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T11:56:26.013716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":30,"reported_epoch":43,"state":"active+clean","last_fresh":"2023-10-06T03:30:46.743697+0000","last_change":"2023-10-06T03:29:29.384279+0000","last_active":"2023-10-06T03:30:46.743697+0000","last_peered":"2023-10-06T03:30:46.743697+0000","last_clean":"2023-10-06T03:30:46.743697+0000","last_became_active":"2023-10-06T03:29:29.383533+0000","last_became_peered":"2023-10-06T03:29:29.383533+0000","last_unstale":"2023-10-06T03:30:46.743697+0000","last_undegraded":"2023-10-06T03:30:46.743697+0000","last_fullsized":"2023-10-06T03:30:46.743697+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:28.357506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:28.357506+0000","last_clean_scrub_stamp":"2023-10-06T03:29:28.357506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T08:34:19.641525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":30,"reported_epoch":43,"state":"active+clean","last_fresh":"2023-10-06T03:30:46.713940+0000","last_change":"2023-10-06T03:29:29.386238+0000","last_active":"2023-10-06T03:30:46.713940+0000","last_peered":"2023-10-06T03:30:46.713940+0000","last_clean":"2023-10-06T03:30:46.713940+0000","last_became_active":"2023-10-06T03:29:29.385865+0000","last_became_peered":"2023-10-06T03:29:29.385865+0000","last_unstale":"2023-10-06T03:30:46.713940+0000","last_undegraded":"2023-10-06T03:30:46.713940+0000","last_fullsized":"2023-10-06T03:30:46.713940+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:28.357506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:28.357506+0000","last_clean_scrub_stamp":"2023-10-06T03:29:28.357506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T12:43:34.714730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":30,"reported_epoch":44,"state":"active+clean","last_fresh":"2023-10-06T03:31:02.100499+0000","last_change":"2023-10-06T03:29:29.387402+0000","last_active":"2023-10-06T03:31:02.100499+0000","last_peered":"2023-10-06T03:31:02.100499+0000","last_clean":"2023-10-06T03:31:02.100499+0000","last_became_active":"2023-10-06T03:29:29.386812+0000","last_became_peered":"2023-10-06T03:29:29.386812+0000","last_unstale":"2023-10-06T03:31:02.100499+0000","last_undegraded":"2023-10-06T03:31:02.100499+0000","last_fullsized":"2023-10-06T03:31:02.100499+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:28.357506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:28.357506+0000","last_clean_scrub_stamp":"2023-10-06T03:29:28.357506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T12:09:43.895810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":308,"num_read_kb":512,"num_write":199,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":139,"ondisk_log_size":139,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509464,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298360,"kb_used_data":1072,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94073480,"statfs":{"total":96636764160,"available":96331243520,"internally_reserved":0,"allocated":1097728,"data_stored":817631,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Oct 6 03:30:26 2023","interfaces":[{"interface":"back","average":{"1min":0.878,"5min":0.878,"15min":0.878},"min":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"max":{"1min":1.228,"5min":1.228,"15min":1.228},"last":0.69899999999999995},{"interface":"front","average":{"1min":0.878,"5min":0.878,"15min":0.878},"min":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":1.181,"5min":1.181,"15min":1.181},"last":0.95399999999999996}]},{"osd":1,"last update":"Fri Oct 6 03:30:26 2023","interfaces":[{"interface":"back","average":{"1min":0.93000000000000005,"5min":0.93000000000000005,"15min":0.93000000000000005},"min":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"max":{"1min":1.2889999999999999,"5min":1.2889999999999999,"15min":1.2889999999999999},"last":0.58799999999999997},{"interface":"front","average":{"1min":0.94299999999999995,"5min":0.94299999999999995,"15min":0.94299999999999995},"min":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"max":{"1min":1.341,"5min":1.341,"15min":1.341},"last":0.61799999999999999}]},{"osd":2,"last update":"Fri Oct 6 03:30:26 2023","interfaces":[{"interface":"back","average":{"1min":0.95299999999999996,"5min":0.95299999999999996,"15min":0.95299999999999996},"min":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.72599999999999998},"max":{"1min":1.2050000000000001,"5min":1.2050000000000001,"15min":1.2050000000000001},"last":0.76800000000000002},{"interface":"front","average":{"1min":1.006,"5min":1.006,"15min":1.006},"min":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"max":{"1min":1.409,"5min":1.409,"15min":1.409},"last":0.73199999999999998}]},{"osd":3,"last update":"Fri Oct 6 03:30:26 2023","interfaces":[{"interface":"back","average":{"1min":0.86299999999999999,"5min":0.86299999999999999,"15min":0.86299999999999999},"min":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"max":{"1min":1.1339999999999999,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.85999999999999999},{"interface":"front","average":{"1min":0.96899999999999997,"5min":0.96899999999999997,"15min":0.96899999999999997},"min":{"1min":0.63400000000000001,"5min":0.63400000000000001,"15min":0.63400000000000001},"max":{"1min":1.256,"5min":1.256,"15min":1.256},"last":0.80800000000000005}]},{"osd":4,"last update":"Fri Oct 6 03:30:26 2023","interfaces":[{"interface":"back","average":{"1min":0.91900000000000004,"5min":0.91900000000000004,"15min":0.91900000000000004},"min":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"max":{"1min":1.2230000000000001,"5min":1.2230000000000001,"15min":1.2230000000000001},"last":0.83699999999999997},{"interface":"front","average":{"1min":1.0529999999999999,"5min":1.0529999999999999,"15min":1.0529999999999999},"min":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"max":{"1min":1.4750000000000001,"5min":1.4750000000000001,"15min":1.4750000000000001},"last":1.0529999999999999}]},{"osd":5,"last update":"Fri Oct 6 03:30:26 2023","interfaces":[{"interface":"back","average":{"1min":1.0229999999999999,"5min":1.0229999999999999,"15min":1.0229999999999999},"min":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":0.77900000000000003},"max":{"1min":1.431,"5min":1.431,"15min":1.431},"last":1.0309999999999999},{"interface":"front","average":{"1min":1.105,"5min":1.105,"15min":1.105},"min":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"max":{"1min":1.407,"5min":1.407,"15min":1.407},"last":0.91200000000000003}]},{"osd":6,"last update":"Fri Oct 6 03:30:26 2023","interfaces":[{"interface":"back","average":{"1min":1.1399999999999999,"5min":1.1399999999999999,"15min":1.1399999999999999},"min":{"1min":0.86799999999999999,"5min":0.86799999999999999,"15min":0.86799999999999999},"max":{"1min":1.526,"5min":1.526,"15min":1.526},"last":1.1259999999999999},{"interface":"front","average":{"1min":1.198,"5min":1.198,"15min":1.198},"min":{"1min":0.98399999999999999,"5min":0.98399999999999999,"15min":0.98399999999999999},"max":{"1min":1.6719999999999999,"5min":1.6719999999999999,"15min":1.6719999999999999},"last":1.1719999999999999}]}]},{"osd":6,"up_from":16,"seq":68719476761,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297776,"kb_used_data":488,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074064,"statfs":{"total":96636764160,"available":96331841536,"internally_reserved":0,"allocated":499712,"data_stored":220402,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Oct 6 03:30:26 2023","interfaces":[{"interface":"back","average":{"1min":0.876,"5min":0.876,"15min":0.876},"min":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"max":{"1min":1.149,"5min":1.149,"15min":1.149},"last":0.77000000000000002},{"interface":"front","average":{"1min":0.90400000000000003,"5min":0.90400000000000003,"15min":0.90400000000000003},"min":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"max":{"1min":1.254,"5min":1.254,"15min":1.254},"last":0.70599999999999996}]},{"osd":1,"last update":"Fri Oct 6 03:30:26 2023","interfaces":[{"interface":"back","average":{"1min":0.81299999999999994,"5min":0.81299999999999994,"15min":0.81299999999999994},"min":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"max":{"1min":1.4730000000000001,"5min":1.4730000000000001,"15min":1.4730000000000001},"last":1.1459999999999999},{"interface":"front","average":{"1min":0.97099999999999997,"5min":0.97099999999999997,"15min":0.97099999999999997},"min":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"max":{"1min":1.3440000000000001,"5min":1.3440000000000001,"15min":1.3440000000000001},"last":1.127}]},{"osd":2,"last update":"Fri Oct 6 03:30:26 2023","interfaces":[{"interface":"back","average":{"1min":0.94099999999999995,"5min":0.94099999999999995,"15min":0.94099999999999995},"min":{"1min":0.629,"5min":0.629,"15min":0.629},"max":{"1min":1.236,"5min":1.236,"15min":1.236},"last":0.90400000000000003},{"interface":"front","average":{"1min":0.95599999999999996,"5min":0.95599999999999996,"15min":0.95599999999999996},"min":{"1min":0.623,"5min":0.623,"15min":0.623},"max":{"1min":1.3660000000000001,"5min":1.3660000000000001,"15min":1.3660000000000001},"last":0.82799999999999996}]},{"osd":3,"last update":"Fri Oct 6 03:30:26 2023","interfaces":[{"interface":"back","average":{"1min":0.99299999999999999,"5min":0.99299999999999999,"15min":0.99299999999999999},"min":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":1.3660000000000001,"5min":1.3660000000000001,"15min":1.3660000000000001},"last":0.98099999999999998},{"interface":"front","average":{"1min":0.95599999999999996,"5min":0.95599999999999996,"15min":0.95599999999999996},"min":{"1min":0.76300000000000001,"5min":0.76300000000000001,"15min":0.76300000000000001},"max":{"1min":1.2609999999999999,"5min":1.2609999999999999,"15min":1.2609999999999999},"last":1.008}]},{"osd":4,"last update":"Fri Oct 6 03:30:26 2023","interfaces":[{"interface":"back","average":{"1min":1.1040000000000001,"5min":1.1040000000000001,"15min":1.1040000000000001},"min":{"1min":0.875,"5min":0.875,"15min":0.875},"max":{"1min":1.2869999999999999,"5min":1.2869999999999999,"15min":1.2869999999999999},"last":1.0700000000000001},{"interface":"front","average":{"1min":1.1140000000000001,"5min":1.1140000000000001,"15min":1.1140000000000001},"min":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.73399999999999999},"max":{"1min":1.4450000000000001,"5min":1.4450000000000001,"15min":1.4450000000000001},"last":1.194}]},{"osd":5,"last update":"Fri Oct 6 03:30:26 2023","interfaces":[{"interface":"back","average":{"1min":1.105,"5min":1.105,"15min":1.105},"min":{"1min":0.86799999999999999,"5min":0.86799999999999999,"15min":0.86799999999999999},"max":{"1min":1.613,"5min":1.613,"15min":1.613},"last":1.5620000000000001},{"interface":"front","average":{"1min":1.1699999999999999,"5min":1.1699999999999999,"15min":1.1699999999999999},"min":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"max":{"1min":1.351,"5min":1.351,"15min":1.351},"last":1.171}]},{"osd":7,"last update":"Fri Oct 6 03:30:26 2023","interfaces":[{"interface":"back","average":{"1min":1.2370000000000001,"5min":1.2370000000000001,"15min":1.2370000000000001},"min":{"1min":1.012,"5min":1.012,"15min":1.012},"max":{"1min":1.5409999999999999,"5min":1.5409999999999999,"15min":1.5409999999999999},"last":1.0980000000000001},{"interface":"front","average":{"1min":1.212,"5min":1.212,"15min":1.212},"min":{"1min":0.96099999999999997,"5min":0.96099999999999997,"15min":0.96099999999999997},"max":{"1min":1.704,"5min":1.704,"15min":1.704},"last":1.6539999999999999}]}]},{"osd":5,"up_from":16,"seq":68719476761,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298116,"kb_used_data":828,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94073724,"statfs":{"total":96636764160,"available":96331493376,"internally_reserved":0,"allocated":847872,"data_stored":562776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Oct 6 03:30:23 2023","interfaces":[{"interface":"back","average":{"1min":0.86399999999999999,"5min":0.86399999999999999,"15min":0.86399999999999999},"min":{"1min":0.505,"5min":0.505,"15min":0.505},"max":{"1min":1.4079999999999999,"5min":1.4079999999999999,"15min":1.4079999999999999},"last":1.141},{"interface":"front","average":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":0.89800000000000002},"min":{"1min":0.64500000000000002,"5min":0.64500000000000002,"15min":0.64500000000000002},"max":{"1min":1.3779999999999999,"5min":1.3779999999999999,"15min":1.3779999999999999},"last":0.93700000000000006}]},{"osd":1,"last update":"Fri Oct 6 03:30:23 2023","interfaces":[{"interface":"back","average":{"1min":0.97799999999999998,"5min":0.97799999999999998,"15min":0.97799999999999998},"min":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":1.2829999999999999,"5min":1.2829999999999999,"15min":1.2829999999999999},"last":0.66200000000000003},{"interface":"front","average":{"1min":0.85999999999999999,"5min":0.85999999999999999,"15min":0.85999999999999999},"min":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"max":{"1min":1.581,"5min":1.581,"15min":1.581},"last":0.72599999999999998}]},{"osd":2,"last update":"Fri Oct 6 03:30:23 2023","interfaces":[{"interface":"back","average":{"1min":1,"5min":1,"15min":1},"min":{"1min":0.67300000000000004,"5min":0.67300000000000004,"15min":0.67300000000000004},"max":{"1min":1.258,"5min":1.258,"15min":1.258},"last":1.1919999999999999},{"interface":"front","average":{"1min":0.90600000000000003,"5min":0.90600000000000003,"15min":0.90600000000000003},"min":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.67100000000000004},"max":{"1min":1.367,"5min":1.367,"15min":1.367},"last":0.84099999999999997}]},{"osd":3,"last update":"Fri Oct 6 03:30:23 2023","interfaces":[{"interface":"back","average":{"1min":1.034,"5min":1.034,"15min":1.034},"min":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.77200000000000002},"max":{"1min":1.4079999999999999,"5min":1.4079999999999999,"15min":1.4079999999999999},"last":1.0920000000000001},{"interface":"front","average":{"1min":1.034,"5min":1.034,"15min":1.034},"min":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"max":{"1min":1.244,"5min":1.244,"15min":1.244},"last":0.999}]},{"osd":4,"last update":"Fri Oct 6 03:30:23 2023","interfaces":[{"interface":"back","average":{"1min":1.0920000000000001,"5min":1.0920000000000001,"15min":1.0920000000000001},"min":{"1min":0.752,"5min":0.752,"15min":0.752},"max":{"1min":2.2050000000000001,"5min":2.2050000000000001,"15min":2.2050000000000001},"last":1.2310000000000001},{"interface":"front","average":{"1min":1.1870000000000001,"5min":1.1870000000000001,"15min":1.1870000000000001},"min":{"1min":0.90900000000000003,"5min":0.90900000000000003,"15min":0.90900000000000003},"max":{"1min":1.518,"5min":1.518,"15min":1.518},"last":1.2629999999999999}]},{"osd":6,"last update":"Fri Oct 6 03:30:23 2023","interfaces":[{"interface":"back","average":{"1min":1.1040000000000001,"5min":1.1040000000000001,"15min":1.1040000000000001},"min":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"max":{"1min":1.3169999999999999,"5min":1.3169999999999999,"15min":1.3169999999999999},"last":1.4259999999999999},{"interface":"front","average":{"1min":1.2370000000000001,"5min":1.2370000000000001,"15min":1.2370000000000001},"min":{"1min":0.94599999999999995,"5min":0.94599999999999995,"15min":0.94599999999999995},"max":{"1min":2.3809999999999998,"5min":2.3809999999999998,"15min":2.3809999999999998},"last":1.3859999999999999}]},{"osd":7,"last update":"Fri Oct 6 03:30:23 2023","interfaces":[{"interface":"back","average":{"1min":1.2370000000000001,"5min":1.2370000000000001,"15min":1.2370000000000001},"min":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.79500000000000004},"max":{"1min":1.637,"5min":1.637,"15min":1.637},"last":1.6060000000000001},{"interface":"front","average":{"1min":1.2929999999999999,"5min":1.2929999999999999,"15min":1.2929999999999999},"min":{"1min":1.0429999999999999,"5min":1.0429999999999999,"15min":1.0429999999999999},"max":{"1min":2.3330000000000002,"5min":2.3330000000000002,"15min":2.3330000000000002},"last":1.339}]}]},{"osd":4,"up_from":16,"seq":68719476760,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298116,"kb_used_data":828,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94073724,"statfs":{"total":96636764160,"available":96331493376,"internally_reserved":0,"allocated":847872,"data_stored":564896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Oct 6 03:30:25 2023","interfaces":[{"interface":"back","average":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"min":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"max":{"1min":1.129,"5min":1.129,"15min":1.129},"last":0.71999999999999997},{"interface":"front","average":{"1min":0.96299999999999997,"5min":0.96299999999999997,"15min":0.96299999999999997},"min":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"max":{"1min":1.629,"5min":1.629,"15min":1.629},"last":0.55800000000000005}]},{"osd":1,"last update":"Fri Oct 6 03:30:25 2023","interfaces":[{"interface":"back","average":{"1min":0.89300000000000002,"5min":0.89300000000000002,"15min":0.89300000000000002},"min":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"max":{"1min":1.23,"5min":1.23,"15min":1.23},"last":1.0680000000000001},{"interface":"front","average":{"1min":0.86799999999999999,"5min":0.86799999999999999,"15min":0.86799999999999999},"min":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"max":{"1min":1.3680000000000001,"5min":1.3680000000000001,"15min":1.3680000000000001},"last":0.59899999999999998}]},{"osd":2,"last update":"Fri Oct 6 03:30:25 2023","interfaces":[{"interface":"back","average":{"1min":0.98999999999999999,"5min":0.98999999999999999,"15min":0.98999999999999999},"min":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":1.5549999999999999,"5min":1.5549999999999999,"15min":1.5549999999999999},"last":0.91300000000000003},{"interface":"front","average":{"1min":0.92100000000000004,"5min":0.92100000000000004,"15min":0.92100000000000004},"min":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.60799999999999998},"max":{"1min":1.306,"5min":1.306,"15min":1.306},"last":0.63400000000000001}]},{"osd":3,"last update":"Fri Oct 6 03:30:25 2023","interfaces":[{"interface":"back","average":{"1min":1.1040000000000001,"5min":1.1040000000000001,"15min":1.1040000000000001},"min":{"1min":0.84499999999999997,"5min":0.84499999999999997,"15min":0.84499999999999997},"max":{"1min":1.5580000000000001,"5min":1.5580000000000001,"15min":1.5580000000000001},"last":1.0229999999999999},{"interface":"front","average":{"1min":1.04,"5min":1.04,"15min":1.04},"min":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.67500000000000004},"max":{"1min":1.6699999999999999,"5min":1.6699999999999999,"15min":1.6699999999999999},"last":0.98699999999999999}]},{"osd":5,"last update":"Fri Oct 6 03:30:25 2023","interfaces":[{"interface":"back","average":{"1min":1.044,"5min":1.044,"15min":1.044},"min":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.73399999999999999},"max":{"1min":1.5169999999999999,"5min":1.5169999999999999,"15min":1.5169999999999999},"last":0.94799999999999995},{"interface":"front","average":{"1min":1.266,"5min":1.266,"15min":1.266},"min":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":0.82799999999999996},"max":{"1min":2.2029999999999998,"5min":2.2029999999999998,"15min":2.2029999999999998},"last":0.79500000000000004}]},{"osd":6,"last update":"Fri Oct 6 03:30:25 2023","interfaces":[{"interface":"back","average":{"1min":1.181,"5min":1.181,"15min":1.181},"min":{"1min":0.79900000000000004,"5min":0.79900000000000004,"15min":0.79900000000000004},"max":{"1min":2.0209999999999999,"5min":2.0209999999999999,"15min":2.0209999999999999},"last":0.76600000000000001},{"interface":"front","average":{"1min":1.179,"5min":1.179,"15min":1.179},"min":{"1min":0.71399999999999997,"5min":0.71399999999999997,"15min":0.71399999999999997},"max":{"1min":2.1440000000000001,"5min":2.1440000000000001,"15min":2.1440000000000001},"last":0.85399999999999998}]},{"osd":7,"last update":"Fri Oct 6 03:30:25 2023","interfaces":[{"interface":"back","average":{"1min":1.337,"5min":1.337,"15min":1.337},"min":{"1min":0.94399999999999995,"5min":0.94399999999999995,"15min":0.94399999999999995},"max":{"1min":2.0939999999999999,"5min":2.0939999999999999,"15min":2.0939999999999999},"last":1.1080000000000001},{"interface":"front","average":{"1min":1.3089999999999999,"5min":1.3089999999999999,"15min":1.3089999999999999},"min":{"1min":0.95899999999999996,"5min":0.95899999999999996,"15min":0.95899999999999996},"max":{"1min":1.968,"5min":1.968,"15min":1.968},"last":1.147}]}]},{"osd":3,"up_from":14,"seq":60129542168,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302204,"kb_used_data":820,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94069636,"statfs":{"total":96636764160,"available":96327307264,"internally_reserved":0,"allocated":839680,"data_stored":555463,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Oct 6 03:30:26 2023","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.81899999999999995,"5min":0.81899999999999995,"15min":0.81899999999999995},"last":0.86899999999999999},{"interface":"front","average":{"1min":0.754,"5min":0.754,"15min":0.754},"min":{"1min":0.505,"5min":0.505,"15min":0.505},"max":{"1min":1.0920000000000001,"5min":1.0920000000000001,"15min":1.0920000000000001},"last":0.54800000000000004}]},{"osd":1,"last update":"Fri Oct 6 03:30:20 2023","interfaces":[{"interface":"back","average":{"1min":0.83199999999999996,"5min":0.83199999999999996,"15min":0.83199999999999996},"min":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":1.2210000000000001,"5min":1.2210000000000001,"15min":1.2210000000000001},"last":0.79600000000000004},{"interface":"front","average":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":1.321,"5min":1.321,"15min":1.321},"last":0.60999999999999999}]},{"osd":2,"last update":"Fri Oct 6 03:30:20 2023","interfaces":[{"interface":"back","average":{"1min":0.86799999999999999,"5min":0.86799999999999999,"15min":0.86799999999999999},"min":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.61199999999999999},"max":{"1min":1.228,"5min":1.228,"15min":1.228},"last":0.92900000000000005},{"interface":"front","average":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"min":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":1.3680000000000001,"5min":1.3680000000000001,"15min":1.3680000000000001},"last":0.47699999999999998}]},{"osd":4,"last update":"Fri Oct 6 03:30:26 2023","interfaces":[{"interface":"back","average":{"1min":1.0129999999999999,"5min":1.0129999999999999,"15min":1.0129999999999999},"min":{"1min":0.76800000000000002,"5min":0.76800000000000002,"15min":0.76800000000000002},"max":{"1min":1.6359999999999999,"5min":1.6359999999999999,"15min":1.6359999999999999},"last":0.98599999999999999},{"interface":"front","average":{"1min":0.96699999999999997,"5min":0.96699999999999997,"15min":0.96699999999999997},"min":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"max":{"1min":1.409,"5min":1.409,"15min":1.409},"last":0.91000000000000003}]},{"osd":5,"last update":"Fri Oct 6 03:30:26 2023","interfaces":[{"interface":"back","average":{"1min":0.98999999999999999,"5min":0.98999999999999999,"15min":0.98999999999999999},"min":{"1min":0.746,"5min":0.746,"15min":0.746},"max":{"1min":1.387,"5min":1.387,"15min":1.387},"last":0.96599999999999997},{"interface":"front","average":{"1min":1.1359999999999999,"5min":1.1359999999999999,"15min":1.1359999999999999},"min":{"1min":0.89000000000000001,"5min":0.89000000000000001,"15min":0.89000000000000001},"max":{"1min":1.615,"5min":1.615,"15min":1.615},"last":1.0269999999999999}]},{"osd":6,"last update":"Fri Oct 6 03:30:26 2023","interfaces":[{"interface":"back","average":{"1min":1.079,"5min":1.079,"15min":1.079},"min":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"max":{"1min":1.643,"5min":1.643,"15min":1.643},"last":0.66800000000000004},{"interface":"front","average":{"1min":1.105,"5min":1.105,"15min":1.105},"min":{"1min":0.74399999999999999,"5min":0.74399999999999999,"15min":0.74399999999999999},"max":{"1min":1.6819999999999999,"5min":1.6819999999999999,"15min":1.6819999999999999},"last":1.0509999999999999}]},{"osd":7,"last update":"Fri Oct 6 03:30:26 2023","interfaces":[{"interface":"back","average":{"1min":1.1850000000000001,"5min":1.1850000000000001,"15min":1.1850000000000001},"min":{"1min":0.92000000000000004,"5min":0.92000000000000004,"15min":0.92000000000000004},"max":{"1min":1.597,"5min":1.597,"15min":1.597},"last":0.70799999999999996},{"interface":"front","average":{"1min":1.111,"5min":1.111,"15min":1.111},"min":{"1min":0.86599999999999999,"5min":0.86599999999999999,"15min":0.86599999999999999},"max":{"1min":1.446,"5min":1.446,"15min":1.446},"last":0.83399999999999996}]}]},{"osd":2,"up_from":14,"seq":60129542167,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":301908,"kb_used_data":524,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94069932,"statfs":{"total":96636764160,"available":96327610368,"internally_reserved":0,"allocated":536576,"data_stored":245337,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Oct 6 03:30:21 2023","interfaces":[{"interface":"back","average":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":1.1240000000000001,"5min":1.1240000000000001,"15min":1.1240000000000001},"last":0.76700000000000002},{"interface":"front","average":{"1min":0.70399999999999996,"5min":0.70399999999999996,"15min":0.70399999999999996},"min":{"1min":0.495,"5min":0.495,"15min":0.495},"max":{"1min":1.1559999999999999,"5min":1.1559999999999999,"15min":1.1559999999999999},"last":0.495}]},{"osd":1,"last update":"Fri Oct 6 03:30:21 2023","interfaces":[{"interface":"back","average":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"min":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":1.1060000000000001,"5min":1.1060000000000001,"15min":1.1060000000000001},"last":1.0449999999999999},{"interface":"front","average":{"1min":0.73499999999999999,"5min":0.73499999999999999,"15min":0.73499999999999999},"min":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":1.2390000000000001,"5min":1.2390000000000001,"15min":1.2390000000000001},"last":0.52700000000000002}]},{"osd":3,"last update":"Fri Oct 6 03:30:21 2023","interfaces":[{"interface":"back","average":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":0.78500000000000003},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":1.165,"5min":1.165,"15min":1.165},"last":0.66300000000000003},{"interface":"front","average":{"1min":0.873,"5min":0.873,"15min":0.873},"min":{"1min":0.626,"5min":0.626,"15min":0.626},"max":{"1min":1.2949999999999999,"5min":1.2949999999999999,"15min":1.2949999999999999},"last":0.45300000000000001}]},{"osd":4,"last update":"Fri Oct 6 03:30:23 2023","interfaces":[{"interface":"back","average":{"1min":1.095,"5min":1.095,"15min":1.095},"min":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"max":{"1min":2.3450000000000002,"5min":2.3450000000000002,"15min":2.3450000000000002},"last":0.86599999999999999},{"interface":"front","average":{"1min":0.95599999999999996,"5min":0.95599999999999996,"15min":0.95599999999999996},"min":{"1min":0.74399999999999999,"5min":0.74399999999999999,"15min":0.74399999999999999},"max":{"1min":1.1950000000000001,"5min":1.1950000000000001,"15min":1.1950000000000001},"last":0.83399999999999996}]},{"osd":5,"last update":"Fri Oct 6 03:30:23 2023","interfaces":[{"interface":"back","average":{"1min":1.139,"5min":1.139,"15min":1.139},"min":{"1min":0.79900000000000004,"5min":0.79900000000000004,"15min":0.79900000000000004},"max":{"1min":2.5169999999999999,"5min":2.5169999999999999,"15min":2.5169999999999999},"last":0.93600000000000005},{"interface":"front","average":{"1min":1.107,"5min":1.107,"15min":1.107},"min":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":0.81799999999999995},"max":{"1min":1.7250000000000001,"5min":1.7250000000000001,"15min":1.7250000000000001},"last":0.81299999999999994}]},{"osd":6,"last update":"Fri Oct 6 03:30:23 2023","interfaces":[{"interface":"back","average":{"1min":1.014,"5min":1.014,"15min":1.014},"min":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"max":{"1min":1.218,"5min":1.218,"15min":1.218},"last":0.55200000000000005},{"interface":"front","average":{"1min":1.1899999999999999,"5min":1.1899999999999999,"15min":1.1899999999999999},"min":{"1min":0.85899999999999999,"5min":0.85899999999999999,"15min":0.85899999999999999},"max":{"1min":2.2450000000000001,"5min":2.2450000000000001,"15min":2.2450000000000001},"last":1.0029999999999999}]},{"osd":7,"last update":"Fri Oct 6 03:30:21 2023","interfaces":[{"interface":"back","average":{"1min":1.1639999999999999,"5min":1.1639999999999999,"15min":1.1639999999999999},"min":{"1min":0.81599999999999995,"5min":0.81599999999999995,"15min":0.81599999999999995},"max":{"1min":1.8140000000000001,"5min":1.8140000000000001,"15min":1.8140000000000001},"last":0.70299999999999996},{"interface":"front","average":{"1min":1.0860000000000001,"5min":1.0860000000000001,"15min":1.0860000000000001},"min":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.79500000000000004},"max":{"1min":1.2749999999999999,"5min":1.2749999999999999,"15min":1.2749999999999999},"last":0.71599999999999997}]}]},{"osd":1,"up_from":14,"seq":60129542167,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302204,"kb_used_data":820,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94069636,"statfs":{"total":96636764160,"available":96327307264,"internally_reserved":0,"allocated":839680,"data_stored":555576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Oct 6 03:30:26 2023","interfaces":[{"interface":"back","average":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.60799999999999998},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.48599999999999999},{"interface":"front","average":{"1min":0.71599999999999997,"5min":0.71599999999999997,"15min":0.71599999999999997},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":1.5329999999999999,"5min":1.5329999999999999,"15min":1.5329999999999999},"last":0.59199999999999997}]},{"osd":2,"last update":"Fri Oct 6 03:30:20 2023","interfaces":[{"interface":"back","average":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.93899999999999995,"5min":0.93899999999999995,"15min":0.93899999999999995},"last":1.018},{"interface":"front","average":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"min":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"max":{"1min":1.278,"5min":1.278,"15min":1.278},"last":0.66600000000000004}]},{"osd":3,"last update":"Fri Oct 6 03:30:20 2023","interfaces":[{"interface":"back","average":{"1min":0.72899999999999998,"5min":0.72899999999999998,"15min":0.72899999999999998},"min":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"max":{"1min":0.90500000000000003,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.53700000000000003},{"interface":"front","average":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.3700000000000001,"5min":1.3700000000000001,"15min":1.3700000000000001},"last":0.80600000000000005}]},{"osd":4,"last update":"Fri Oct 6 03:30:26 2023","interfaces":[{"interface":"back","average":{"1min":0.94299999999999995,"5min":0.94299999999999995,"15min":0.94299999999999995},"min":{"1min":0.66400000000000003,"5min":0.66400000000000003,"15min":0.66400000000000003},"max":{"1min":1.7549999999999999,"5min":1.7549999999999999,"15min":1.7549999999999999},"last":0.96799999999999997},{"interface":"front","average":{"1min":0.97499999999999998,"5min":0.97499999999999998,"15min":0.97499999999999998},"min":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":1.415,"5min":1.415,"15min":1.415},"last":1.2250000000000001}]},{"osd":5,"last update":"Fri Oct 6 03:30:26 2023","interfaces":[{"interface":"back","average":{"1min":0.93999999999999995,"5min":0.93999999999999995,"15min":0.93999999999999995},"min":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"max":{"1min":1.4910000000000001,"5min":1.4910000000000001,"15min":1.4910000000000001},"last":1.0980000000000001},{"interface":"front","average":{"1min":1.0049999999999999,"5min":1.0049999999999999,"15min":1.0049999999999999},"min":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":0.70499999999999996},"max":{"1min":1.4370000000000001,"5min":1.4370000000000001,"15min":1.4370000000000001},"last":1.01}]},{"osd":6,"last update":"Fri Oct 6 03:30:26 2023","interfaces":[{"interface":"back","average":{"1min":1.022,"5min":1.022,"15min":1.022},"min":{"1min":0.69199999999999995,"5min":0.69199999999999995,"15min":0.69199999999999995},"max":{"1min":1.603,"5min":1.603,"15min":1.603},"last":1.3},{"interface":"front","average":{"1min":1.1080000000000001,"5min":1.1080000000000001,"15min":1.1080000000000001},"min":{"1min":0.69199999999999995,"5min":0.69199999999999995,"15min":0.69199999999999995},"max":{"1min":1.5069999999999999,"5min":1.5069999999999999,"15min":1.5069999999999999},"last":1.365}]},{"osd":7,"last update":"Fri Oct 6 03:30:26 2023","interfaces":[{"interface":"back","average":{"1min":1.04,"5min":1.04,"15min":1.04},"min":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.60799999999999998},"max":{"1min":1.6160000000000001,"5min":1.6160000000000001,"15min":1.6160000000000001},"last":0.98299999999999998},{"interface":"front","average":{"1min":1.071,"5min":1.071,"15min":1.071},"min":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.67500000000000004},"max":{"1min":1.5700000000000001,"5min":1.5700000000000001,"15min":1.5700000000000001},"last":1.1299999999999999}]}]},{"osd":0,"up_from":15,"seq":64424509465,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302468,"kb_used_data":1084,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94069372,"statfs":{"total":96636764160,"available":96327036928,"internally_reserved":0,"allocated":1110016,"data_stored":824876,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Oct 6 03:30:22 2023","interfaces":[{"interface":"back","average":{"1min":0.629,"5min":0.629,"15min":0.629},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":1.0029999999999999,"5min":1.0029999999999999,"15min":1.0029999999999999},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.74199999999999999},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":1.179,"5min":1.179,"15min":1.179},"last":0.63300000000000001}]},{"osd":2,"last update":"Fri Oct 6 03:30:22 2023","interfaces":[{"interface":"back","average":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":1.1879999999999999,"5min":1.1879999999999999,"15min":1.1879999999999999},"last":0.439},{"interface":"front","average":{"1min":0.72499999999999998,"5min":0.72499999999999998,"15min":0.72499999999999998},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":1.3879999999999999,"5min":1.3879999999999999,"15min":1.3879999999999999},"last":0.52300000000000002}]},{"osd":3,"last update":"Fri Oct 6 03:30:22 2023","interfaces":[{"interface":"back","average":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":1.1930000000000001,"5min":1.1930000000000001,"15min":1.1930000000000001},"last":0.67500000000000004},{"interface":"front","average":{"1min":0.86399999999999999,"5min":0.86399999999999999,"15min":0.86399999999999999},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":1.1970000000000001,"5min":1.1970000000000001,"15min":1.1970000000000001},"last":0.71399999999999997}]},{"osd":4,"last update":"Fri Oct 6 03:30:22 2023","interfaces":[{"interface":"back","average":{"1min":1.0189999999999999,"5min":1.0189999999999999,"15min":1.0189999999999999},"min":{"1min":0.72899999999999998,"5min":0.72899999999999998,"15min":0.72899999999999998},"max":{"1min":1.55,"5min":1.55,"15min":1.55},"last":0.98499999999999999},{"interface":"front","average":{"1min":0.99099999999999999,"5min":0.99099999999999999,"15min":0.99099999999999999},"min":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"max":{"1min":1.5269999999999999,"5min":1.5269999999999999,"15min":1.5269999999999999},"last":1.2969999999999999}]},{"osd":5,"last update":"Fri Oct 6 03:30:22 2023","interfaces":[{"interface":"back","average":{"1min":0.98499999999999999,"5min":0.98499999999999999,"15min":0.98499999999999999},"min":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"max":{"1min":1.5409999999999999,"5min":1.5409999999999999,"15min":1.5409999999999999},"last":0.91100000000000003},{"interface":"front","average":{"1min":1.087,"5min":1.087,"15min":1.087},"min":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"max":{"1min":1.353,"5min":1.353,"15min":1.353},"last":1.24}]},{"osd":6,"last update":"Fri Oct 6 03:30:22 2023","interfaces":[{"interface":"back","average":{"1min":1.145,"5min":1.145,"15min":1.145},"min":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"max":{"1min":1.502,"5min":1.502,"15min":1.502},"last":1.1859999999999999},{"interface":"front","average":{"1min":1.1299999999999999,"5min":1.1299999999999999,"15min":1.1299999999999999},"min":{"1min":0.64900000000000002,"5min":0.64900000000000002,"15min":0.64900000000000002},"max":{"1min":1.6439999999999999,"5min":1.6439999999999999,"15min":1.6439999999999999},"last":1.085}]},{"osd":7,"last update":"Fri Oct 6 03:30:22 2023","interfaces":[{"interface":"back","average":{"1min":1.099,"5min":1.099,"15min":1.099},"min":{"1min":0.874,"5min":0.874,"15min":0.874},"max":{"1min":1.369,"5min":1.369,"15min":1.369},"last":1.046},{"interface":"front","average":{"1min":1.2030000000000001,"5min":1.2030000000000001,"15min":1.2030000000000001},"min":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"max":{"1min":1.573,"5min":1.573,"15min":1.573},"last":1.0209999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-06T03:31:09.094 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-06T03:31:09.202 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:09.196+0000 7f6f4bfff700 1 Processor -- start 2023-10-06T03:31:09.202 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:09.196+0000 7f6f4bfff700 1 -- start start 2023-10-06T03:31:09.203 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:09.196+0000 7f6f4bfff700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6f4c15cf70 0x7f6f4c06bbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:31:09.204 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:09.196+0000 7f6f4bfff700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f6f4c15d340 0x7f6f4c0cc030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:31:09.204 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:09.196+0000 7f6f4bfff700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f4c06b750 con 0x7f6f4c0c9e00 2023-10-06T03:31:09.204 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:09.196+0000 7f6f4bfff700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6f4c06ab40 con 0x7f6f4c15d340 2023-10-06T03:31:09.204 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:09.196+0000 7f6f4bfff700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f4c06ae20 con 0x7f6f4c15cf70 2023-10-06T03:31:09.204 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:09.200+0000 7f6f4a7fc700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f6f4c15d340 0x7f6f4c0cc030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:31:09.204 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:09.200+0000 7f6f4b7fe700 1 --1- >> v1:172.21.15.192:6789/0 conn(0x7f6f4c0c9e00 0x7f6f4c0c6420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.110:52012/0 (socket says 172.21.15.110:52012) 2023-10-06T03:31:09.204 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:09.200+0000 7f6f4a7fc700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f6f4c15d340 0x7f6f4c0cc030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.110:55282/0 (socket says 172.21.15.110:55282) 2023-10-06T03:31:09.204 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:09.200+0000 7f6f4affd700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6f4c15cf70 0x7f6f4c06bbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:31:09.204 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:09.200+0000 7f6f4b7fe700 1 -- 172.21.15.110:0/504021479 learned_addr learned my addr 172.21.15.110:0/504021479 (peer_addr_for_me v1:172.21.15.110:0/0) 2023-10-06T03:31:09.206 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:09.200+0000 7f6f4a7fc700 1 -- 172.21.15.110:0/504021479 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6f4c15cf70 msgr2=0x7f6f4c06bbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:31:09.206 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:09.200+0000 7f6f4a7fc700 1 --2- 172.21.15.110:0/504021479 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6f4c15cf70 0x7f6f4c06bbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:09.206 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:09.200+0000 7f6f4a7fc700 1 -- 172.21.15.110:0/504021479 >> v1:172.21.15.192:6789/0 conn(0x7f6f4c0c9e00 legacy=0x7f6f4c0c6420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:31:09.206 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:09.200+0000 7f6f4a7fc700 1 -- 172.21.15.110:0/504021479 --> v2:172.21.15.192:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f4c06b100 con 0x7f6f4c15d340 2023-10-06T03:31:09.206 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:09.200+0000 7f6f49ffb700 1 -- 172.21.15.110:0/504021479 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 139098208 0 0) 0x7f6f4c06b750 con 0x7f6f4c0c9e00 2023-10-06T03:31:09.206 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:09.200+0000 7f6f4a7fc700 1 --2- 172.21.15.110:0/504021479 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6f4c15d340 0x7f6f4c0cc030 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f6f34009e60 tx=0x7f6f34009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=15b77fe42d7d3d7b server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:31:09.206 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:09.200+0000 7f6f49ffb700 1 -- 172.21.15.110:0/504021479 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f3400e070 con 0x7f6f4c15d340 2023-10-06T03:31:09.207 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:09.200+0000 7f6f49ffb700 1 -- 172.21.15.110:0/504021479 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f3400bb60 con 0x7f6f4c15d340 2023-10-06T03:31:09.207 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:09.200+0000 7f6f49ffb700 1 -- 172.21.15.110:0/504021479 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f34012430 con 0x7f6f4c15d340 2023-10-06T03:31:09.207 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:09.200+0000 7f6f4bfff700 1 -- 172.21.15.110:0/504021479 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6f4c15d340 msgr2=0x7f6f4c0cc030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:31:09.207 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:09.200+0000 7f6f4bfff700 1 --2- 172.21.15.110:0/504021479 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6f4c15d340 0x7f6f4c0cc030 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f6f34009e60 tx=0x7f6f34009780 comp rx=0 tx=0).stop 2023-10-06T03:31:09.207 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:09.200+0000 7f6f4bfff700 1 -- 172.21.15.110:0/504021479 shutdown_connections 2023-10-06T03:31:09.207 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:09.200+0000 7f6f4bfff700 1 --2- 172.21.15.110:0/504021479 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6f4c15d340 0x7f6f4c0cc030 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:09.207 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:09.200+0000 7f6f4bfff700 1 --2- 172.21.15.110:0/504021479 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6f4c15cf70 0x7f6f4c06bbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:09.207 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:09.200+0000 7f6f4bfff700 1 -- 172.21.15.110:0/504021479 >> 172.21.15.110:0/504021479 conn(0x7f6f4c06c760 msgr2=0x7f6f4c06cb60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:31:09.207 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:09.200+0000 7f6f4bfff700 1 -- 172.21.15.110:0/504021479 shutdown_connections 2023-10-06T03:31:09.207 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:09.200+0000 7f6f4bfff700 1 -- 172.21.15.110:0/504021479 wait complete. 2023-10-06T03:31:09.208 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:09.204+0000 7f6f4bfff700 1 Processor -- start 2023-10-06T03:31:09.208 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:09.204+0000 7f6f4bfff700 1 -- start start 2023-10-06T03:31:09.209 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:09.204+0000 7f6f4bfff700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6f4c0c9e00 0x7f6f4c0c29d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:31:09.209 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:09.204+0000 7f6f4bfff700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6f4c15cf70 0x7f6f4c160080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:31:09.209 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:09.204+0000 7f6f4bfff700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6f4c15d340 0x7f6f4c170e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:31:09.209 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:09.204+0000 7f6f4bfff700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f4c1706b0 con 0x7f6f4c15d340 2023-10-06T03:31:09.210 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:09.204+0000 7f6f4bfff700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f4c0c3220 con 0x7f6f4c0c9e00 2023-10-06T03:31:09.210 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:09.204+0000 7f6f4bfff700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f4c0c3500 con 0x7f6f4c15cf70 2023-10-06T03:31:09.210 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:09.204+0000 7f6f4b7fe700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6f4c15d340 0x7f6f4c170e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:31:09.210 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:09.204+0000 7f6f4a7fc700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6f4c15cf70 0x7f6f4c160080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:31:09.210 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:09.204+0000 7f6f4b7fe700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6f4c15d340 0x7f6f4c170e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:38930/0 (socket says 172.21.15.110:38930) 2023-10-06T03:31:09.210 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:09.204+0000 7f6f4b7fe700 1 -- 172.21.15.110:0/1024261551 learned_addr learned my addr 172.21.15.110:0/1024261551 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:31:09.210 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:09.204+0000 7f6f4a7fc700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6f4c15cf70 0x7f6f4c160080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3301/0 says I am v2:172.21.15.110:54990/0 (socket says 172.21.15.110:54990) 2023-10-06T03:31:09.210 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:09.204+0000 7f6f4affd700 1 --2- 172.21.15.110:0/1024261551 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6f4c0c9e00 0x7f6f4c0c29d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:31:09.211 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:09.204+0000 7f6f4b7fe700 1 -- 172.21.15.110:0/1024261551 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6f4c15cf70 msgr2=0x7f6f4c160080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:31:09.211 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:09.204+0000 7f6f4b7fe700 1 --2- 172.21.15.110:0/1024261551 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6f4c15cf70 0x7f6f4c160080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:09.211 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:09.204+0000 7f6f4b7fe700 1 -- 172.21.15.110:0/1024261551 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6f4c0c9e00 msgr2=0x7f6f4c0c29d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:31:09.211 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:09.204+0000 7f6f4b7fe700 1 --2- 172.21.15.110:0/1024261551 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6f4c0c9e00 0x7f6f4c0c29d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:09.212 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:09.204+0000 7f6f4b7fe700 1 -- 172.21.15.110:0/1024261551 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f34009a20 con 0x7f6f4c15d340 2023-10-06T03:31:09.212 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:09.204+0000 7f6f4b7fe700 1 --2- 172.21.15.110:0/1024261551 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6f4c15d340 0x7f6f4c170e90 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f6f3c00c7e0 tx=0x7f6f3c007d90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:31:09.212 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:09.204+0000 7f6f42ffd700 1 -- 172.21.15.110:0/1024261551 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f3c005c00 con 0x7f6f4c15d340 2023-10-06T03:31:09.212 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:09.204+0000 7f6f42ffd700 1 -- 172.21.15.110:0/1024261551 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f3c005d80 con 0x7f6f4c15d340 2023-10-06T03:31:09.212 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:09.204+0000 7f6f4bfff700 1 -- 172.21.15.110:0/1024261551 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f4c0c37e0 con 0x7f6f4c15d340 2023-10-06T03:31:09.212 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:09.204+0000 7f6f42ffd700 1 -- 172.21.15.110:0/1024261551 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f3c007320 con 0x7f6f4c15d340 2023-10-06T03:31:09.212 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:09.204+0000 7f6f4bfff700 1 -- 172.21.15.110:0/1024261551 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f4c0c2f10 con 0x7f6f4c15d340 2023-10-06T03:31:09.213 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:09.208+0000 7f6f4bfff700 1 -- 172.21.15.110:0/1024261551 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f4c064770 con 0x7f6f4c15d340 2023-10-06T03:31:09.213 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:09.208+0000 7f6f42ffd700 1 -- 172.21.15.110:0/1024261551 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 13) v1 ==== 80061+0+0 (secure 0 0 0) 0x7f6f3c01b3b0 con 0x7f6f4c15d340 2023-10-06T03:31:09.213 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:09.208+0000 7f6f42ffd700 1 --2- 172.21.15.110:0/1024261551 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f6f30060670 0x7f6f30062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:31:09.214 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:09.208+0000 7f6f4affd700 1 --2- 172.21.15.110:0/1024261551 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f6f30060670 0x7f6f30062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:31:09.214 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:09.208+0000 7f6f42ffd700 1 -- 172.21.15.110:0/1024261551 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 6171+0+0 (secure 0 0 0) 0x7f6f3c07c0d0 con 0x7f6f4c15d340 2023-10-06T03:31:09.214 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:09.208+0000 7f6f4affd700 1 --2- 172.21.15.110:0/1024261551 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f6f30060670 0x7f6f30062b20 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f6f44009db0 tx=0x7f6f440091a0 comp rx=0 tx=0).ready entity=mgr.5505 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:31:09.218 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:09.212+0000 7f6f42ffd700 1 -- 172.21.15.110:0/1024261551 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f6f3c04c340 con 0x7f6f4c15d340 2023-10-06T03:31:09.381 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:09.376+0000 7f6f4bfff700 1 -- 172.21.15.110:0/1024261551 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6f4c06b3e0 con 0x7f6f4c15d340 2023-10-06T03:31:09.383 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:09.376+0000 7f6f42ffd700 1 -- 172.21.15.110:0/1024261551 <== mon.0 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v47) v1 ==== 74+0+14848 (secure 0 0 0) 0x7f6f3c04f960 con 0x7f6f4c15d340 2023-10-06T03:31:09.383 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-10-06T03:31:09.384 INFO:teuthology.orchestra.run.smithi110.stdout:{"epoch":47,"fsid":"93b8f6ac-4c92-4b45-b8e4-b59617024aa2","created":"2023-10-06T03:29:10.519583+0000","modified":"2023-10-06T03:31:05.388828+0000","last_up_change":"2023-10-06T03:29:21.960511+0000","last_in_change":"2023-10-06T03:29:12.560436+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-06T03:29:20.703743+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2023-10-06T03:29:28.307281+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"5866eb6c-2adc-46e4-be7a-e5bcda0f4065","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6824","nonce":33394},{"type":"v1","addr":"172.21.15.110:6825","nonce":33394}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6826","nonce":33394},{"type":"v1","addr":"172.21.15.110:6827","nonce":33394}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6830","nonce":33394},{"type":"v1","addr":"172.21.15.110:6831","nonce":33394}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6828","nonce":33394},{"type":"v1","addr":"172.21.15.110:6829","nonce":33394}]},"public_addr":"172.21.15.110:6825/33394","cluster_addr":"172.21.15.110:6827/33394","heartbeat_back_addr":"172.21.15.110:6831/33394","heartbeat_front_addr":"172.21.15.110:6829/33394","state":["exists","up"]},{"osd":1,"uuid":"a38de118-062b-4a05-9782-2ceddf6ff4df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6800","nonce":33396},{"type":"v1","addr":"172.21.15.110:6801","nonce":33396}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6802","nonce":33396},{"type":"v1","addr":"172.21.15.110:6803","nonce":33396}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6806","nonce":33396},{"type":"v1","addr":"172.21.15.110:6807","nonce":33396}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6804","nonce":33396},{"type":"v1","addr":"172.21.15.110:6805","nonce":33396}]},"public_addr":"172.21.15.110:6801/33396","cluster_addr":"172.21.15.110:6803/33396","heartbeat_back_addr":"172.21.15.110:6807/33396","heartbeat_front_addr":"172.21.15.110:6805/33396","state":["exists","up"]},{"osd":2,"uuid":"56e39be2-3407-4b69-877e-cf16617b06d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6816","nonce":33397},{"type":"v1","addr":"172.21.15.110:6817","nonce":33397}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6818","nonce":33397},{"type":"v1","addr":"172.21.15.110:6819","nonce":33397}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6822","nonce":33397},{"type":"v1","addr":"172.21.15.110:6823","nonce":33397}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6820","nonce":33397},{"type":"v1","addr":"172.21.15.110:6821","nonce":33397}]},"public_addr":"172.21.15.110:6817/33397","cluster_addr":"172.21.15.110:6819/33397","heartbeat_back_addr":"172.21.15.110:6823/33397","heartbeat_front_addr":"172.21.15.110:6821/33397","state":["exists","up"]},{"osd":3,"uuid":"01c82d13-c1d4-4bde-b55e-3e0089b40115","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6808","nonce":33395},{"type":"v1","addr":"172.21.15.110:6809","nonce":33395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6810","nonce":33395},{"type":"v1","addr":"172.21.15.110:6811","nonce":33395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6814","nonce":33395},{"type":"v1","addr":"172.21.15.110:6815","nonce":33395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6812","nonce":33395},{"type":"v1","addr":"172.21.15.110:6813","nonce":33395}]},"public_addr":"172.21.15.110:6809/33395","cluster_addr":"172.21.15.110:6811/33395","heartbeat_back_addr":"172.21.15.110:6815/33395","heartbeat_front_addr":"172.21.15.110:6813/33395","state":["exists","up"]},{"osd":4,"uuid":"86983430-5fa3-4d17-998f-03e2cd843237","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":33683},{"type":"v1","addr":"172.21.15.192:6817","nonce":33683}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":33683},{"type":"v1","addr":"172.21.15.192:6819","nonce":33683}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":33683},{"type":"v1","addr":"172.21.15.192:6823","nonce":33683}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6820","nonce":33683},{"type":"v1","addr":"172.21.15.192:6821","nonce":33683}]},"public_addr":"172.21.15.192:6817/33683","cluster_addr":"172.21.15.192:6819/33683","heartbeat_back_addr":"172.21.15.192:6823/33683","heartbeat_front_addr":"172.21.15.192:6821/33683","state":["exists","up"]},{"osd":5,"uuid":"6945257f-821d-4619-a63f-ebe857eba6f4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":33684},{"type":"v1","addr":"172.21.15.192:6801","nonce":33684}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":33684},{"type":"v1","addr":"172.21.15.192:6803","nonce":33684}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":33684},{"type":"v1","addr":"172.21.15.192:6807","nonce":33684}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6804","nonce":33684},{"type":"v1","addr":"172.21.15.192:6805","nonce":33684}]},"public_addr":"172.21.15.192:6801/33684","cluster_addr":"172.21.15.192:6803/33684","heartbeat_back_addr":"172.21.15.192:6807/33684","heartbeat_front_addr":"172.21.15.192:6805/33684","state":["exists","up"]},{"osd":6,"uuid":"7854d1ae-7060-4780-8f97-73cea5e7a12f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6824","nonce":33686},{"type":"v1","addr":"172.21.15.192:6825","nonce":33686}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":33686},{"type":"v1","addr":"172.21.15.192:6827","nonce":33686}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":33686},{"type":"v1","addr":"172.21.15.192:6831","nonce":33686}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6828","nonce":33686},{"type":"v1","addr":"172.21.15.192:6829","nonce":33686}]},"public_addr":"172.21.15.192:6825/33686","cluster_addr":"172.21.15.192:6827/33686","heartbeat_back_addr":"172.21.15.192:6831/33686","heartbeat_front_addr":"172.21.15.192:6829/33686","state":["exists","up"]},{"osd":7,"uuid":"f1ebdfd5-b859-482b-9f97-c18c4a26f445","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6808","nonce":33685},{"type":"v1","addr":"172.21.15.192:6809","nonce":33685}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":33685},{"type":"v1","addr":"172.21.15.192:6811","nonce":33685}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":33685},{"type":"v1","addr":"172.21.15.192:6815","nonce":33685}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6812","nonce":33685},{"type":"v1","addr":"172.21.15.192:6813","nonce":33685}]},"public_addr":"172.21.15.192:6809/33685","cluster_addr":"172.21.15.192:6811/33685","heartbeat_back_addr":"172.21.15.192:6815/33685","heartbeat_front_addr":"172.21.15.192:6813/33685","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T03:29:18.002048+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T03:29:17.478576+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T03:29:18.034944+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T03:29:17.628513+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T03:29:18.047338+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T03:29:17.772349+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T03:29:17.883958+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T03:29:18.151838+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.110:6836/1379510962":"2023-10-07T03:31:02.317023+0000","172.21.15.110:0/3461792981":"2023-10-07T03:31:01.923482+0000","172.21.15.110:0/1315925152":"2023-10-07T03:31:01.923482+0000","172.21.15.110:0/3495425172":"2023-10-07T03:31:01.923482+0000","172.21.15.110:0/3149105774":"2023-10-07T03:31:01.923482+0000","172.21.15.110:0/3193317457":"2023-10-07T03:31:01.923482+0000","172.21.15.110:0/2107091982":"2023-10-07T03:31:01.923482+0000","172.21.15.110:0/2944476429":"2023-10-07T03:31:01.923482+0000","172.21.15.110:0/1857995858":"2023-10-07T03:30:46.697197+0000","172.21.15.110:0/1318624357":"2023-10-07T03:30:46.697197+0000","172.21.15.110:0/1139019390":"2023-10-07T03:30:46.697197+0000","172.21.15.110:6837/1379510962":"2023-10-07T03:31:02.317023+0000","172.21.15.110:6833/33268":"2023-10-07T03:30:46.697197+0000","172.21.15.110:0/463196587":"2023-10-07T03:30:46.697197+0000","172.21.15.110:6832/33268":"2023-10-07T03:30:46.697197+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-06T03:31:09.385 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:09.380+0000 7f6f4bfff700 1 -- 172.21.15.110:0/1024261551 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f6f30060670 msgr2=0x7f6f30062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:31:09.386 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:09.380+0000 7f6f4bfff700 1 --2- 172.21.15.110:0/1024261551 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f6f30060670 0x7f6f30062b20 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f6f44009db0 tx=0x7f6f440091a0 comp rx=0 tx=0).stop 2023-10-06T03:31:09.386 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:09.380+0000 7f6f4bfff700 1 -- 172.21.15.110:0/1024261551 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6f4c15d340 msgr2=0x7f6f4c170e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:31:09.386 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:09.380+0000 7f6f4bfff700 1 --2- 172.21.15.110:0/1024261551 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6f4c15d340 0x7f6f4c170e90 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f6f3c00c7e0 tx=0x7f6f3c007d90 comp rx=0 tx=0).stop 2023-10-06T03:31:09.386 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:09.380+0000 7f6f4bfff700 1 -- 172.21.15.110:0/1024261551 shutdown_connections 2023-10-06T03:31:09.386 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:09.380+0000 7f6f4bfff700 1 --2- 172.21.15.110:0/1024261551 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f6f30060670 0x7f6f30062b20 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:09.386 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:09.380+0000 7f6f4bfff700 1 --2- 172.21.15.110:0/1024261551 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6f4c15d340 0x7f6f4c170e90 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:09.386 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:09.380+0000 7f6f4bfff700 1 --2- 172.21.15.110:0/1024261551 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f6f4c15cf70 0x7f6f4c160080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:09.386 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:09.380+0000 7f6f4bfff700 1 --2- 172.21.15.110:0/1024261551 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6f4c0c9e00 0x7f6f4c0c29d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:09.386 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:09.380+0000 7f6f4bfff700 1 -- 172.21.15.110:0/1024261551 >> 172.21.15.110:0/1024261551 conn(0x7f6f4c06c760 msgr2=0x7f6f4c0c87d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:31:09.386 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:09.380+0000 7f6f4bfff700 1 -- 172.21.15.110:0/1024261551 shutdown_connections 2023-10-06T03:31:09.387 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:09.380+0000 7f6f4bfff700 1 -- 172.21.15.110:0/1024261551 wait complete. 2023-10-06T03:31:10.399 INFO:tasks.ceph:Scrubbing osd.0 2023-10-06T03:31:10.399 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2023-10-06T03:31:10.523 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:10.516+0000 7f76dcee7700 1 Processor -- start 2023-10-06T03:31:10.523 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:10.516+0000 7f76dcee7700 1 -- start start 2023-10-06T03:31:10.525 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:10.520+0000 7f76dcee7700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f76d8158cd0 0x7f76d81590a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:31:10.525 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:10.520+0000 7f76dcee7700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f76d80ce8e0 0x7f76d80aff10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:31:10.525 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:10.520+0000 7f76dcee7700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76d80cab30 con 0x7f76d80ce490 2023-10-06T03:31:10.526 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:10.520+0000 7f76dcee7700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f76d80cef30 con 0x7f76d8158cd0 2023-10-06T03:31:10.526 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:10.520+0000 7f76dcee7700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76d80cf210 con 0x7f76d80ce8e0 2023-10-06T03:31:10.526 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:10.520+0000 7f76d5d9b700 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f76d80ce490 0x7f76d80cafc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:60386/0 (socket says 172.21.15.110:60386) 2023-10-06T03:31:10.526 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:10.520+0000 7f76d5d9b700 1 -- 172.21.15.110:0/3971595328 learned_addr learned my addr 172.21.15.110:0/3971595328 (peer_addr_for_me v1:172.21.15.110:0/0) 2023-10-06T03:31:10.526 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:10.520+0000 7f76d659c700 1 --2- 172.21.15.110:0/3971595328 >> v2:172.21.15.192:3300/0 conn(0x7f76d8158cd0 0x7f76d81590a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:31:10.526 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:10.520+0000 7f76d6d9d700 1 --2- 172.21.15.110:0/3971595328 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f76d80ce8e0 0x7f76d80aff10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:31:10.526 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:10.520+0000 7f76d559a700 1 -- 172.21.15.110:0/3971595328 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3171712127 0 0) 0x7f76d80cab30 con 0x7f76d80ce490 2023-10-06T03:31:10.526 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:10.520+0000 7f76d559a700 1 -- 172.21.15.110:0/3971595328 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f76c4003570 con 0x7f76d80ce490 2023-10-06T03:31:10.528 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:10.520+0000 7f76d659c700 1 -- 172.21.15.110:0/3971595328 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f76d80ce8e0 msgr2=0x7f76d80aff10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:31:10.528 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:10.520+0000 7f76d659c700 1 --2- 172.21.15.110:0/3971595328 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f76d80ce8e0 0x7f76d80aff10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:10.528 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:10.520+0000 7f76d659c700 1 -- 172.21.15.110:0/3971595328 >> v1:172.21.15.110:6789/0 conn(0x7f76d80ce490 legacy=0x7f76d80cafc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:31:10.528 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:10.520+0000 7f76d559a700 1 -- 172.21.15.110:0/3971595328 <== mon.0 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1736085024 0 0) 0x7f76c4003570 con 0x7f76d80ce490 2023-10-06T03:31:10.528 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:10.520+0000 7f76d659c700 1 -- 172.21.15.110:0/3971595328 --> v2:172.21.15.192:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76d80cf4f0 con 0x7f76d8158cd0 2023-10-06T03:31:10.528 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:10.520+0000 7f76d6d9d700 1 --2- 172.21.15.110:0/3971595328 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f76d80ce8e0 0x7f76d80aff10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-06T03:31:10.528 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:10.520+0000 7f76d659c700 1 --2- 172.21.15.110:0/3971595328 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f76d8158cd0 0x7f76d81590a0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f76c80098d0 tx=0x7f76c8009120 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c229404b3bd64e54 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:31:10.528 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:10.520+0000 7f76d559a700 1 -- 172.21.15.110:0/3971595328 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76c8005c10 con 0x7f76d8158cd0 2023-10-06T03:31:10.529 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:10.520+0000 7f76d559a700 1 -- 172.21.15.110:0/3971595328 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f76c8005d70 con 0x7f76d8158cd0 2023-10-06T03:31:10.529 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:10.520+0000 7f76d559a700 1 -- 172.21.15.110:0/3971595328 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76c8006260 con 0x7f76d8158cd0 2023-10-06T03:31:10.529 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:10.524+0000 7f76dcee7700 1 -- 172.21.15.110:0/3971595328 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f76d8158cd0 msgr2=0x7f76d81590a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:31:10.529 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:10.524+0000 7f76dcee7700 1 --2- 172.21.15.110:0/3971595328 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f76d8158cd0 0x7f76d81590a0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f76c80098d0 tx=0x7f76c8009120 comp rx=0 tx=0).stop 2023-10-06T03:31:10.529 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:10.524+0000 7f76dcee7700 1 -- 172.21.15.110:0/3971595328 shutdown_connections 2023-10-06T03:31:10.529 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:10.524+0000 7f76dcee7700 1 --2- 172.21.15.110:0/3971595328 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f76d80ce8e0 0x7f76d80aff10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:10.529 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:10.524+0000 7f76dcee7700 1 --2- 172.21.15.110:0/3971595328 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f76d8158cd0 0x7f76d81590a0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:10.529 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:10.524+0000 7f76dcee7700 1 -- 172.21.15.110:0/3971595328 >> 172.21.15.110:0/3971595328 conn(0x7f76d8067040 msgr2=0x7f76d8067440 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:31:10.529 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:10.524+0000 7f76dcee7700 1 -- 172.21.15.110:0/3971595328 shutdown_connections 2023-10-06T03:31:10.529 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:10.524+0000 7f76dcee7700 1 -- 172.21.15.110:0/3971595328 wait complete. 2023-10-06T03:31:10.531 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:10.524+0000 7f76dcee7700 1 Processor -- start 2023-10-06T03:31:10.531 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:10.524+0000 7f76dcee7700 1 -- start start 2023-10-06T03:31:10.533 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:10.528+0000 7f76dcee7700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f76d80ce490 0x7f76d80b5760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:31:10.534 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:10.528+0000 7f76dcee7700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f76d80ce8e0 0x7f76d816cde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:31:10.534 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:10.528+0000 7f76dcee7700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f76d8158cd0 0x7f76d80c6080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:31:10.534 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:10.528+0000 7f76dcee7700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76d80b2d60 con 0x7f76d80ce8e0 2023-10-06T03:31:10.534 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:10.528+0000 7f76dcee7700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76d80b5ca0 con 0x7f76d8158cd0 2023-10-06T03:31:10.534 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:10.528+0000 7f76dcee7700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76d814a7d0 con 0x7f76d80ce490 2023-10-06T03:31:10.534 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:10.528+0000 7f76d5d9b700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f76d80ce8e0 0x7f76d816cde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:31:10.534 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:10.528+0000 7f76d5d9b700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f76d80ce8e0 0x7f76d816cde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:38934/0 (socket says 172.21.15.110:38934) 2023-10-06T03:31:10.534 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:10.528+0000 7f76d5d9b700 1 -- 172.21.15.110:0/3047939399 learned_addr learned my addr 172.21.15.110:0/3047939399 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:31:10.534 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:10.528+0000 7f76d6d9d700 1 --2- 172.21.15.110:0/3047939399 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f76d8158cd0 0x7f76d80c6080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:31:10.534 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:10.528+0000 7f76d659c700 1 --2- 172.21.15.110:0/3047939399 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f76d80ce490 0x7f76d80b5760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:31:10.535 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:10.528+0000 7f76d5d9b700 1 -- 172.21.15.110:0/3047939399 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f76d80ce490 msgr2=0x7f76d80b5760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:31:10.536 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:10.528+0000 7f76d5d9b700 1 --2- 172.21.15.110:0/3047939399 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f76d80ce490 0x7f76d80b5760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:10.536 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:10.528+0000 7f76d5d9b700 1 -- 172.21.15.110:0/3047939399 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f76d8158cd0 msgr2=0x7f76d80c6080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:31:10.536 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:10.528+0000 7f76d5d9b700 1 --2- 172.21.15.110:0/3047939399 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f76d8158cd0 0x7f76d80c6080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:10.536 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:10.528+0000 7f76d5d9b700 1 -- 172.21.15.110:0/3047939399 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76d0009a20 con 0x7f76d80ce8e0 2023-10-06T03:31:10.536 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:10.528+0000 7f76d5d9b700 1 --2- 172.21.15.110:0/3047939399 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f76d80ce8e0 0x7f76d816cde0 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f76c000c810 tx=0x7f76c0007bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:31:10.536 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:10.528+0000 7f76ce7fc700 1 -- 172.21.15.110:0/3047939399 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76c0007360 con 0x7f76d80ce8e0 2023-10-06T03:31:10.536 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:10.528+0000 7f76ce7fc700 1 -- 172.21.15.110:0/3047939399 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f76c00074e0 con 0x7f76d80ce8e0 2023-10-06T03:31:10.536 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:10.528+0000 7f76dcee7700 1 -- 172.21.15.110:0/3047939399 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76c80093c0 con 0x7f76d80ce8e0 2023-10-06T03:31:10.536 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:10.528+0000 7f76ce7fc700 1 -- 172.21.15.110:0/3047939399 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76c0004830 con 0x7f76d80ce8e0 2023-10-06T03:31:10.536 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:10.528+0000 7f76dcee7700 1 -- 172.21.15.110:0/3047939399 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76d814ad90 con 0x7f76d80ce8e0 2023-10-06T03:31:10.537 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:10.532+0000 7f76dcee7700 1 -- 172.21.15.110:0/3047939399 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f76d8064770 con 0x7f76d80ce8e0 2023-10-06T03:31:10.537 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:10.532+0000 7f76ce7fc700 1 -- 172.21.15.110:0/3047939399 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 13) v1 ==== 80061+0+0 (secure 0 0 0) 0x7f76c0004be0 con 0x7f76d80ce8e0 2023-10-06T03:31:10.538 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:10.532+0000 7f76ce7fc700 1 --2- 172.21.15.110:0/3047939399 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f76b8060670 0x7f76b8062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:31:10.538 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:10.532+0000 7f76d659c700 1 --2- 172.21.15.110:0/3047939399 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f76b8060670 0x7f76b8062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:31:10.538 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:10.532+0000 7f76ce7fc700 1 -- 172.21.15.110:0/3047939399 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 6171+0+0 (secure 0 0 0) 0x7f76c007c850 con 0x7f76d80ce8e0 2023-10-06T03:31:10.538 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:10.532+0000 7f76ce7fc700 1 --2- 172.21.15.110:0/3047939399 >> [v2:172.21.15.110:6824/33394,v1:172.21.15.110:6825/33394] conn(0x7f76b8067500 0x7f76b8069930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:31:10.538 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:10.532+0000 7f76ce7fc700 1 -- 172.21.15.110:0/3047939399 --> [v2:172.21.15.110:6824/33394,v1:172.21.15.110:6825/33394] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f76b806a000 con 0x7f76b8067500 2023-10-06T03:31:10.538 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:10.532+0000 7f76ce7fc700 1 -- 172.21.15.110:0/3047939399 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_get_version_reply(handle=1 version=47) v2 ==== 24+0+0 (secure 0 0 0) 0x7f76c0081240 con 0x7f76d80ce8e0 2023-10-06T03:31:10.539 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:10.532+0000 7f76d6d9d700 1 --2- 172.21.15.110:0/3047939399 >> [v2:172.21.15.110:6824/33394,v1:172.21.15.110:6825/33394] conn(0x7f76b8067500 0x7f76b8069930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:31:10.539 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:10.532+0000 7f76d659c700 1 --2- 172.21.15.110:0/3047939399 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f76b8060670 0x7f76b8062b20 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f76c8009160 tx=0x7f76c8016000 comp rx=0 tx=0).ready entity=mgr.5505 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:31:10.539 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:10.532+0000 7f76d6d9d700 1 --2- 172.21.15.110:0/3047939399 >> [v2:172.21.15.110:6824/33394,v1:172.21.15.110:6825/33394] conn(0x7f76b8067500 0x7f76b8069930 crc :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:31:10.540 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:10.536+0000 7f76ce7fc700 1 -- 172.21.15.110:0/3047939399 <== osd.0 v2:172.21.15.110:6824/33394 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23999 (crc 0 0 0) 0x7f76b806a000 con 0x7f76b8067500 2023-10-06T03:31:10.558 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:10.552+0000 7f76dcee7700 1 -- 172.21.15.110:0/3047939399 --> [v2:172.21.15.110:6824/33394,v1:172.21.15.110:6825/33394] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f76d80cdfb0 con 0x7f76b8067500 2023-10-06T03:31:10.559 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:10.552+0000 7f76ce7fc700 1 -- 172.21.15.110:0/3047939399 <== osd.0 v2:172.21.15.110:6824/33394 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7f76d80cdfb0 con 0x7f76b8067500 2023-10-06T03:31:10.560 INFO:teuthology.orchestra.run.smithi110.stdout:{ 2023-10-06T03:31:10.560 INFO:teuthology.orchestra.run.smithi110.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-10-06T03:31:10.560 INFO:teuthology.orchestra.run.smithi110.stdout:} 2023-10-06T03:31:10.561 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:10.552+0000 7f76dcee7700 1 -- 172.21.15.110:0/3047939399 >> [v2:172.21.15.110:6824/33394,v1:172.21.15.110:6825/33394] conn(0x7f76b8067500 msgr2=0x7f76b8069930 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:31:10.561 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:10.552+0000 7f76dcee7700 1 --2- 172.21.15.110:0/3047939399 >> [v2:172.21.15.110:6824/33394,v1:172.21.15.110:6825/33394] conn(0x7f76b8067500 0x7f76b8069930 crc :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:10.562 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:10.552+0000 7f76dcee7700 1 -- 172.21.15.110:0/3047939399 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f76b8060670 msgr2=0x7f76b8062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:31:10.562 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:10.552+0000 7f76dcee7700 1 --2- 172.21.15.110:0/3047939399 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f76b8060670 0x7f76b8062b20 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f76c8009160 tx=0x7f76c8016000 comp rx=0 tx=0).stop 2023-10-06T03:31:10.562 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:10.552+0000 7f76dcee7700 1 -- 172.21.15.110:0/3047939399 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f76d80ce8e0 msgr2=0x7f76d816cde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:31:10.562 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:10.552+0000 7f76dcee7700 1 --2- 172.21.15.110:0/3047939399 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f76d80ce8e0 0x7f76d816cde0 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f76c000c810 tx=0x7f76c0007bb0 comp rx=0 tx=0).stop 2023-10-06T03:31:10.562 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:10.552+0000 7f76d6d9d700 1 -- 172.21.15.110:0/3047939399 reap_dead start 2023-10-06T03:31:10.562 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:10.552+0000 7f76dcee7700 1 -- 172.21.15.110:0/3047939399 shutdown_connections 2023-10-06T03:31:10.562 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:10.552+0000 7f76dcee7700 1 -- 172.21.15.110:0/3047939399 >> 172.21.15.110:0/3047939399 conn(0x7f76d8067040 msgr2=0x7f76d8152c90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:31:10.563 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:10.556+0000 7f76dcee7700 1 -- 172.21.15.110:0/3047939399 shutdown_connections 2023-10-06T03:31:10.563 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:10.556+0000 7f76dcee7700 1 -- 172.21.15.110:0/3047939399 wait complete. 2023-10-06T03:31:10.572 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2023-10-06T03:31:10.683 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:10.676+0000 7f3cf96a3700 1 Processor -- start 2023-10-06T03:31:10.683 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:10.676+0000 7f3cf96a3700 1 -- start start 2023-10-06T03:31:10.685 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:10.680+0000 7f3cf96a3700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f3cf40c86d0 0x7f3cf40c8aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:31:10.685 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:10.680+0000 7f3cf96a3700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f3cf40c6af0 0x7f3cf4158a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:31:10.685 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:10.680+0000 7f3cf96a3700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3cf406a4e0 con 0x7f3cf40cc5a0 2023-10-06T03:31:10.685 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:10.680+0000 7f3cf96a3700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3cf406ad80 con 0x7f3cf40c86d0 2023-10-06T03:31:10.685 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:10.680+0000 7f3cf96a3700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3cf40c73b0 con 0x7f3cf40c6af0 2023-10-06T03:31:10.685 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:10.680+0000 7f3cf2ffd700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f3cf40c86d0 0x7f3cf40c8aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:31:10.685 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:10.680+0000 7f3cf2ffd700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f3cf40c86d0 0x7f3cf40c8aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:38946/0 (socket says 172.21.15.110:38946) 2023-10-06T03:31:10.685 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:10.680+0000 7f3cf2ffd700 1 -- 172.21.15.110:0/1073533195 learned_addr learned my addr 172.21.15.110:0/1073533195 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:31:10.685 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:10.680+0000 7f3cf37fe700 1 --2- 172.21.15.110:0/1073533195 >> v2:172.21.15.192:3300/0 conn(0x7f3cf40c6af0 0x7f3cf4158a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:31:10.686 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:10.680+0000 7f3cf2ffd700 1 -- 172.21.15.110:0/1073533195 >> v2:172.21.15.192:3300/0 conn(0x7f3cf40c6af0 msgr2=0x7f3cf4158a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:31:10.686 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:10.680+0000 7f3cf2ffd700 1 --2- 172.21.15.110:0/1073533195 >> v2:172.21.15.192:3300/0 conn(0x7f3cf40c6af0 0x7f3cf4158a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:10.686 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:10.680+0000 7f3cf2ffd700 1 -- 172.21.15.110:0/1073533195 >> v1:172.21.15.192:6789/0 conn(0x7f3cf40cc5a0 legacy=0x7f3cf406b210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:31:10.687 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:10.680+0000 7f3cf2ffd700 1 -- 172.21.15.110:0/1073533195 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3cf406a730 con 0x7f3cf40c86d0 2023-10-06T03:31:10.687 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:10.680+0000 7f3cf2ffd700 1 --2- 172.21.15.110:0/1073533195 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3cf40c86d0 0x7f3cf40c8aa0 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f3cf4068680 tx=0x7f3ce8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=11cf1c6121642af5 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:31:10.687 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:10.680+0000 7f3cf1ffb700 1 -- 172.21.15.110:0/1073533195 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ce8009e70 con 0x7f3cf40c86d0 2023-10-06T03:31:10.687 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:10.680+0000 7f3cf1ffb700 1 -- 172.21.15.110:0/1073533195 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ce800b860 con 0x7f3cf40c86d0 2023-10-06T03:31:10.687 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:10.680+0000 7f3cf1ffb700 1 -- 172.21.15.110:0/1073533195 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ce800bc20 con 0x7f3cf40c86d0 2023-10-06T03:31:10.688 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:10.680+0000 7f3cf96a3700 1 -- 172.21.15.110:0/1073533195 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3cf40c86d0 msgr2=0x7f3cf40c8aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:31:10.688 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:10.680+0000 7f3cf96a3700 1 --2- 172.21.15.110:0/1073533195 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3cf40c86d0 0x7f3cf40c8aa0 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f3cf4068680 tx=0x7f3ce8009250 comp rx=0 tx=0).stop 2023-10-06T03:31:10.688 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:10.680+0000 7f3cf96a3700 1 -- 172.21.15.110:0/1073533195 shutdown_connections 2023-10-06T03:31:10.688 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:10.680+0000 7f3cf96a3700 1 --2- 172.21.15.110:0/1073533195 >> v2:172.21.15.192:3300/0 conn(0x7f3cf40c6af0 0x7f3cf4158a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:10.688 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:10.680+0000 7f3cf96a3700 1 --2- 172.21.15.110:0/1073533195 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3cf40c86d0 0x7f3cf40c8aa0 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:10.688 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:10.680+0000 7f3cf96a3700 1 -- 172.21.15.110:0/1073533195 >> 172.21.15.110:0/1073533195 conn(0x7f3cf4067040 msgr2=0x7f3cf4067410 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:31:10.688 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:10.680+0000 7f3cf96a3700 1 -- 172.21.15.110:0/1073533195 shutdown_connections 2023-10-06T03:31:10.688 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:10.684+0000 7f3cf96a3700 1 -- 172.21.15.110:0/1073533195 wait complete. 2023-10-06T03:31:10.689 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:10.684+0000 7f3cf96a3700 1 Processor -- start 2023-10-06T03:31:10.690 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:10.684+0000 7f3cf96a3700 1 -- start start 2023-10-06T03:31:10.690 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:10.684+0000 7f3cf96a3700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3cf40c6af0 0x7f3cf4154730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:31:10.690 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:10.684+0000 7f3cf96a3700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3cf40c86d0 0x7f3cf4157840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:31:10.690 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:10.684+0000 7f3cf96a3700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3cf40cc5a0 0x7f3cf4157d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:31:10.690 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:10.684+0000 7f3cf96a3700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cf40ad2b0 con 0x7f3cf40cc5a0 2023-10-06T03:31:10.690 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:10.684+0000 7f3cf96a3700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cf40ad590 con 0x7f3cf40c6af0 2023-10-06T03:31:10.690 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:10.684+0000 7f3cf96a3700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3cf40ad870 con 0x7f3cf40c86d0 2023-10-06T03:31:10.690 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:10.684+0000 7f3cf37fe700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3cf40cc5a0 0x7f3cf4157d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:31:10.690 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:10.684+0000 7f3cf37fe700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3cf40cc5a0 0x7f3cf4157d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:38962/0 (socket says 172.21.15.110:38962) 2023-10-06T03:31:10.691 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:10.684+0000 7f3cf37fe700 1 -- 172.21.15.110:0/3914806390 learned_addr learned my addr 172.21.15.110:0/3914806390 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:31:10.691 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:10.684+0000 7f3cf2ffd700 1 --2- 172.21.15.110:0/3914806390 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3cf40c6af0 0x7f3cf4154730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:31:10.691 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:10.684+0000 7f3cf27fc700 1 --2- 172.21.15.110:0/3914806390 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3cf40c86d0 0x7f3cf4157840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:31:10.692 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:10.684+0000 7f3cf37fe700 1 -- 172.21.15.110:0/3914806390 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3cf40c86d0 msgr2=0x7f3cf4157840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:31:10.692 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:10.684+0000 7f3cf37fe700 1 --2- 172.21.15.110:0/3914806390 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3cf40c86d0 0x7f3cf4157840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:10.693 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:10.684+0000 7f3cf37fe700 1 -- 172.21.15.110:0/3914806390 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3cf40c6af0 msgr2=0x7f3cf4154730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:31:10.693 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:10.684+0000 7f3cf37fe700 1 --2- 172.21.15.110:0/3914806390 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3cf40c6af0 0x7f3cf4154730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:10.693 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:10.684+0000 7f3cf37fe700 1 -- 172.21.15.110:0/3914806390 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ce80094f0 con 0x7f3cf40cc5a0 2023-10-06T03:31:10.693 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:10.684+0000 7f3cf37fe700 1 --2- 172.21.15.110:0/3914806390 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3cf40cc5a0 0x7f3cf4157d80 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f3cec00e8d0 tx=0x7f3cec0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:31:10.693 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:10.684+0000 7f3cdaffd700 1 -- 172.21.15.110:0/3914806390 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cec013680 con 0x7f3cf40cc5a0 2023-10-06T03:31:10.693 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:10.684+0000 7f3cdaffd700 1 -- 172.21.15.110:0/3914806390 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3cec0137e0 con 0x7f3cf40cc5a0 2023-10-06T03:31:10.693 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:10.684+0000 7f3cdaffd700 1 -- 172.21.15.110:0/3914806390 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cec013ba0 con 0x7f3cf40cc5a0 2023-10-06T03:31:10.693 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:10.684+0000 7f3cf96a3700 1 -- 172.21.15.110:0/3914806390 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3cf40adb50 con 0x7f3cf40cc5a0 2023-10-06T03:31:10.693 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:10.684+0000 7f3cf96a3700 1 -- 172.21.15.110:0/3914806390 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3cf4154e90 con 0x7f3cf40cc5a0 2023-10-06T03:31:10.694 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:10.688+0000 7f3cf96a3700 1 -- 172.21.15.110:0/3914806390 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3cf4064770 con 0x7f3cf40cc5a0 2023-10-06T03:31:10.694 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:10.688+0000 7f3cdaffd700 1 -- 172.21.15.110:0/3914806390 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 13) v1 ==== 80061+0+0 (secure 0 0 0) 0x7f3cec0253d0 con 0x7f3cf40cc5a0 2023-10-06T03:31:10.694 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:10.688+0000 7f3cdaffd700 1 --2- 172.21.15.110:0/3914806390 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f3ce0060670 0x7f3ce0062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:31:10.694 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:10.688+0000 7f3cf2ffd700 1 --2- 172.21.15.110:0/3914806390 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f3ce0060670 0x7f3ce0062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:31:10.695 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:10.688+0000 7f3cdaffd700 1 -- 172.21.15.110:0/3914806390 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 6171+0+0 (secure 0 0 0) 0x7f3cec07d420 con 0x7f3cf40cc5a0 2023-10-06T03:31:10.695 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:10.692+0000 7f3cf2ffd700 1 --2- 172.21.15.110:0/3914806390 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f3ce0060670 0x7f3ce0062b20 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f3cf4068680 tx=0x7f3ce8017000 comp rx=0 tx=0).ready entity=mgr.5505 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:31:10.698 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:10.692+0000 7f3cdaffd700 1 -- 172.21.15.110:0/3914806390 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f3cec04d690 con 0x7f3cf40cc5a0 2023-10-06T03:31:10.851 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:10.844+0000 7f3cf96a3700 1 -- 172.21.15.110:0/3914806390 --> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "0", "target": ["mon-mgr", ""]}) v1 -- 0x7f3cf406aa10 con 0x7f3ce0060670 2023-10-06T03:31:10.851 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:10.848+0000 7f3cdaffd700 1 -- 172.21.15.110:0/3914806390 <== mgr.5505 v2:172.21.15.110:6832/33268 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 0 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f3cf406aa10 con 0x7f3ce0060670 2023-10-06T03:31:10.854 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:10.848+0000 7f3cf96a3700 1 -- 172.21.15.110:0/3914806390 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f3ce0060670 msgr2=0x7f3ce0062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:31:10.854 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:10.848+0000 7f3cf96a3700 1 --2- 172.21.15.110:0/3914806390 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f3ce0060670 0x7f3ce0062b20 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f3cf4068680 tx=0x7f3ce8017000 comp rx=0 tx=0).stop 2023-10-06T03:31:10.854 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:10.848+0000 7f3cf96a3700 1 -- 172.21.15.110:0/3914806390 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3cf40cc5a0 msgr2=0x7f3cf4157d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:31:10.855 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:10.848+0000 7f3cf96a3700 1 --2- 172.21.15.110:0/3914806390 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3cf40cc5a0 0x7f3cf4157d80 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f3cec00e8d0 tx=0x7f3cec0091a0 comp rx=0 tx=0).stop 2023-10-06T03:31:10.855 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:10.848+0000 7f3cf96a3700 1 -- 172.21.15.110:0/3914806390 shutdown_connections 2023-10-06T03:31:10.855 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:10.848+0000 7f3cf96a3700 1 --2- 172.21.15.110:0/3914806390 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f3ce0060670 0x7f3ce0062b20 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:10.855 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:10.848+0000 7f3cf96a3700 1 --2- 172.21.15.110:0/3914806390 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3cf40cc5a0 0x7f3cf4157d80 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:10.855 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:10.848+0000 7f3cf96a3700 1 --2- 172.21.15.110:0/3914806390 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f3cf40c86d0 0x7f3cf4157840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:10.855 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:10.848+0000 7f3cf96a3700 1 --2- 172.21.15.110:0/3914806390 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3cf40c6af0 0x7f3cf4154730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:10.855 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:10.848+0000 7f3cf96a3700 1 -- 172.21.15.110:0/3914806390 >> 172.21.15.110:0/3914806390 conn(0x7f3cf4067040 msgr2=0x7f3cf40b0d70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:31:10.855 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:10.848+0000 7f3cf96a3700 1 -- 172.21.15.110:0/3914806390 shutdown_connections 2023-10-06T03:31:10.855 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:10.848+0000 7f3cf96a3700 1 -- 172.21.15.110:0/3914806390 wait complete. 2023-10-06T03:31:10.856 INFO:teuthology.orchestra.run.smithi110.stderr:instructed osd(s) 0 to deep-scrub 2023-10-06T03:31:10.865 INFO:tasks.ceph:Scrubbing osd.1 2023-10-06T03:31:10.866 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2023-10-06T03:31:10.978 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:10.972+0000 7f1cfb50e700 1 Processor -- start 2023-10-06T03:31:10.979 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:10.972+0000 7f1cfb50e700 1 -- start start 2023-10-06T03:31:10.981 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:10.976+0000 7f1cfb50e700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f1cf4158d60 0x7f1cf40afe80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:31:10.981 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:10.976+0000 7f1cfb50e700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f1cf4159130 0x7f1cf40cfdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:31:10.981 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:10.976+0000 7f1cfb50e700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1cf40d02f0 0x7f1cf40ca6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:31:10.981 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:10.976+0000 7f1cfb50e700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1cf40d0770 con 0x7f1cf4159130 2023-10-06T03:31:10.982 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:10.976+0000 7f1cfb50e700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1cf406a760 con 0x7f1cf4158d60 2023-10-06T03:31:10.982 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:10.976+0000 7f1cfb50e700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1cf406ad20 con 0x7f1cf40d02f0 2023-10-06T03:31:10.982 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:10.976+0000 7f1cf8aa9700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f1cf4159130 0x7f1cf40cfdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:31:10.982 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:10.976+0000 7f1cf8aa9700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f1cf4159130 0x7f1cf40cfdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:54608/0 (socket says 172.21.15.110:54608) 2023-10-06T03:31:10.982 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:10.976+0000 7f1cf8aa9700 1 -- 172.21.15.110:0/921469329 learned_addr learned my addr 172.21.15.110:0/921469329 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:31:10.983 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:10.976+0000 7f1cf9aab700 1 --2- 172.21.15.110:0/921469329 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1cf40d02f0 0x7f1cf40ca6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:31:10.983 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:10.976+0000 7f1cf92aa700 1 --2- 172.21.15.110:0/921469329 >> v2:172.21.15.192:3300/0 conn(0x7f1cf4158d60 0x7f1cf40afe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:31:10.983 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:10.976+0000 7f1cf8aa9700 1 -- 172.21.15.110:0/921469329 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1cf40d02f0 msgr2=0x7f1cf40ca6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:31:10.983 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:10.976+0000 7f1cf8aa9700 1 --2- 172.21.15.110:0/921469329 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1cf40d02f0 0x7f1cf40ca6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:10.983 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:10.976+0000 7f1cf8aa9700 1 -- 172.21.15.110:0/921469329 >> v2:172.21.15.192:3300/0 conn(0x7f1cf4158d60 msgr2=0x7f1cf40afe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:31:10.984 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:10.976+0000 7f1cf8aa9700 1 --2- 172.21.15.110:0/921469329 >> v2:172.21.15.192:3300/0 conn(0x7f1cf4158d60 0x7f1cf40afe80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:10.984 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:10.976+0000 7f1cf8aa9700 1 -- 172.21.15.110:0/921469329 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1cf40cb440 con 0x7f1cf4159130 2023-10-06T03:31:10.984 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:10.976+0000 7f1cf9aab700 1 --2- 172.21.15.110:0/921469329 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1cf40d02f0 0x7f1cf40ca6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-06T03:31:10.984 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:10.976+0000 7f1cf8aa9700 1 --2- 172.21.15.110:0/921469329 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1cf4159130 0x7f1cf40cfdb0 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f1cf40cd9d0 tx=0x7f1cf0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e3901fe288198c9 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:31:10.984 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:10.976+0000 7f1cebfff700 1 -- 172.21.15.110:0/921469329 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cf000b860 con 0x7f1cf4159130 2023-10-06T03:31:10.985 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:10.976+0000 7f1cebfff700 1 -- 172.21.15.110:0/921469329 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1cf000b9c0 con 0x7f1cf4159130 2023-10-06T03:31:10.985 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:10.976+0000 7f1cebfff700 1 -- 172.21.15.110:0/921469329 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cf000bd60 con 0x7f1cf4159130 2023-10-06T03:31:10.987 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:10.976+0000 7f1cfb50e700 1 -- 172.21.15.110:0/921469329 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1cf4159130 msgr2=0x7f1cf40cfdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:31:10.987 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:10.976+0000 7f1cfb50e700 1 --2- 172.21.15.110:0/921469329 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1cf4159130 0x7f1cf40cfdb0 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f1cf40cd9d0 tx=0x7f1cf0009250 comp rx=0 tx=0).stop 2023-10-06T03:31:10.988 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:10.976+0000 7f1cfb50e700 1 -- 172.21.15.110:0/921469329 shutdown_connections 2023-10-06T03:31:10.988 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:10.976+0000 7f1cfb50e700 1 --2- 172.21.15.110:0/921469329 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1cf40d02f0 0x7f1cf40ca6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:10.988 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:10.976+0000 7f1cfb50e700 1 --2- 172.21.15.110:0/921469329 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1cf4159130 0x7f1cf40cfdb0 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:10.988 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:10.976+0000 7f1cfb50e700 1 --2- 172.21.15.110:0/921469329 >> v2:172.21.15.192:3300/0 conn(0x7f1cf4158d60 0x7f1cf40afe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:10.988 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:10.976+0000 7f1cfb50e700 1 -- 172.21.15.110:0/921469329 >> 172.21.15.110:0/921469329 conn(0x7f1cf4067040 msgr2=0x7f1cf4067440 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:31:10.989 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:10.976+0000 7f1cfb50e700 1 -- 172.21.15.110:0/921469329 shutdown_connections 2023-10-06T03:31:10.989 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:10.976+0000 7f1cfb50e700 1 -- 172.21.15.110:0/921469329 wait complete. 2023-10-06T03:31:10.989 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:10.980+0000 7f1cfb50e700 1 Processor -- start 2023-10-06T03:31:10.989 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:10.980+0000 7f1cfb50e700 1 -- start start 2023-10-06T03:31:10.989 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:10.980+0000 7f1cfb50e700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1cf40d02f0 0x7f1cf416a310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:31:10.990 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:10.980+0000 7f1cfb50e700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1cf4158d60 0x7f1cf4152040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:31:10.990 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:10.980+0000 7f1cfb50e700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1cf4159130 0x7f1cf40ced80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:31:10.990 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:10.980+0000 7f1cfb50e700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cf416d470 con 0x7f1cf4159130 2023-10-06T03:31:10.990 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:10.980+0000 7f1cfb50e700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cf416a850 con 0x7f1cf4158d60 2023-10-06T03:31:10.991 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:10.980+0000 7f1cfb50e700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1cf4152580 con 0x7f1cf40d02f0 2023-10-06T03:31:10.991 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:10.980+0000 7f1cf9aab700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1cf4159130 0x7f1cf40ced80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:31:10.991 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:10.980+0000 7f1cf8aa9700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1cf4158d60 0x7f1cf4152040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:31:10.991 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:10.980+0000 7f1cf92aa700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1cf40d02f0 0x7f1cf416a310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:31:10.992 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:10.980+0000 7f1cf9aab700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1cf4159130 0x7f1cf40ced80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:54612/0 (socket says 172.21.15.110:54612) 2023-10-06T03:31:10.992 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:10.980+0000 7f1cf9aab700 1 -- 172.21.15.110:0/4219955926 learned_addr learned my addr 172.21.15.110:0/4219955926 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:31:10.992 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:10.980+0000 7f1cf9aab700 1 -- 172.21.15.110:0/4219955926 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1cf40d02f0 msgr2=0x7f1cf416a310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:31:10.992 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:10.980+0000 7f1cf9aab700 1 --2- 172.21.15.110:0/4219955926 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f1cf40d02f0 0x7f1cf416a310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:10.992 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:10.980+0000 7f1cf9aab700 1 -- 172.21.15.110:0/4219955926 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1cf4158d60 msgr2=0x7f1cf4152040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:31:10.993 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:10.980+0000 7f1cf9aab700 1 --2- 172.21.15.110:0/4219955926 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1cf4158d60 0x7f1cf4152040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:10.993 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:10.980+0000 7f1cf9aab700 1 -- 172.21.15.110:0/4219955926 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1cf00094f0 con 0x7f1cf4159130 2023-10-06T03:31:10.993 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:10.980+0000 7f1cf9aab700 1 --2- 172.21.15.110:0/4219955926 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1cf4159130 0x7f1cf40ced80 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f1cec00d100 tx=0x7f1cec0099b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:31:10.993 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:10.980+0000 7f1ce97fa700 1 -- 172.21.15.110:0/4219955926 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cec00dc40 con 0x7f1cf4159130 2023-10-06T03:31:10.993 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:10.980+0000 7f1ce97fa700 1 -- 172.21.15.110:0/4219955926 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1cec00f040 con 0x7f1cf4159130 2023-10-06T03:31:10.994 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:10.980+0000 7f1ce97fa700 1 -- 172.21.15.110:0/4219955926 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cec015440 con 0x7f1cf4159130 2023-10-06T03:31:10.994 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:10.980+0000 7f1cfb50e700 1 -- 172.21.15.110:0/4219955926 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1cf415d010 con 0x7f1cf4159130 2023-10-06T03:31:10.994 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:10.980+0000 7f1cfb50e700 1 -- 172.21.15.110:0/4219955926 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1cf415d5d0 con 0x7f1cf4159130 2023-10-06T03:31:10.994 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:10.984+0000 7f1cfb50e700 1 -- 172.21.15.110:0/4219955926 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f1cf4064770 con 0x7f1cf4159130 2023-10-06T03:31:10.996 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:10.984+0000 7f1ce97fa700 1 -- 172.21.15.110:0/4219955926 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 13) v1 ==== 80061+0+0 (secure 0 0 0) 0x7f1cec01e3b0 con 0x7f1cf4159130 2023-10-06T03:31:10.996 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:10.984+0000 7f1ce97fa700 1 --2- 172.21.15.110:0/4219955926 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f1cd8060670 0x7f1cd8062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:31:10.996 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:10.984+0000 7f1ce97fa700 1 -- 172.21.15.110:0/4219955926 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 6171+0+0 (secure 0 0 0) 0x7f1cec07d760 con 0x7f1cf4159130 2023-10-06T03:31:10.996 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:10.984+0000 7f1cf92aa700 1 --2- 172.21.15.110:0/4219955926 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f1cd8060670 0x7f1cd8062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:31:10.996 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:10.984+0000 7f1ce97fa700 1 --2- 172.21.15.110:0/4219955926 >> [v2:172.21.15.110:6800/33396,v1:172.21.15.110:6801/33396] conn(0x7f1cd8067500 0x7f1cd8069930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:31:10.997 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:10.984+0000 7f1ce97fa700 1 -- 172.21.15.110:0/4219955926 --> [v2:172.21.15.110:6800/33396,v1:172.21.15.110:6801/33396] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1cd806a000 con 0x7f1cd8067500 2023-10-06T03:31:10.997 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:10.984+0000 7f1ce97fa700 1 -- 172.21.15.110:0/4219955926 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_get_version_reply(handle=1 version=47) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1cec07db30 con 0x7f1cf4159130 2023-10-06T03:31:10.997 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:10.984+0000 7f1cf8aa9700 1 --2- 172.21.15.110:0/4219955926 >> [v2:172.21.15.110:6800/33396,v1:172.21.15.110:6801/33396] conn(0x7f1cd8067500 0x7f1cd8069930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:31:10.997 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:10.984+0000 7f1cf92aa700 1 --2- 172.21.15.110:0/4219955926 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f1cd8060670 0x7f1cd8062b20 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f1ce4006fd0 tx=0x7f1ce4008040 comp rx=0 tx=0).ready entity=mgr.5505 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:31:10.997 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:10.984+0000 7f1cf8aa9700 1 --2- 172.21.15.110:0/4219955926 >> [v2:172.21.15.110:6800/33396,v1:172.21.15.110:6801/33396] conn(0x7f1cd8067500 0x7f1cd8069930 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:31:10.998 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:10.988+0000 7f1ce97fa700 1 -- 172.21.15.110:0/4219955926 <== osd.1 v2:172.21.15.110:6800/33396 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23999 (crc 0 0 0) 0x7f1cd806a000 con 0x7f1cd8067500 2023-10-06T03:31:11.005 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.000+0000 7f1cfb50e700 1 -- 172.21.15.110:0/4219955926 --> [v2:172.21.15.110:6800/33396,v1:172.21.15.110:6801/33396] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f1cf415d9c0 con 0x7f1cd8067500 2023-10-06T03:31:11.006 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.000+0000 7f1ce97fa700 1 -- 172.21.15.110:0/4219955926 <== osd.1 v2:172.21.15.110:6800/33396 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7f1cf415d9c0 con 0x7f1cd8067500 2023-10-06T03:31:11.006 INFO:teuthology.orchestra.run.smithi110.stdout:{ 2023-10-06T03:31:11.006 INFO:teuthology.orchestra.run.smithi110.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-10-06T03:31:11.006 INFO:teuthology.orchestra.run.smithi110.stdout:} 2023-10-06T03:31:11.007 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.000+0000 7f1cfb50e700 1 -- 172.21.15.110:0/4219955926 >> [v2:172.21.15.110:6800/33396,v1:172.21.15.110:6801/33396] conn(0x7f1cd8067500 msgr2=0x7f1cd8069930 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:31:11.007 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.000+0000 7f1cfb50e700 1 --2- 172.21.15.110:0/4219955926 >> [v2:172.21.15.110:6800/33396,v1:172.21.15.110:6801/33396] conn(0x7f1cd8067500 0x7f1cd8069930 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:11.007 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.000+0000 7f1cfb50e700 1 -- 172.21.15.110:0/4219955926 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f1cd8060670 msgr2=0x7f1cd8062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:31:11.007 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.000+0000 7f1cfb50e700 1 --2- 172.21.15.110:0/4219955926 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f1cd8060670 0x7f1cd8062b20 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f1ce4006fd0 tx=0x7f1ce4008040 comp rx=0 tx=0).stop 2023-10-06T03:31:11.008 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.000+0000 7f1cfb50e700 1 -- 172.21.15.110:0/4219955926 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1cf4159130 msgr2=0x7f1cf40ced80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:31:11.008 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.000+0000 7f1cfb50e700 1 --2- 172.21.15.110:0/4219955926 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1cf4159130 0x7f1cf40ced80 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f1cec00d100 tx=0x7f1cec0099b0 comp rx=0 tx=0).stop 2023-10-06T03:31:11.008 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.000+0000 7f1cf9aab700 1 -- 172.21.15.110:0/4219955926 reap_dead start 2023-10-06T03:31:11.008 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.000+0000 7f1cfb50e700 1 -- 172.21.15.110:0/4219955926 shutdown_connections 2023-10-06T03:31:11.008 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.000+0000 7f1cfb50e700 1 -- 172.21.15.110:0/4219955926 >> 172.21.15.110:0/4219955926 conn(0x7f1cf4067040 msgr2=0x7f1cf40b4d70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:31:11.008 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.000+0000 7f1cfb50e700 1 -- 172.21.15.110:0/4219955926 shutdown_connections 2023-10-06T03:31:11.008 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.000+0000 7f1cfb50e700 1 -- 172.21.15.110:0/4219955926 wait complete. 2023-10-06T03:31:11.018 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2023-10-06T03:31:11.124 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.120+0000 7f76b91cf700 1 Processor -- start 2023-10-06T03:31:11.124 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.120+0000 7f76b91cf700 1 -- start start 2023-10-06T03:31:11.125 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.120+0000 7f76b91cf700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f76b40c06b0 0x7f76b4156cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:31:11.126 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.120+0000 7f76b91cf700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76b415b6a0 con 0x7f76b41568f0 2023-10-06T03:31:11.126 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.120+0000 7f76b91cf700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76b40b0f70 con 0x7f76b40b0800 2023-10-06T03:31:11.126 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.120+0000 7f76b91cf700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76b40b1250 con 0x7f76b40c06b0 2023-10-06T03:31:11.126 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.120+0000 7f76b3fff700 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f76b41568f0 0x7f76b415af90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:48132/0 (socket says 172.21.15.110:48132) 2023-10-06T03:31:11.126 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.120+0000 7f76b3fff700 1 -- 172.21.15.110:0/705507536 learned_addr learned my addr 172.21.15.110:0/705507536 (peer_addr_for_me v1:172.21.15.110:0/0) 2023-10-06T03:31:11.126 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.120+0000 7f76b37fe700 1 --2- 172.21.15.110:0/705507536 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f76b40c06b0 0x7f76b4156cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:31:11.126 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.120+0000 7f76b2ffd700 1 -- 172.21.15.110:0/705507536 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 475993357 0 0) 0x7f76b415b6a0 con 0x7f76b41568f0 2023-10-06T03:31:11.126 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.120+0000 7f76b2ffd700 1 -- 172.21.15.110:0/705507536 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f769c003570 con 0x7f76b41568f0 2023-10-06T03:31:11.126 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.120+0000 7f76b2ffd700 1 -- 172.21.15.110:0/705507536 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1460449477 0 0) 0x7f76b40b0f70 con 0x7f76b40b0800 2023-10-06T03:31:11.126 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.120+0000 7f76b2ffd700 1 -- 172.21.15.110:0/705507536 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f76b415b6a0 con 0x7f76b40b0800 2023-10-06T03:31:11.126 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.120+0000 7f76b37fe700 1 -- 172.21.15.110:0/705507536 >> v1:172.21.15.192:6789/0 conn(0x7f76b40b0800 legacy=0x7f76b40cb7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:31:11.127 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.120+0000 7f76b37fe700 1 -- 172.21.15.110:0/705507536 >> v1:172.21.15.110:6789/0 conn(0x7f76b41568f0 legacy=0x7f76b415af90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:31:11.127 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.120+0000 7f76b2ffd700 1 -- 172.21.15.110:0/705507536 <== mon.0 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3469049272 0 0) 0x7f769c003570 con 0x7f76b41568f0 2023-10-06T03:31:11.127 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.120+0000 7f76b37fe700 1 -- 172.21.15.110:0/705507536 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76b40b1530 con 0x7f76b40c06b0 2023-10-06T03:31:11.128 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.120+0000 7f76b37fe700 1 --2- 172.21.15.110:0/705507536 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f76b40c06b0 0x7f76b4156cc0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f76a8009910 tx=0x7f76a80091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e3ee00d72d3dff15 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:31:11.128 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.120+0000 7f76b2ffd700 1 -- 172.21.15.110:0/705507536 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76a4001470 con 0x7f76b40c06b0 2023-10-06T03:31:11.128 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.120+0000 7f76b2ffd700 1 -- 172.21.15.110:0/705507536 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f76a8011420 con 0x7f76b40c06b0 2023-10-06T03:31:11.128 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.120+0000 7f76b2ffd700 1 -- 172.21.15.110:0/705507536 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76a80117c0 con 0x7f76b40c06b0 2023-10-06T03:31:11.129 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.120+0000 7f76b91cf700 1 -- 172.21.15.110:0/705507536 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f76b40c06b0 msgr2=0x7f76b4156cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:31:11.129 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.120+0000 7f76b91cf700 1 --2- 172.21.15.110:0/705507536 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f76b40c06b0 0x7f76b4156cc0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f76a8009910 tx=0x7f76a80091a0 comp rx=0 tx=0).stop 2023-10-06T03:31:11.129 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.120+0000 7f76b91cf700 1 -- 172.21.15.110:0/705507536 shutdown_connections 2023-10-06T03:31:11.129 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.120+0000 7f76b91cf700 1 --2- 172.21.15.110:0/705507536 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f76b40c06b0 0x7f76b4156cc0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:11.129 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.120+0000 7f76b91cf700 1 -- 172.21.15.110:0/705507536 >> 172.21.15.110:0/705507536 conn(0x7f76b406c760 msgr2=0x7f76b406cb60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:31:11.129 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.120+0000 7f76b91cf700 1 -- 172.21.15.110:0/705507536 shutdown_connections 2023-10-06T03:31:11.129 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.124+0000 7f76b91cf700 1 -- 172.21.15.110:0/705507536 wait complete. 2023-10-06T03:31:11.129 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.124+0000 7f76b91cf700 1 Processor -- start 2023-10-06T03:31:11.129 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.124+0000 7f76b91cf700 1 -- start start 2023-10-06T03:31:11.131 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.124+0000 7f76b91cf700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f76b40b0800 0x7f76b4168f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:31:11.131 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.124+0000 7f76b91cf700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f76b40c06b0 0x7f76b414dc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:31:11.132 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.124+0000 7f76b91cf700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f76b41568f0 0x7f76b4154dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:31:11.132 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.124+0000 7f76b91cf700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76b414d7d0 con 0x7f76b41568f0 2023-10-06T03:31:11.132 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.124+0000 7f76b91cf700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76b416da60 con 0x7f76b40c06b0 2023-10-06T03:31:11.132 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.124+0000 7f76b91cf700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76b414d180 con 0x7f76b40b0800 2023-10-06T03:31:11.132 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.124+0000 7f76b89ce700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f76b41568f0 0x7f76b4154dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:31:11.132 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.124+0000 7f76b89ce700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f76b41568f0 0x7f76b4154dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:54626/0 (socket says 172.21.15.110:54626) 2023-10-06T03:31:11.132 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.124+0000 7f76b89ce700 1 -- 172.21.15.110:0/488412552 learned_addr learned my addr 172.21.15.110:0/488412552 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:31:11.132 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.124+0000 7f76b37fe700 1 --2- 172.21.15.110:0/488412552 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f76b40c06b0 0x7f76b414dc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:31:11.132 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.124+0000 7f76b3fff700 1 --2- 172.21.15.110:0/488412552 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f76b40b0800 0x7f76b4168f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:31:11.132 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.124+0000 7f76b89ce700 1 -- 172.21.15.110:0/488412552 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f76b40b0800 msgr2=0x7f76b4168f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:31:11.133 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.124+0000 7f76b89ce700 1 --2- 172.21.15.110:0/488412552 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f76b40b0800 0x7f76b4168f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:11.133 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.124+0000 7f76b89ce700 1 -- 172.21.15.110:0/488412552 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f76b40c06b0 msgr2=0x7f76b414dc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:31:11.133 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.124+0000 7f76b89ce700 1 --2- 172.21.15.110:0/488412552 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f76b40c06b0 0x7f76b414dc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:11.133 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.124+0000 7f76b89ce700 1 -- 172.21.15.110:0/488412552 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76a8009420 con 0x7f76b41568f0 2023-10-06T03:31:11.133 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.124+0000 7f76b89ce700 1 --2- 172.21.15.110:0/488412552 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f76b41568f0 0x7f76b4154dc0 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f76ac00c860 tx=0x7f76ac002700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:31:11.133 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.124+0000 7f769bfff700 1 -- 172.21.15.110:0/488412552 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76ac002030 con 0x7f76b41568f0 2023-10-06T03:31:11.133 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.124+0000 7f769bfff700 1 -- 172.21.15.110:0/488412552 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f76ac0021b0 con 0x7f76b41568f0 2023-10-06T03:31:11.133 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.124+0000 7f76b91cf700 1 -- 172.21.15.110:0/488412552 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76b414d460 con 0x7f76b41568f0 2023-10-06T03:31:11.133 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.124+0000 7f769bfff700 1 -- 172.21.15.110:0/488412552 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76ac012630 con 0x7f76b41568f0 2023-10-06T03:31:11.133 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.124+0000 7f76b91cf700 1 -- 172.21.15.110:0/488412552 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76b4169440 con 0x7f76b41568f0 2023-10-06T03:31:11.133 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.128+0000 7f76b91cf700 1 -- 172.21.15.110:0/488412552 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f76b4071020 con 0x7f76b41568f0 2023-10-06T03:31:11.134 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.128+0000 7f769bfff700 1 -- 172.21.15.110:0/488412552 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 13) v1 ==== 80061+0+0 (secure 0 0 0) 0x7f76ac012a20 con 0x7f76b41568f0 2023-10-06T03:31:11.134 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.128+0000 7f769bfff700 1 --2- 172.21.15.110:0/488412552 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f76a00606a0 0x7f76a0062b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:31:11.134 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.128+0000 7f769bfff700 1 -- 172.21.15.110:0/488412552 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 6171+0+0 (secure 0 0 0) 0x7f76ac07ca90 con 0x7f76b41568f0 2023-10-06T03:31:11.134 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.128+0000 7f76b3fff700 1 --2- 172.21.15.110:0/488412552 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f76a00606a0 0x7f76a0062b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:31:11.135 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.128+0000 7f76b3fff700 1 --2- 172.21.15.110:0/488412552 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f76a00606a0 0x7f76a0062b50 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f76a40015a0 tx=0x7f76a4000bc0 comp rx=0 tx=0).ready entity=mgr.5505 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:31:11.141 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.136+0000 7f769bfff700 1 -- 172.21.15.110:0/488412552 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f76ac04cda0 con 0x7f76b41568f0 2023-10-06T03:31:11.304 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.300+0000 7f76b91cf700 1 -- 172.21.15.110:0/488412552 --> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "1", "target": ["mon-mgr", ""]}) v1 -- 0x7f76b414d000 con 0x7f76a00606a0 2023-10-06T03:31:11.304 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.300+0000 7f769bfff700 1 -- 172.21.15.110:0/488412552 <== mgr.5505 v2:172.21.15.110:6832/33268 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 1 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f76a4005e10 con 0x7f76a00606a0 2023-10-06T03:31:11.308 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.304+0000 7f76b91cf700 1 -- 172.21.15.110:0/488412552 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f76a00606a0 msgr2=0x7f76a0062b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:31:11.308 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.304+0000 7f76b91cf700 1 --2- 172.21.15.110:0/488412552 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f76a00606a0 0x7f76a0062b50 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f76a40015a0 tx=0x7f76a4000bc0 comp rx=0 tx=0).stop 2023-10-06T03:31:11.308 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.304+0000 7f76b91cf700 1 -- 172.21.15.110:0/488412552 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f76b41568f0 msgr2=0x7f76b4154dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:31:11.309 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.304+0000 7f76b91cf700 1 --2- 172.21.15.110:0/488412552 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f76b41568f0 0x7f76b4154dc0 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f76ac00c860 tx=0x7f76ac002700 comp rx=0 tx=0).stop 2023-10-06T03:31:11.309 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.304+0000 7f76b91cf700 1 -- 172.21.15.110:0/488412552 shutdown_connections 2023-10-06T03:31:11.309 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.304+0000 7f76b91cf700 1 --2- 172.21.15.110:0/488412552 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f76a00606a0 0x7f76a0062b50 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:11.309 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.304+0000 7f76b91cf700 1 --2- 172.21.15.110:0/488412552 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f76b41568f0 0x7f76b4154dc0 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:11.309 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.304+0000 7f76b91cf700 1 --2- 172.21.15.110:0/488412552 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f76b40c06b0 0x7f76b414dc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:11.309 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.304+0000 7f76b91cf700 1 --2- 172.21.15.110:0/488412552 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f76b40b0800 0x7f76b4168f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:11.309 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.304+0000 7f76b91cf700 1 -- 172.21.15.110:0/488412552 >> 172.21.15.110:0/488412552 conn(0x7f76b406c760 msgr2=0x7f76b406b220 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:31:11.309 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.304+0000 7f76b91cf700 1 -- 172.21.15.110:0/488412552 shutdown_connections 2023-10-06T03:31:11.309 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.304+0000 7f76b91cf700 1 -- 172.21.15.110:0/488412552 wait complete. 2023-10-06T03:31:11.310 INFO:teuthology.orchestra.run.smithi110.stderr:instructed osd(s) 1 to deep-scrub 2023-10-06T03:31:11.321 INFO:tasks.ceph:Scrubbing osd.2 2023-10-06T03:31:11.322 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2023-10-06T03:31:11.440 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.436+0000 7f9f91d49700 1 Processor -- start 2023-10-06T03:31:11.441 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.436+0000 7f9f91d49700 1 -- start start 2023-10-06T03:31:11.442 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.436+0000 7f9f91d49700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9f8c0abfe0 0x7f9f8c069b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:31:11.443 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.436+0000 7f9f91d49700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f9f8c06ce70 0x7f9f8c15aa00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:31:11.443 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.436+0000 7f9f91d49700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f8c069670 con 0x7f9f8c0abc10 2023-10-06T03:31:11.443 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.436+0000 7f9f91d49700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9f8c15b7a0 con 0x7f9f8c06ce70 2023-10-06T03:31:11.443 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.436+0000 7f9f91d49700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f8c0ad3c0 con 0x7f9f8c0abfe0 2023-10-06T03:31:11.443 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.436+0000 7f9f91548700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f9f8c06ce70 0x7f9f8c15aa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:31:11.443 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.436+0000 7f9f91548700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f9f8c06ce70 0x7f9f8c15aa00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:54638/0 (socket says 172.21.15.110:54638) 2023-10-06T03:31:11.443 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.436+0000 7f9f91548700 1 -- 172.21.15.110:0/1075558209 learned_addr learned my addr 172.21.15.110:0/1075558209 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:31:11.443 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.436+0000 7f9f8bfff700 1 --2- 172.21.15.110:0/1075558209 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9f8c0abfe0 0x7f9f8c069b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:31:11.444 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.436+0000 7f9f91548700 1 -- 172.21.15.110:0/1075558209 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9f8c0abfe0 msgr2=0x7f9f8c069b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:31:11.444 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.436+0000 7f9f91548700 1 --2- 172.21.15.110:0/1075558209 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9f8c0abfe0 0x7f9f8c069b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:11.444 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.436+0000 7f9f91548700 1 -- 172.21.15.110:0/1075558209 >> v1:172.21.15.192:6789/0 conn(0x7f9f8c0abc10 legacy=0x7f9f8c06c760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:31:11.444 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.436+0000 7f9f91548700 1 -- 172.21.15.110:0/1075558209 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f8c068d40 con 0x7f9f8c06ce70 2023-10-06T03:31:11.444 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.436+0000 7f9f91548700 1 --2- 172.21.15.110:0/1075558209 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9f8c06ce70 0x7f9f8c15aa00 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f9f84009f20 tx=0x7f9f84009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=723de0892fcac007 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:31:11.444 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.436+0000 7f9f8b7fe700 1 -- 172.21.15.110:0/1075558209 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f8400e070 con 0x7f9f8c06ce70 2023-10-06T03:31:11.444 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.436+0000 7f9f8b7fe700 1 -- 172.21.15.110:0/1075558209 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f8400ba60 con 0x7f9f8c06ce70 2023-10-06T03:31:11.444 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.436+0000 7f9f8b7fe700 1 -- 172.21.15.110:0/1075558209 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f840092e0 con 0x7f9f8c06ce70 2023-10-06T03:31:11.445 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.440+0000 7f9f91d49700 1 -- 172.21.15.110:0/1075558209 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9f8c06ce70 msgr2=0x7f9f8c15aa00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:31:11.445 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.440+0000 7f9f91d49700 1 --2- 172.21.15.110:0/1075558209 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9f8c06ce70 0x7f9f8c15aa00 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f9f84009f20 tx=0x7f9f84009840 comp rx=0 tx=0).stop 2023-10-06T03:31:11.446 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.440+0000 7f9f91d49700 1 -- 172.21.15.110:0/1075558209 shutdown_connections 2023-10-06T03:31:11.446 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.440+0000 7f9f91d49700 1 --2- 172.21.15.110:0/1075558209 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9f8c06ce70 0x7f9f8c15aa00 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:11.446 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.440+0000 7f9f91d49700 1 --2- 172.21.15.110:0/1075558209 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9f8c0abfe0 0x7f9f8c069b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:11.446 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.440+0000 7f9f91d49700 1 -- 172.21.15.110:0/1075558209 >> 172.21.15.110:0/1075558209 conn(0x7f9f8c06b4c0 msgr2=0x7f9f8c06b8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:31:11.446 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.440+0000 7f9f91d49700 1 -- 172.21.15.110:0/1075558209 shutdown_connections 2023-10-06T03:31:11.446 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.440+0000 7f9f91d49700 1 -- 172.21.15.110:0/1075558209 wait complete. 2023-10-06T03:31:11.447 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.440+0000 7f9f91d49700 1 Processor -- start 2023-10-06T03:31:11.447 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.440+0000 7f9f91d49700 1 -- start start 2023-10-06T03:31:11.448 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.444+0000 7f9f91d49700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9f8c06ce70 0x7f9f8c152150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:31:11.449 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.444+0000 7f9f91d49700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9f8c0abc10 0x7f9f8c159680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:31:11.449 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.444+0000 7f9f91d49700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9f8c0abfe0 0x7f9f8c159bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:31:11.449 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.444+0000 7f9f91d49700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f8c0b0a60 con 0x7f9f8c0abfe0 2023-10-06T03:31:11.449 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.444+0000 7f9f91d49700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f8c0b01a0 con 0x7f9f8c06ce70 2023-10-06T03:31:11.449 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.444+0000 7f9f91d49700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f8c0c3ad0 con 0x7f9f8c0abc10 2023-10-06T03:31:11.449 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.444+0000 7f9f91548700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9f8c0abfe0 0x7f9f8c159bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:31:11.449 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.444+0000 7f9f91548700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9f8c0abfe0 0x7f9f8c159bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:54640/0 (socket says 172.21.15.110:54640) 2023-10-06T03:31:11.449 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.444+0000 7f9f91548700 1 -- 172.21.15.110:0/45256798 learned_addr learned my addr 172.21.15.110:0/45256798 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:31:11.449 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.444+0000 7f9f90d47700 1 --2- 172.21.15.110:0/45256798 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9f8c06ce70 0x7f9f8c152150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:31:11.450 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.444+0000 7f9f8bfff700 1 --2- 172.21.15.110:0/45256798 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9f8c0abc10 0x7f9f8c159680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:31:11.450 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.444+0000 7f9f91548700 1 -- 172.21.15.110:0/45256798 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9f8c0abc10 msgr2=0x7f9f8c159680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:31:11.450 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.444+0000 7f9f91548700 1 --2- 172.21.15.110:0/45256798 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f9f8c0abc10 0x7f9f8c159680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:11.450 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.444+0000 7f9f91548700 1 -- 172.21.15.110:0/45256798 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9f8c06ce70 msgr2=0x7f9f8c152150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:31:11.450 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.444+0000 7f9f91548700 1 --2- 172.21.15.110:0/45256798 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9f8c06ce70 0x7f9f8c152150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:11.450 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.444+0000 7f9f91548700 1 -- 172.21.15.110:0/45256798 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f84009ae0 con 0x7f9f8c0abfe0 2023-10-06T03:31:11.450 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.444+0000 7f9f91548700 1 --2- 172.21.15.110:0/45256798 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9f8c0abfe0 0x7f9f8c159bc0 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f9f8400b610 tx=0x7f9f84018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:31:11.451 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.444+0000 7f9f88ff9700 1 -- 172.21.15.110:0/45256798 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f840032f0 con 0x7f9f8c0abfe0 2023-10-06T03:31:11.451 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.444+0000 7f9f88ff9700 1 -- 172.21.15.110:0/45256798 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f84008ce0 con 0x7f9f8c0abfe0 2023-10-06T03:31:11.451 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.444+0000 7f9f88ff9700 1 -- 172.21.15.110:0/45256798 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f8401c620 con 0x7f9f8c0abfe0 2023-10-06T03:31:11.451 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.444+0000 7f9f91d49700 1 -- 172.21.15.110:0/45256798 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f8c0b0410 con 0x7f9f8c0abfe0 2023-10-06T03:31:11.452 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.444+0000 7f9f91d49700 1 -- 172.21.15.110:0/45256798 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f8c15a1d0 con 0x7f9f8c0abfe0 2023-10-06T03:31:11.452 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.444+0000 7f9f91d49700 1 -- 172.21.15.110:0/45256798 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f9f8c06a9e0 con 0x7f9f8c0abfe0 2023-10-06T03:31:11.453 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.448+0000 7f9f88ff9700 1 -- 172.21.15.110:0/45256798 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 13) v1 ==== 80061+0+0 (secure 0 0 0) 0x7f9f84003040 con 0x7f9f8c0abfe0 2023-10-06T03:31:11.453 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.448+0000 7f9f88ff9700 1 --2- 172.21.15.110:0/45256798 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f9f6c060670 0x7f9f6c062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:31:11.453 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.448+0000 7f9f88ff9700 1 -- 172.21.15.110:0/45256798 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 6171+0+0 (secure 0 0 0) 0x7f9f8407e2e0 con 0x7f9f8c0abfe0 2023-10-06T03:31:11.453 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.448+0000 7f9f90d47700 1 --2- 172.21.15.110:0/45256798 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f9f6c060670 0x7f9f6c062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:31:11.453 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.448+0000 7f9f88ff9700 1 --2- 172.21.15.110:0/45256798 >> [v2:172.21.15.110:6816/33397,v1:172.21.15.110:6817/33397] conn(0x7f9f6c067500 0x7f9f6c069930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:31:11.453 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.448+0000 7f9f88ff9700 1 -- 172.21.15.110:0/45256798 --> [v2:172.21.15.110:6816/33397,v1:172.21.15.110:6817/33397] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9f6c06a000 con 0x7f9f6c067500 2023-10-06T03:31:11.453 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.448+0000 7f9f88ff9700 1 -- 172.21.15.110:0/45256798 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_get_version_reply(handle=1 version=47) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9f840807e0 con 0x7f9f8c0abfe0 2023-10-06T03:31:11.453 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.448+0000 7f9f8bfff700 1 --2- 172.21.15.110:0/45256798 >> [v2:172.21.15.110:6816/33397,v1:172.21.15.110:6817/33397] conn(0x7f9f6c067500 0x7f9f6c069930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:31:11.454 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.448+0000 7f9f90d47700 1 --2- 172.21.15.110:0/45256798 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f9f6c060670 0x7f9f6c062b20 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f9f7c003a80 tx=0x7f9f7c007ec0 comp rx=0 tx=0).ready entity=mgr.5505 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:31:11.454 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.448+0000 7f9f8bfff700 1 --2- 172.21.15.110:0/45256798 >> [v2:172.21.15.110:6816/33397,v1:172.21.15.110:6817/33397] conn(0x7f9f6c067500 0x7f9f6c069930 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:31:11.455 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.448+0000 7f9f88ff9700 1 -- 172.21.15.110:0/45256798 <== osd.2 v2:172.21.15.110:6816/33397 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23999 (crc 0 0 0) 0x7f9f6c06a000 con 0x7f9f6c067500 2023-10-06T03:31:11.467 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.464+0000 7f9f91d49700 1 -- 172.21.15.110:0/45256798 --> [v2:172.21.15.110:6816/33397,v1:172.21.15.110:6817/33397] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f9f8c069020 con 0x7f9f6c067500 2023-10-06T03:31:11.468 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.464+0000 7f9f88ff9700 1 -- 172.21.15.110:0/45256798 <== osd.2 v2:172.21.15.110:6816/33397 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7f9f8c069020 con 0x7f9f6c067500 2023-10-06T03:31:11.468 INFO:teuthology.orchestra.run.smithi110.stdout:{ 2023-10-06T03:31:11.469 INFO:teuthology.orchestra.run.smithi110.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-10-06T03:31:11.469 INFO:teuthology.orchestra.run.smithi110.stdout:} 2023-10-06T03:31:11.470 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.464+0000 7f9f91d49700 1 -- 172.21.15.110:0/45256798 >> [v2:172.21.15.110:6816/33397,v1:172.21.15.110:6817/33397] conn(0x7f9f6c067500 msgr2=0x7f9f6c069930 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:31:11.470 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.464+0000 7f9f91d49700 1 --2- 172.21.15.110:0/45256798 >> [v2:172.21.15.110:6816/33397,v1:172.21.15.110:6817/33397] conn(0x7f9f6c067500 0x7f9f6c069930 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:11.470 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.464+0000 7f9f91d49700 1 -- 172.21.15.110:0/45256798 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f9f6c060670 msgr2=0x7f9f6c062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:31:11.471 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.464+0000 7f9f91d49700 1 --2- 172.21.15.110:0/45256798 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f9f6c060670 0x7f9f6c062b20 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f9f7c003a80 tx=0x7f9f7c007ec0 comp rx=0 tx=0).stop 2023-10-06T03:31:11.471 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.464+0000 7f9f91d49700 1 -- 172.21.15.110:0/45256798 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9f8c0abfe0 msgr2=0x7f9f8c159bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:31:11.471 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.464+0000 7f9f91d49700 1 --2- 172.21.15.110:0/45256798 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9f8c0abfe0 0x7f9f8c159bc0 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f9f8400b610 tx=0x7f9f84018000 comp rx=0 tx=0).stop 2023-10-06T03:31:11.471 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.464+0000 7f9f91548700 1 -- 172.21.15.110:0/45256798 reap_dead start 2023-10-06T03:31:11.471 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.464+0000 7f9f91d49700 1 -- 172.21.15.110:0/45256798 shutdown_connections 2023-10-06T03:31:11.472 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.464+0000 7f9f91d49700 1 -- 172.21.15.110:0/45256798 >> 172.21.15.110:0/45256798 conn(0x7f9f8c06b4c0 msgr2=0x7f9f8c0add60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:31:11.472 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.464+0000 7f9f91d49700 1 -- 172.21.15.110:0/45256798 shutdown_connections 2023-10-06T03:31:11.472 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.464+0000 7f9f91d49700 1 -- 172.21.15.110:0/45256798 wait complete. 2023-10-06T03:31:11.480 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2023-10-06T03:31:11.592 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.588+0000 7f0b3120c700 1 Processor -- start 2023-10-06T03:31:11.593 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.588+0000 7f0b3120c700 1 -- start start 2023-10-06T03:31:11.595 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.588+0000 7f0b3120c700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f0b2c1592c0 0x7f0b2c0cfd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:31:11.595 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.588+0000 7f0b3120c700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0b2c0d0280 0x7f0b2c0b3380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:31:11.595 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.588+0000 7f0b3120c700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b2c0d06a0 con 0x7f0b2c158ef0 2023-10-06T03:31:11.595 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.588+0000 7f0b3120c700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0b2c0ce7f0 con 0x7f0b2c1592c0 2023-10-06T03:31:11.595 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.588+0000 7f0b3120c700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b2c0cd2d0 con 0x7f0b2c0d0280 2023-10-06T03:31:11.595 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.588+0000 7f0b2a59c700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f0b2c1592c0 0x7f0b2c0cfd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:31:11.595 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.588+0000 7f0b2ad9d700 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f0b2c158ef0 0x7f0b2c0cb100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:48134/0 (socket says 172.21.15.110:48134) 2023-10-06T03:31:11.596 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.588+0000 7f0b2ad9d700 1 -- 172.21.15.110:0/3417644874 learned_addr learned my addr 172.21.15.110:0/3417644874 (peer_addr_for_me v1:172.21.15.110:0/0) 2023-10-06T03:31:11.596 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.588+0000 7f0b2b59e700 1 --2- 172.21.15.110:0/3417644874 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0b2c0d0280 0x7f0b2c0b3380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:31:11.596 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.588+0000 7f0b2a59c700 1 -- 172.21.15.110:0/3417644874 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0b2c0d0280 msgr2=0x7f0b2c0b3380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:31:11.596 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.588+0000 7f0b29d9b700 1 -- 172.21.15.110:0/3417644874 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3522923722 0 0) 0x7f0b2c0d06a0 con 0x7f0b2c158ef0 2023-10-06T03:31:11.596 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.588+0000 7f0b2a59c700 1 --2- 172.21.15.110:0/3417644874 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0b2c0d0280 0x7f0b2c0b3380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:11.596 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.588+0000 7f0b2a59c700 1 -- 172.21.15.110:0/3417644874 >> v1:172.21.15.110:6789/0 conn(0x7f0b2c158ef0 legacy=0x7f0b2c0cb100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:31:11.596 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.588+0000 7f0b2a59c700 1 -- 172.21.15.110:0/3417644874 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b2c0cd5b0 con 0x7f0b2c1592c0 2023-10-06T03:31:11.596 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.588+0000 7f0b2b59e700 1 --2- 172.21.15.110:0/3417644874 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0b2c0d0280 0x7f0b2c0b3380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-06T03:31:11.596 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.588+0000 7f0b2a59c700 1 --2- 172.21.15.110:0/3417644874 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0b2c1592c0 0x7f0b2c0cfd40 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f0b140098e0 tx=0x7f0b140091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c926bb5ff717a892 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:31:11.596 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.588+0000 7f0b29d9b700 1 -- 172.21.15.110:0/3417644874 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b14009e80 con 0x7f0b2c1592c0 2023-10-06T03:31:11.597 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.588+0000 7f0b29d9b700 1 -- 172.21.15.110:0/3417644874 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b1400b860 con 0x7f0b2c1592c0 2023-10-06T03:31:11.597 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.588+0000 7f0b29d9b700 1 -- 172.21.15.110:0/3417644874 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b1400bbe0 con 0x7f0b2c1592c0 2023-10-06T03:31:11.597 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.592+0000 7f0b3120c700 1 -- 172.21.15.110:0/3417644874 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0b2c1592c0 msgr2=0x7f0b2c0cfd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:31:11.598 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.592+0000 7f0b3120c700 1 --2- 172.21.15.110:0/3417644874 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0b2c1592c0 0x7f0b2c0cfd40 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f0b140098e0 tx=0x7f0b140091a0 comp rx=0 tx=0).stop 2023-10-06T03:31:11.598 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.592+0000 7f0b3120c700 1 -- 172.21.15.110:0/3417644874 shutdown_connections 2023-10-06T03:31:11.598 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.592+0000 7f0b3120c700 1 --2- 172.21.15.110:0/3417644874 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0b2c0d0280 0x7f0b2c0b3380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:11.598 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.592+0000 7f0b3120c700 1 --2- 172.21.15.110:0/3417644874 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0b2c1592c0 0x7f0b2c0cfd40 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:11.598 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.592+0000 7f0b3120c700 1 -- 172.21.15.110:0/3417644874 >> 172.21.15.110:0/3417644874 conn(0x7f0b2c067040 msgr2=0x7f0b2c067440 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:31:11.598 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.592+0000 7f0b3120c700 1 -- 172.21.15.110:0/3417644874 shutdown_connections 2023-10-06T03:31:11.598 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.592+0000 7f0b3120c700 1 -- 172.21.15.110:0/3417644874 wait complete. 2023-10-06T03:31:11.601 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.592+0000 7f0b3120c700 1 Processor -- start 2023-10-06T03:31:11.601 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.592+0000 7f0b3120c700 1 -- start start 2023-10-06T03:31:11.601 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.596+0000 7f0b3120c700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0b2c0d0280 0x7f0b2c0c4e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:31:11.601 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.596+0000 7f0b3120c700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0b2c158ef0 0x7f0b2c0b5220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:31:11.601 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.596+0000 7f0b3120c700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0b2c1592c0 0x7f0b2c1533e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:31:11.601 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.596+0000 7f0b3120c700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b2c0c4990 con 0x7f0b2c0d0280 2023-10-06T03:31:11.601 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.596+0000 7f0b3120c700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b2c0b5760 con 0x7f0b2c158ef0 2023-10-06T03:31:11.601 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.596+0000 7f0b3120c700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b2c0c3d80 con 0x7f0b2c1592c0 2023-10-06T03:31:11.602 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.596+0000 7f0b2ad9d700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0b2c0d0280 0x7f0b2c0c4e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:31:11.602 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.596+0000 7f0b2ad9d700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0b2c0d0280 0x7f0b2c0c4e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:54658/0 (socket says 172.21.15.110:54658) 2023-10-06T03:31:11.602 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.596+0000 7f0b2ad9d700 1 -- 172.21.15.110:0/310424020 learned_addr learned my addr 172.21.15.110:0/310424020 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:31:11.602 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.596+0000 7f0b2b59e700 1 --2- 172.21.15.110:0/310424020 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0b2c1592c0 0x7f0b2c1533e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:31:11.602 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.596+0000 7f0b2a59c700 1 --2- 172.21.15.110:0/310424020 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0b2c158ef0 0x7f0b2c0b5220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:31:11.602 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.596+0000 7f0b2ad9d700 1 -- 172.21.15.110:0/310424020 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0b2c1592c0 msgr2=0x7f0b2c1533e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:31:11.602 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.596+0000 7f0b2ad9d700 1 --2- 172.21.15.110:0/310424020 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0b2c1592c0 0x7f0b2c1533e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:11.602 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.596+0000 7f0b2ad9d700 1 -- 172.21.15.110:0/310424020 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0b2c158ef0 msgr2=0x7f0b2c0b5220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:31:11.602 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.596+0000 7f0b2ad9d700 1 --2- 172.21.15.110:0/310424020 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0b2c158ef0 0x7f0b2c0b5220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:11.602 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.596+0000 7f0b2ad9d700 1 -- 172.21.15.110:0/310424020 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b14009420 con 0x7f0b2c0d0280 2023-10-06T03:31:11.603 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.596+0000 7f0b2a59c700 1 --2- 172.21.15.110:0/310424020 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0b2c158ef0 0x7f0b2c0b5220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-06T03:31:11.603 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.596+0000 7f0b2ad9d700 1 --2- 172.21.15.110:0/310424020 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0b2c0d0280 0x7f0b2c0c4e80 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f0b1c00c7e0 tx=0x7f0b1c007df0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:31:11.603 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.596+0000 7f0b22ffd700 1 -- 172.21.15.110:0/310424020 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b1c005b30 con 0x7f0b2c0d0280 2023-10-06T03:31:11.603 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.596+0000 7f0b22ffd700 1 -- 172.21.15.110:0/310424020 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b1c005c90 con 0x7f0b2c0d0280 2023-10-06T03:31:11.603 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.596+0000 7f0b3120c700 1 -- 172.21.15.110:0/310424020 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b2c0c4060 con 0x7f0b2c0d0280 2023-10-06T03:31:11.603 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.596+0000 7f0b22ffd700 1 -- 172.21.15.110:0/310424020 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b1c007300 con 0x7f0b2c0d0280 2023-10-06T03:31:11.603 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.596+0000 7f0b3120c700 1 -- 172.21.15.110:0/310424020 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b2c0c4620 con 0x7f0b2c0d0280 2023-10-06T03:31:11.604 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.596+0000 7f0b3120c700 1 -- 172.21.15.110:0/310424020 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b2c064770 con 0x7f0b2c0d0280 2023-10-06T03:31:11.604 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.596+0000 7f0b22ffd700 1 -- 172.21.15.110:0/310424020 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 13) v1 ==== 80061+0+0 (secure 0 0 0) 0x7f0b1c01b400 con 0x7f0b2c0d0280 2023-10-06T03:31:11.604 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.600+0000 7f0b22ffd700 1 --2- 172.21.15.110:0/310424020 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f0b0c060670 0x7f0b0c062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:31:11.604 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.600+0000 7f0b22ffd700 1 -- 172.21.15.110:0/310424020 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 6171+0+0 (secure 0 0 0) 0x7f0b1c07c710 con 0x7f0b2c0d0280 2023-10-06T03:31:11.604 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.600+0000 7f0b2a59c700 1 --2- 172.21.15.110:0/310424020 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f0b0c060670 0x7f0b0c062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:31:11.605 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.600+0000 7f0b2a59c700 1 --2- 172.21.15.110:0/310424020 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f0b0c060670 0x7f0b0c062b20 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f0b2c1596f0 tx=0x7f0b14017000 comp rx=0 tx=0).ready entity=mgr.5505 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:31:11.612 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.608+0000 7f0b22ffd700 1 -- 172.21.15.110:0/310424020 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f0b1c04ca20 con 0x7f0b2c0d0280 2023-10-06T03:31:11.771 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.764+0000 7f0b3120c700 1 -- 172.21.15.110:0/310424020 --> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "2", "target": ["mon-mgr", ""]}) v1 -- 0x7f0b2c0cdfa0 con 0x7f0b0c060670 2023-10-06T03:31:11.771 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.768+0000 7f0b22ffd700 1 -- 172.21.15.110:0/310424020 <== mgr.5505 v2:172.21.15.110:6832/33268 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 2 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f0b2c0cdfa0 con 0x7f0b0c060670 2023-10-06T03:31:11.776 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.772+0000 7f0b3120c700 1 -- 172.21.15.110:0/310424020 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f0b0c060670 msgr2=0x7f0b0c062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:31:11.776 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.772+0000 7f0b3120c700 1 --2- 172.21.15.110:0/310424020 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f0b0c060670 0x7f0b0c062b20 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f0b2c1596f0 tx=0x7f0b14017000 comp rx=0 tx=0).stop 2023-10-06T03:31:11.776 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.772+0000 7f0b3120c700 1 -- 172.21.15.110:0/310424020 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0b2c0d0280 msgr2=0x7f0b2c0c4e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:31:11.776 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.772+0000 7f0b3120c700 1 --2- 172.21.15.110:0/310424020 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0b2c0d0280 0x7f0b2c0c4e80 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f0b1c00c7e0 tx=0x7f0b1c007df0 comp rx=0 tx=0).stop 2023-10-06T03:31:11.777 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.772+0000 7f0b3120c700 1 -- 172.21.15.110:0/310424020 shutdown_connections 2023-10-06T03:31:11.777 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.772+0000 7f0b3120c700 1 --2- 172.21.15.110:0/310424020 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f0b0c060670 0x7f0b0c062b20 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:11.777 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.772+0000 7f0b3120c700 1 --2- 172.21.15.110:0/310424020 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f0b2c1592c0 0x7f0b2c1533e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:11.777 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.772+0000 7f0b3120c700 1 --2- 172.21.15.110:0/310424020 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0b2c158ef0 0x7f0b2c0b5220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:11.777 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.772+0000 7f0b3120c700 1 --2- 172.21.15.110:0/310424020 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0b2c0d0280 0x7f0b2c0c4e80 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:11.777 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.772+0000 7f0b3120c700 1 -- 172.21.15.110:0/310424020 >> 172.21.15.110:0/310424020 conn(0x7f0b2c067040 msgr2=0x7f0b2c0cb000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:31:11.777 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.772+0000 7f0b3120c700 1 -- 172.21.15.110:0/310424020 shutdown_connections 2023-10-06T03:31:11.777 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.772+0000 7f0b3120c700 1 -- 172.21.15.110:0/310424020 wait complete. 2023-10-06T03:31:11.777 INFO:teuthology.orchestra.run.smithi110.stderr:instructed osd(s) 2 to deep-scrub 2023-10-06T03:31:11.791 INFO:tasks.ceph:Scrubbing osd.3 2023-10-06T03:31:11.791 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2023-10-06T03:31:11.903 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.900+0000 7fec632f4700 1 Processor -- start 2023-10-06T03:31:11.904 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.900+0000 7fec632f4700 1 -- start start 2023-10-06T03:31:11.906 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.900+0000 7fec632f4700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fec5c0cf9e0 0x7fec5c067640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:31:11.906 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.900+0000 7fec632f4700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fec5c1563d0 0x7fec5c156830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:31:11.906 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.900+0000 7fec632f4700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec5c156d70 con 0x7fec5c067270 2023-10-06T03:31:11.906 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.900+0000 7fec632f4700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fec5c0ce300 con 0x7fec5c0cf9e0 2023-10-06T03:31:11.906 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.900+0000 7fec632f4700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec5c0cfe60 con 0x7fec5c1563d0 2023-10-06T03:31:11.906 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.900+0000 7fec61af1700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fec5c0cf9e0 0x7fec5c067640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:31:11.906 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.900+0000 7fec61af1700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fec5c0cf9e0 0x7fec5c067640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:54670/0 (socket says 172.21.15.110:54670) 2023-10-06T03:31:11.906 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.900+0000 7fec61af1700 1 -- 172.21.15.110:0/243223841 learned_addr learned my addr 172.21.15.110:0/243223841 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:31:11.906 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.900+0000 7fec62af3700 1 --2- 172.21.15.110:0/243223841 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fec5c1563d0 0x7fec5c156830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:31:11.906 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.900+0000 7fec61af1700 1 -- 172.21.15.110:0/243223841 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fec5c1563d0 msgr2=0x7fec5c156830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:31:11.907 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.900+0000 7fec61af1700 1 --2- 172.21.15.110:0/243223841 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fec5c1563d0 0x7fec5c156830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:11.907 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.900+0000 7fec61af1700 1 -- 172.21.15.110:0/243223841 >> v1:172.21.15.110:6789/0 conn(0x7fec5c067270 legacy=0x7fec5c159d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:31:11.907 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.900+0000 7fec61af1700 1 -- 172.21.15.110:0/243223841 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec5c0d0140 con 0x7fec5c0cf9e0 2023-10-06T03:31:11.907 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.900+0000 7fec612f0700 1 -- 172.21.15.110:0/243223841 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1782852705 0 0) 0x7fec5c156d70 con 0x7fec5c067270 2023-10-06T03:31:11.907 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.900+0000 7fec62af3700 1 --2- 172.21.15.110:0/243223841 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fec5c1563d0 0x7fec5c156830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-06T03:31:11.907 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.900+0000 7fec61af1700 1 --2- 172.21.15.110:0/243223841 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fec5c0cf9e0 0x7fec5c067640 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7fec4c0099b0 tx=0x7fec4c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=78c8ac0d9a0e688a server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:31:11.907 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.900+0000 7fec612f0700 1 -- 172.21.15.110:0/243223841 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec4c00b860 con 0x7fec5c0cf9e0 2023-10-06T03:31:11.907 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.900+0000 7fec612f0700 1 -- 172.21.15.110:0/243223841 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec4c00b9c0 con 0x7fec5c0cf9e0 2023-10-06T03:31:11.907 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.900+0000 7fec612f0700 1 -- 172.21.15.110:0/243223841 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec4c00bd60 con 0x7fec5c0cf9e0 2023-10-06T03:31:11.909 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.900+0000 7fec632f4700 1 -- 172.21.15.110:0/243223841 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fec5c0cf9e0 msgr2=0x7fec5c067640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:31:11.910 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.900+0000 7fec632f4700 1 --2- 172.21.15.110:0/243223841 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fec5c0cf9e0 0x7fec5c067640 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7fec4c0099b0 tx=0x7fec4c009250 comp rx=0 tx=0).stop 2023-10-06T03:31:11.910 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.900+0000 7fec632f4700 1 -- 172.21.15.110:0/243223841 shutdown_connections 2023-10-06T03:31:11.910 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.900+0000 7fec632f4700 1 --2- 172.21.15.110:0/243223841 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fec5c1563d0 0x7fec5c156830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:11.910 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.900+0000 7fec632f4700 1 --2- 172.21.15.110:0/243223841 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fec5c0cf9e0 0x7fec5c067640 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:11.910 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.900+0000 7fec632f4700 1 -- 172.21.15.110:0/243223841 >> 172.21.15.110:0/243223841 conn(0x7fec5c062ba0 msgr2=0x7fec5c062fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:31:11.910 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.900+0000 7fec632f4700 1 -- 172.21.15.110:0/243223841 shutdown_connections 2023-10-06T03:31:11.910 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.900+0000 7fec632f4700 1 -- 172.21.15.110:0/243223841 wait complete. 2023-10-06T03:31:11.910 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.904+0000 7fec632f4700 1 Processor -- start 2023-10-06T03:31:11.910 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.904+0000 7fec632f4700 1 -- start start 2023-10-06T03:31:11.910 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.904+0000 7fec632f4700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fec5c067270 0x7fec5c0b4cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:31:11.911 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.904+0000 7fec632f4700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fec5c1563d0 0x7fec5c0cb400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:31:11.911 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.904+0000 7fec632f4700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fec5c0cb940 0x7fec5c0c92c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:31:11.911 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.904+0000 7fec632f4700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec5c0c6e80 con 0x7fec5c1563d0 2023-10-06T03:31:11.911 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.904+0000 7fec632f4700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec5c0c7160 con 0x7fec5c067270 2023-10-06T03:31:11.911 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.904+0000 7fec632f4700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec5c0c7440 con 0x7fec5c0cb940 2023-10-06T03:31:11.911 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.904+0000 7fec61af1700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fec5c1563d0 0x7fec5c0cb400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:31:11.911 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.904+0000 7fec61af1700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fec5c1563d0 0x7fec5c0cb400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:54678/0 (socket says 172.21.15.110:54678) 2023-10-06T03:31:11.911 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.904+0000 7fec61af1700 1 -- 172.21.15.110:0/2503630975 learned_addr learned my addr 172.21.15.110:0/2503630975 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:31:11.911 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.904+0000 7fec62af3700 1 --2- 172.21.15.110:0/2503630975 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fec5c0cb940 0x7fec5c0c92c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:31:11.911 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.904+0000 7fec622f2700 1 --2- 172.21.15.110:0/2503630975 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fec5c067270 0x7fec5c0b4cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:31:11.911 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.904+0000 7fec61af1700 1 -- 172.21.15.110:0/2503630975 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fec5c0cb940 msgr2=0x7fec5c0c92c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:31:11.912 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.904+0000 7fec61af1700 1 --2- 172.21.15.110:0/2503630975 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fec5c0cb940 0x7fec5c0c92c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:11.912 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.904+0000 7fec61af1700 1 -- 172.21.15.110:0/2503630975 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fec5c067270 msgr2=0x7fec5c0b4cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:31:11.912 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.904+0000 7fec61af1700 1 --2- 172.21.15.110:0/2503630975 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fec5c067270 0x7fec5c0b4cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:11.912 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.904+0000 7fec61af1700 1 -- 172.21.15.110:0/2503630975 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec4c0094f0 con 0x7fec5c1563d0 2023-10-06T03:31:11.912 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.904+0000 7fec61af1700 1 --2- 172.21.15.110:0/2503630975 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fec5c1563d0 0x7fec5c0cb400 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7fec5c069eb0 tx=0x7fec4c017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:31:11.912 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.904+0000 7fec527fc700 1 -- 172.21.15.110:0/2503630975 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec4c024440 con 0x7fec5c1563d0 2023-10-06T03:31:11.912 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.904+0000 7fec527fc700 1 -- 172.21.15.110:0/2503630975 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec4c0245a0 con 0x7fec5c1563d0 2023-10-06T03:31:11.912 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.904+0000 7fec632f4700 1 -- 172.21.15.110:0/2503630975 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec5c0c8da0 con 0x7fec5c1563d0 2023-10-06T03:31:11.912 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.904+0000 7fec527fc700 1 -- 172.21.15.110:0/2503630975 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec4c024960 con 0x7fec5c1563d0 2023-10-06T03:31:11.914 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.904+0000 7fec632f4700 1 -- 172.21.15.110:0/2503630975 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec5c0b51f0 con 0x7fec5c1563d0 2023-10-06T03:31:11.914 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.904+0000 7fec632f4700 1 -- 172.21.15.110:0/2503630975 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fec5c0b5410 con 0x7fec5c1563d0 2023-10-06T03:31:11.914 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.904+0000 7fec527fc700 1 -- 172.21.15.110:0/2503630975 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 13) v1 ==== 80061+0+0 (secure 0 0 0) 0x7fec4c024d50 con 0x7fec5c1563d0 2023-10-06T03:31:11.914 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.908+0000 7fec527fc700 1 --2- 172.21.15.110:0/2503630975 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fec40060670 0x7fec40062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:31:11.914 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.908+0000 7fec527fc700 1 -- 172.21.15.110:0/2503630975 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 6171+0+0 (secure 0 0 0) 0x7fec4c07e6e0 con 0x7fec5c1563d0 2023-10-06T03:31:11.914 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.908+0000 7fec622f2700 1 --2- 172.21.15.110:0/2503630975 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fec40060670 0x7fec40062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:31:11.914 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.908+0000 7fec527fc700 1 --2- 172.21.15.110:0/2503630975 >> [v2:172.21.15.110:6808/33395,v1:172.21.15.110:6809/33395] conn(0x7fec40067500 0x7fec40069930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:31:11.915 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.908+0000 7fec527fc700 1 -- 172.21.15.110:0/2503630975 --> [v2:172.21.15.110:6808/33395,v1:172.21.15.110:6809/33395] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fec4006a000 con 0x7fec40067500 2023-10-06T03:31:11.915 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.908+0000 7fec527fc700 1 -- 172.21.15.110:0/2503630975 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_get_version_reply(handle=1 version=47) v2 ==== 24+0+0 (secure 0 0 0) 0x7fec4c080d20 con 0x7fec5c1563d0 2023-10-06T03:31:11.915 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.908+0000 7fec62af3700 1 --2- 172.21.15.110:0/2503630975 >> [v2:172.21.15.110:6808/33395,v1:172.21.15.110:6809/33395] conn(0x7fec40067500 0x7fec40069930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:31:11.915 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.908+0000 7fec622f2700 1 --2- 172.21.15.110:0/2503630975 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fec40060670 0x7fec40062b20 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fec580011e0 tx=0x7fec58009040 comp rx=0 tx=0).ready entity=mgr.5505 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:31:11.915 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.908+0000 7fec62af3700 1 --2- 172.21.15.110:0/2503630975 >> [v2:172.21.15.110:6808/33395,v1:172.21.15.110:6809/33395] conn(0x7fec40067500 0x7fec40069930 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:31:11.915 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.908+0000 7fec527fc700 1 -- 172.21.15.110:0/2503630975 <== osd.3 v2:172.21.15.110:6808/33395 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23999 (crc 0 0 0) 0x7fec4006a000 con 0x7fec40067500 2023-10-06T03:31:11.931 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.924+0000 7fec632f4700 1 -- 172.21.15.110:0/2503630975 --> [v2:172.21.15.110:6808/33395,v1:172.21.15.110:6809/33395] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7fec5c0c8bf0 con 0x7fec40067500 2023-10-06T03:31:11.931 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.928+0000 7fec527fc700 1 -- 172.21.15.110:0/2503630975 <== osd.3 v2:172.21.15.110:6808/33395 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7fec5c0c8bf0 con 0x7fec40067500 2023-10-06T03:31:11.932 INFO:teuthology.orchestra.run.smithi110.stdout:{ 2023-10-06T03:31:11.932 INFO:teuthology.orchestra.run.smithi110.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-10-06T03:31:11.932 INFO:teuthology.orchestra.run.smithi110.stdout:} 2023-10-06T03:31:11.933 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.928+0000 7fec632f4700 1 -- 172.21.15.110:0/2503630975 >> [v2:172.21.15.110:6808/33395,v1:172.21.15.110:6809/33395] conn(0x7fec40067500 msgr2=0x7fec40069930 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:31:11.933 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.928+0000 7fec632f4700 1 --2- 172.21.15.110:0/2503630975 >> [v2:172.21.15.110:6808/33395,v1:172.21.15.110:6809/33395] conn(0x7fec40067500 0x7fec40069930 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:11.933 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.928+0000 7fec632f4700 1 -- 172.21.15.110:0/2503630975 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fec40060670 msgr2=0x7fec40062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:31:11.933 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.928+0000 7fec632f4700 1 --2- 172.21.15.110:0/2503630975 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fec40060670 0x7fec40062b20 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fec580011e0 tx=0x7fec58009040 comp rx=0 tx=0).stop 2023-10-06T03:31:11.933 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.928+0000 7fec632f4700 1 -- 172.21.15.110:0/2503630975 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fec5c1563d0 msgr2=0x7fec5c0cb400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:31:11.934 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.928+0000 7fec632f4700 1 --2- 172.21.15.110:0/2503630975 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fec5c1563d0 0x7fec5c0cb400 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7fec5c069eb0 tx=0x7fec4c017000 comp rx=0 tx=0).stop 2023-10-06T03:31:11.934 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.928+0000 7fec62af3700 1 -- 172.21.15.110:0/2503630975 reap_dead start 2023-10-06T03:31:11.934 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.928+0000 7fec632f4700 1 -- 172.21.15.110:0/2503630975 shutdown_connections 2023-10-06T03:31:11.934 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.928+0000 7fec632f4700 1 -- 172.21.15.110:0/2503630975 >> 172.21.15.110:0/2503630975 conn(0x7fec5c062ba0 msgr2=0x7fec5c15c890 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:31:11.934 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.928+0000 7fec632f4700 1 -- 172.21.15.110:0/2503630975 shutdown_connections 2023-10-06T03:31:11.934 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:11.928+0000 7fec632f4700 1 -- 172.21.15.110:0/2503630975 wait complete. 2023-10-06T03:31:11.943 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2023-10-06T03:31:12.057 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.052+0000 7fcdbffff700 1 Processor -- start 2023-10-06T03:31:12.057 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.052+0000 7fcdbffff700 1 -- start start 2023-10-06T03:31:12.061 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.052+0000 7fcdbffff700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fcdc00b13a0 0x7fcdc0155aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:31:12.061 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.052+0000 7fcdbffff700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7fcdc006d770 0x7fcdc006dbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:31:12.061 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.052+0000 7fcdbffff700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcdc006e110 con 0x7fcdc01556d0 2023-10-06T03:31:12.061 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.056+0000 7fcdbffff700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcdc00b0f10 con 0x7fcdc00b13a0 2023-10-06T03:31:12.061 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.056+0000 7fcdbffff700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcdc00b00a0 con 0x7fcdc006d770 2023-10-06T03:31:12.061 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.056+0000 7fcdbe7fc700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fcdc00b13a0 0x7fcdc0155aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:31:12.061 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.056+0000 7fcdbeffd700 1 --1- >> v1:172.21.15.192:6789/0 conn(0x7fcdc01556d0 0x7fcdc00af990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.110:60660/0 (socket says 172.21.15.110:60660) 2023-10-06T03:31:12.061 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.056+0000 7fcdbeffd700 1 -- 172.21.15.110:0/2739664340 learned_addr learned my addr 172.21.15.110:0/2739664340 (peer_addr_for_me v1:172.21.15.110:0/0) 2023-10-06T03:31:12.062 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.056+0000 7fcdbf7fe700 1 --2- 172.21.15.110:0/2739664340 >> v2:172.21.15.192:3300/0 conn(0x7fcdc006d770 0x7fcdc006dbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:31:12.062 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.056+0000 7fcdbe7fc700 1 -- 172.21.15.110:0/2739664340 >> v2:172.21.15.192:3300/0 conn(0x7fcdc006d770 msgr2=0x7fcdc006dbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:31:12.062 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.056+0000 7fcdbdffb700 1 -- 172.21.15.110:0/2739664340 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 819958375 0 0) 0x7fcdb0001110 con 0x7fcdc01556d0 2023-10-06T03:31:12.062 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.056+0000 7fcdbe7fc700 1 --2- 172.21.15.110:0/2739664340 >> v2:172.21.15.192:3300/0 conn(0x7fcdc006d770 0x7fcdc006dbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:12.062 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.056+0000 7fcdbe7fc700 1 -- 172.21.15.110:0/2739664340 >> v1:172.21.15.192:6789/0 conn(0x7fcdc01556d0 legacy=0x7fcdc00af990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:31:12.062 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.056+0000 7fcdbe7fc700 1 -- 172.21.15.110:0/2739664340 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcdc00b0300 con 0x7fcdc00b13a0 2023-10-06T03:31:12.062 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.056+0000 7fcdbe7fc700 1 --2- 172.21.15.110:0/2739664340 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcdc00b13a0 0x7fcdc0155aa0 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7fcdc00b18b0 tx=0x7fcdb40091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca5a3c62aaff3c06 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:31:12.062 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.056+0000 7fcdbdffb700 1 -- 172.21.15.110:0/2739664340 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdb4011630 con 0x7fcdc00b13a0 2023-10-06T03:31:12.062 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.056+0000 7fcdbdffb700 1 -- 172.21.15.110:0/2739664340 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcdb4011790 con 0x7fcdc00b13a0 2023-10-06T03:31:12.063 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.056+0000 7fcdbdffb700 1 -- 172.21.15.110:0/2739664340 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdb4011b30 con 0x7fcdc00b13a0 2023-10-06T03:31:12.064 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.056+0000 7fcdbffff700 1 -- 172.21.15.110:0/2739664340 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcdc00b13a0 msgr2=0x7fcdc0155aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:31:12.064 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.056+0000 7fcdbffff700 1 --2- 172.21.15.110:0/2739664340 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcdc00b13a0 0x7fcdc0155aa0 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7fcdc00b18b0 tx=0x7fcdb40091a0 comp rx=0 tx=0).stop 2023-10-06T03:31:12.064 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.056+0000 7fcdbffff700 1 -- 172.21.15.110:0/2739664340 shutdown_connections 2023-10-06T03:31:12.064 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.056+0000 7fcdbffff700 1 --2- 172.21.15.110:0/2739664340 >> v2:172.21.15.192:3300/0 conn(0x7fcdc006d770 0x7fcdc006dbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:12.064 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.056+0000 7fcdbffff700 1 --2- 172.21.15.110:0/2739664340 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcdc00b13a0 0x7fcdc0155aa0 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:12.064 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.056+0000 7fcdbffff700 1 -- 172.21.15.110:0/2739664340 >> 172.21.15.110:0/2739664340 conn(0x7fcdc006b710 msgr2=0x7fcdc00adbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:31:12.064 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.056+0000 7fcdbffff700 1 -- 172.21.15.110:0/2739664340 shutdown_connections 2023-10-06T03:31:12.064 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.056+0000 7fcdbffff700 1 -- 172.21.15.110:0/2739664340 wait complete. 2023-10-06T03:31:12.065 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.060+0000 7fcdbffff700 1 Processor -- start 2023-10-06T03:31:12.065 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.060+0000 7fcdbffff700 1 -- start start 2023-10-06T03:31:12.066 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.060+0000 7fcdbffff700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcdc006d770 0x7fcdc0148d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:31:12.066 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.060+0000 7fcdbffff700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fcdc00b13a0 0x7fcdc01598c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:31:12.066 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.060+0000 7fcdbffff700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcdc01556d0 0x7fcdc00aef80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:31:12.067 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.060+0000 7fcdbffff700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdc014d3f0 con 0x7fcdc006d770 2023-10-06T03:31:12.067 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.060+0000 7fcdbffff700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdc014fd00 con 0x7fcdc01556d0 2023-10-06T03:31:12.067 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.060+0000 7fcdbffff700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcdc014ffe0 con 0x7fcdc00b13a0 2023-10-06T03:31:12.067 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.060+0000 7fcdbeffd700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcdc006d770 0x7fcdc0148d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:31:12.067 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.060+0000 7fcdbeffd700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcdc006d770 0x7fcdc0148d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:54696/0 (socket says 172.21.15.110:54696) 2023-10-06T03:31:12.067 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.060+0000 7fcdbeffd700 1 -- 172.21.15.110:0/91710025 learned_addr learned my addr 172.21.15.110:0/91710025 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:31:12.067 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.060+0000 7fcdbe7fc700 1 --2- 172.21.15.110:0/91710025 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fcdc00b13a0 0x7fcdc01598c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:31:12.067 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.060+0000 7fcdbf7fe700 1 --2- 172.21.15.110:0/91710025 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcdc01556d0 0x7fcdc00aef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:31:12.068 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.060+0000 7fcdbeffd700 1 -- 172.21.15.110:0/91710025 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fcdc00b13a0 msgr2=0x7fcdc01598c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:31:12.068 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.060+0000 7fcdbeffd700 1 --2- 172.21.15.110:0/91710025 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fcdc00b13a0 0x7fcdc01598c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:12.068 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.060+0000 7fcdbeffd700 1 -- 172.21.15.110:0/91710025 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcdc01556d0 msgr2=0x7fcdc00aef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:31:12.068 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.060+0000 7fcdbeffd700 1 --2- 172.21.15.110:0/91710025 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcdc01556d0 0x7fcdc00aef80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:12.068 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.060+0000 7fcdbeffd700 1 -- 172.21.15.110:0/91710025 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcdb4009420 con 0x7fcdc006d770 2023-10-06T03:31:12.068 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.060+0000 7fcdbeffd700 1 --2- 172.21.15.110:0/91710025 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcdc006d770 0x7fcdc0148d90 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7fcdb000c810 tx=0x7fcdb0007c50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:31:12.068 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.060+0000 7fcda6ffd700 1 -- 172.21.15.110:0/91710025 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdb0005b30 con 0x7fcdc006d770 2023-10-06T03:31:12.068 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.060+0000 7fcda6ffd700 1 -- 172.21.15.110:0/91710025 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcdb0005c90 con 0x7fcdc006d770 2023-10-06T03:31:12.068 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.060+0000 7fcdbffff700 1 -- 172.21.15.110:0/91710025 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcdc01502c0 con 0x7fcdc006d770 2023-10-06T03:31:12.069 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.060+0000 7fcda6ffd700 1 -- 172.21.15.110:0/91710025 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdb0007340 con 0x7fcdc006d770 2023-10-06T03:31:12.069 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.060+0000 7fcdbffff700 1 -- 172.21.15.110:0/91710025 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcdc0150880 con 0x7fcdc006d770 2023-10-06T03:31:12.069 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.060+0000 7fcdbffff700 1 -- 172.21.15.110:0/91710025 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcdc0150a70 con 0x7fcdc006d770 2023-10-06T03:31:12.070 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.064+0000 7fcda6ffd700 1 -- 172.21.15.110:0/91710025 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 13) v1 ==== 80061+0+0 (secure 0 0 0) 0x7fcdb001b400 con 0x7fcdc006d770 2023-10-06T03:31:12.070 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.064+0000 7fcda6ffd700 1 --2- 172.21.15.110:0/91710025 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fcdac060670 0x7fcdac062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:31:12.070 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.064+0000 7fcda6ffd700 1 -- 172.21.15.110:0/91710025 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 6171+0+0 (secure 0 0 0) 0x7fcdb007c450 con 0x7fcdc006d770 2023-10-06T03:31:12.070 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.064+0000 7fcdbe7fc700 1 --2- 172.21.15.110:0/91710025 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fcdac060670 0x7fcdac062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:31:12.070 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.064+0000 7fcdbe7fc700 1 --2- 172.21.15.110:0/91710025 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fcdac060670 0x7fcdac062b20 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fcdc00cb040 tx=0x7fcdb4017000 comp rx=0 tx=0).ready entity=mgr.5505 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:31:12.074 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.068+0000 7fcda6ffd700 1 -- 172.21.15.110:0/91710025 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fcdb004c760 con 0x7fcdc006d770 2023-10-06T03:31:12.234 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.228+0000 7fcdbffff700 1 -- 172.21.15.110:0/91710025 --> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "3", "target": ["mon-mgr", ""]}) v1 -- 0x7fcdc006e110 con 0x7fcdac060670 2023-10-06T03:31:12.234 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.228+0000 7fcda6ffd700 1 -- 172.21.15.110:0/91710025 <== mgr.5505 v2:172.21.15.110:6832/33268 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 3 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7fcdc006e110 con 0x7fcdac060670 2023-10-06T03:31:12.236 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.232+0000 7fcdbffff700 1 -- 172.21.15.110:0/91710025 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fcdac060670 msgr2=0x7fcdac062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:31:12.237 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.232+0000 7fcdbffff700 1 --2- 172.21.15.110:0/91710025 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fcdac060670 0x7fcdac062b20 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fcdc00cb040 tx=0x7fcdb4017000 comp rx=0 tx=0).stop 2023-10-06T03:31:12.237 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.232+0000 7fcdbffff700 1 -- 172.21.15.110:0/91710025 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcdc006d770 msgr2=0x7fcdc0148d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:31:12.237 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.232+0000 7fcdbffff700 1 --2- 172.21.15.110:0/91710025 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcdc006d770 0x7fcdc0148d90 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7fcdb000c810 tx=0x7fcdb0007c50 comp rx=0 tx=0).stop 2023-10-06T03:31:12.237 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.232+0000 7fcdbffff700 1 -- 172.21.15.110:0/91710025 shutdown_connections 2023-10-06T03:31:12.237 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.232+0000 7fcdbffff700 1 --2- 172.21.15.110:0/91710025 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fcdac060670 0x7fcdac062b20 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:12.237 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.232+0000 7fcdbffff700 1 --2- 172.21.15.110:0/91710025 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcdc01556d0 0x7fcdc00aef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:12.237 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.232+0000 7fcdbffff700 1 --2- 172.21.15.110:0/91710025 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fcdc00b13a0 0x7fcdc01598c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:12.237 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.232+0000 7fcdbffff700 1 --2- 172.21.15.110:0/91710025 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcdc006d770 0x7fcdc0148d90 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:12.238 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.232+0000 7fcdbffff700 1 -- 172.21.15.110:0/91710025 >> 172.21.15.110:0/91710025 conn(0x7fcdc006b710 msgr2=0x7fcdc0153980 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:31:12.238 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.232+0000 7fcdbffff700 1 -- 172.21.15.110:0/91710025 shutdown_connections 2023-10-06T03:31:12.238 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.232+0000 7fcdbffff700 1 -- 172.21.15.110:0/91710025 wait complete. 2023-10-06T03:31:12.238 INFO:teuthology.orchestra.run.smithi110.stderr:instructed osd(s) 3 to deep-scrub 2023-10-06T03:31:12.253 INFO:tasks.ceph:Scrubbing osd.4 2023-10-06T03:31:12.253 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 2023-10-06T03:31:12.367 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.360+0000 7f8a2f59e700 1 Processor -- start 2023-10-06T03:31:12.368 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.360+0000 7f8a2f59e700 1 -- start start 2023-10-06T03:31:12.369 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.364+0000 7f8a2f59e700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8a30158c40 0x7f8a300cb5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:31:12.370 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.364+0000 7f8a2f59e700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f8a3015a020 0x7f8a3006ab40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:31:12.370 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.364+0000 7f8a2f59e700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a300c8100 con 0x7f8a3015d3a0 2023-10-06T03:31:12.370 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.364+0000 7f8a2f59e700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8a300cbc50 con 0x7f8a3015a020 2023-10-06T03:31:12.370 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.364+0000 7f8a2f59e700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a300cbf30 con 0x7f8a30158c40 2023-10-06T03:31:12.370 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.364+0000 7f8a2ed9d700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f8a3015a020 0x7f8a3006ab40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:31:12.370 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.364+0000 7f8a2ed9d700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f8a3015a020 0x7f8a3006ab40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:54710/0 (socket says 172.21.15.110:54710) 2023-10-06T03:31:12.370 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.364+0000 7f8a2ed9d700 1 -- 172.21.15.110:0/2289862492 learned_addr learned my addr 172.21.15.110:0/2289862492 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:31:12.370 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.364+0000 7f8a2e59c700 1 --2- 172.21.15.110:0/2289862492 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8a30158c40 0x7f8a300cb5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:31:12.371 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.364+0000 7f8a2ed9d700 1 -- 172.21.15.110:0/2289862492 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8a30158c40 msgr2=0x7f8a300cb5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:31:12.371 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.364+0000 7f8a2ed9d700 1 --2- 172.21.15.110:0/2289862492 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8a30158c40 0x7f8a300cb5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:12.371 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.364+0000 7f8a2ed9d700 1 -- 172.21.15.110:0/2289862492 >> v1:172.21.15.192:6789/0 conn(0x7f8a3015d3a0 legacy=0x7f8a300c9f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:31:12.371 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.364+0000 7f8a2ed9d700 1 -- 172.21.15.110:0/2289862492 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a300cc210 con 0x7f8a3015a020 2023-10-06T03:31:12.371 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.364+0000 7f8a2ed9d700 1 --2- 172.21.15.110:0/2289862492 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8a3015a020 0x7f8a3006ab40 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f8a280099b0 tx=0x7f8a28009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dab9678b1997c1ac server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:31:12.371 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.364+0000 7f8a2d59a700 1 -- 172.21.15.110:0/2289862492 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a2800b860 con 0x7f8a3015a020 2023-10-06T03:31:12.371 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.364+0000 7f8a2d59a700 1 -- 172.21.15.110:0/2289862492 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a2800b9c0 con 0x7f8a3015a020 2023-10-06T03:31:12.371 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.364+0000 7f8a2d59a700 1 -- 172.21.15.110:0/2289862492 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a2800bd60 con 0x7f8a3015a020 2023-10-06T03:31:12.372 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.364+0000 7f8a2f59e700 1 -- 172.21.15.110:0/2289862492 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8a3015a020 msgr2=0x7f8a3006ab40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:31:12.372 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.364+0000 7f8a2f59e700 1 --2- 172.21.15.110:0/2289862492 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8a3015a020 0x7f8a3006ab40 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f8a280099b0 tx=0x7f8a28009250 comp rx=0 tx=0).stop 2023-10-06T03:31:12.372 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.364+0000 7f8a2f59e700 1 -- 172.21.15.110:0/2289862492 shutdown_connections 2023-10-06T03:31:12.372 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.364+0000 7f8a2f59e700 1 --2- 172.21.15.110:0/2289862492 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8a3015a020 0x7f8a3006ab40 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:12.373 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.364+0000 7f8a2f59e700 1 --2- 172.21.15.110:0/2289862492 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8a30158c40 0x7f8a300cb5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:12.373 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.364+0000 7f8a2f59e700 1 -- 172.21.15.110:0/2289862492 >> 172.21.15.110:0/2289862492 conn(0x7f8a3006c760 msgr2=0x7f8a3006cb60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:31:12.373 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.364+0000 7f8a2f59e700 1 -- 172.21.15.110:0/2289862492 shutdown_connections 2023-10-06T03:31:12.373 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.364+0000 7f8a2f59e700 1 -- 172.21.15.110:0/2289862492 wait complete. 2023-10-06T03:31:12.373 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.368+0000 7f8a2f59e700 1 Processor -- start 2023-10-06T03:31:12.374 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.368+0000 7f8a2f59e700 1 -- start start 2023-10-06T03:31:12.377 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.372+0000 7f8a2f59e700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8a30158c40 0x7f8a3016cd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:31:12.377 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.372+0000 7f8a2f59e700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8a3015a020 0x7f8a300c46e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:31:12.377 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.372+0000 7f8a2f59e700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8a3015d3a0 0x7f8a30168de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:31:12.377 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.372+0000 7f8a2f59e700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a30169d60 con 0x7f8a30158c40 2023-10-06T03:31:12.377 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.372+0000 7f8a2e59c700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8a30158c40 0x7f8a3016cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:31:12.377 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.372+0000 7f8a2f59e700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a3016c8d0 con 0x7f8a3015a020 2023-10-06T03:31:12.377 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.372+0000 7f8a2f59e700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a3016d2a0 con 0x7f8a3015d3a0 2023-10-06T03:31:12.377 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.372+0000 7f8a2e59c700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8a30158c40 0x7f8a3016cd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:54722/0 (socket says 172.21.15.110:54722) 2023-10-06T03:31:12.378 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.372+0000 7f8a2e59c700 1 -- 172.21.15.110:0/983212764 learned_addr learned my addr 172.21.15.110:0/983212764 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:31:12.378 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.372+0000 7f8a2dd9b700 1 --2- 172.21.15.110:0/983212764 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8a3015a020 0x7f8a300c46e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:31:12.378 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.372+0000 7f8a2ed9d700 1 --2- 172.21.15.110:0/983212764 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8a3015d3a0 0x7f8a30168de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:31:12.378 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.372+0000 7f8a2e59c700 1 -- 172.21.15.110:0/983212764 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8a3015d3a0 msgr2=0x7f8a30168de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:31:12.378 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.372+0000 7f8a2e59c700 1 --2- 172.21.15.110:0/983212764 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f8a3015d3a0 0x7f8a30168de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:12.378 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.372+0000 7f8a2e59c700 1 -- 172.21.15.110:0/983212764 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8a3015a020 msgr2=0x7f8a300c46e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:31:12.378 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.372+0000 7f8a2e59c700 1 --2- 172.21.15.110:0/983212764 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8a3015a020 0x7f8a300c46e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:12.378 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.372+0000 7f8a2e59c700 1 -- 172.21.15.110:0/983212764 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a280094f0 con 0x7f8a30158c40 2023-10-06T03:31:12.380 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.372+0000 7f8a2e59c700 1 --2- 172.21.15.110:0/983212764 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8a30158c40 0x7f8a3016cd60 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f8a2000e190 tx=0x7f8a20009920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:31:12.380 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.372+0000 7f8a267fc700 1 -- 172.21.15.110:0/983212764 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a2000ecd0 con 0x7f8a30158c40 2023-10-06T03:31:12.380 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.372+0000 7f8a267fc700 1 -- 172.21.15.110:0/983212764 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a2000ee30 con 0x7f8a30158c40 2023-10-06T03:31:12.380 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.372+0000 7f8a2f59e700 1 -- 172.21.15.110:0/983212764 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a301697a0 con 0x7f8a30158c40 2023-10-06T03:31:12.380 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.372+0000 7f8a267fc700 1 -- 172.21.15.110:0/983212764 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a20014670 con 0x7f8a30158c40 2023-10-06T03:31:12.380 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.372+0000 7f8a2f59e700 1 -- 172.21.15.110:0/983212764 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a3016c560 con 0x7f8a30158c40 2023-10-06T03:31:12.380 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.372+0000 7f8a2f59e700 1 -- 172.21.15.110:0/983212764 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f8a30064770 con 0x7f8a30158c40 2023-10-06T03:31:12.381 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.376+0000 7f8a267fc700 1 -- 172.21.15.110:0/983212764 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 13) v1 ==== 80061+0+0 (secure 0 0 0) 0x7f8a20014a20 con 0x7f8a30158c40 2023-10-06T03:31:12.381 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.376+0000 7f8a267fc700 1 --2- 172.21.15.110:0/983212764 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f8a14060670 0x7f8a14062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:31:12.382 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.376+0000 7f8a2dd9b700 1 --2- 172.21.15.110:0/983212764 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f8a14060670 0x7f8a14062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:31:12.382 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.376+0000 7f8a267fc700 1 -- 172.21.15.110:0/983212764 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 6171+0+0 (secure 0 0 0) 0x7f8a2007d220 con 0x7f8a30158c40 2023-10-06T03:31:12.382 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.376+0000 7f8a267fc700 1 --2- 172.21.15.110:0/983212764 >> [v2:172.21.15.192:6816/33683,v1:172.21.15.192:6817/33683] conn(0x7f8a14067500 0x7f8a14069930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:31:12.382 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.376+0000 7f8a267fc700 1 -- 172.21.15.110:0/983212764 --> [v2:172.21.15.192:6816/33683,v1:172.21.15.192:6817/33683] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8a1406a000 con 0x7f8a14067500 2023-10-06T03:31:12.382 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.376+0000 7f8a267fc700 1 -- 172.21.15.110:0/983212764 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_get_version_reply(handle=1 version=47) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8a2007f780 con 0x7f8a30158c40 2023-10-06T03:31:12.382 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.376+0000 7f8a2dd9b700 1 --2- 172.21.15.110:0/983212764 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f8a14060670 0x7f8a14062b20 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f8a30158bb0 tx=0x7f8a18007ec0 comp rx=0 tx=0).ready entity=mgr.5505 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:31:12.382 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.376+0000 7f8a2ed9d700 1 --2- 172.21.15.110:0/983212764 >> [v2:172.21.15.192:6816/33683,v1:172.21.15.192:6817/33683] conn(0x7f8a14067500 0x7f8a14069930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:31:12.382 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.376+0000 7f8a2ed9d700 1 --2- 172.21.15.110:0/983212764 >> [v2:172.21.15.192:6816/33683,v1:172.21.15.192:6817/33683] conn(0x7f8a14067500 0x7f8a14069930 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:31:12.383 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.380+0000 7f8a267fc700 1 -- 172.21.15.110:0/983212764 <== osd.4 v2:172.21.15.192:6816/33683 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23999 (crc 0 0 0) 0x7f8a1406a000 con 0x7f8a14067500 2023-10-06T03:31:12.402 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.396+0000 7f8a2f59e700 1 -- 172.21.15.110:0/983212764 --> [v2:172.21.15.192:6816/33683,v1:172.21.15.192:6817/33683] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f8a300cc4f0 con 0x7f8a14067500 2023-10-06T03:31:12.402 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.396+0000 7f8a267fc700 1 -- 172.21.15.110:0/983212764 <== osd.4 v2:172.21.15.192:6816/33683 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7f8a300cc4f0 con 0x7f8a14067500 2023-10-06T03:31:12.403 INFO:teuthology.orchestra.run.smithi110.stdout:{ 2023-10-06T03:31:12.403 INFO:teuthology.orchestra.run.smithi110.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-10-06T03:31:12.403 INFO:teuthology.orchestra.run.smithi110.stdout:} 2023-10-06T03:31:12.404 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.396+0000 7f8a2f59e700 1 -- 172.21.15.110:0/983212764 >> [v2:172.21.15.192:6816/33683,v1:172.21.15.192:6817/33683] conn(0x7f8a14067500 msgr2=0x7f8a14069930 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:31:12.404 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.396+0000 7f8a2f59e700 1 --2- 172.21.15.110:0/983212764 >> [v2:172.21.15.192:6816/33683,v1:172.21.15.192:6817/33683] conn(0x7f8a14067500 0x7f8a14069930 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:12.404 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.396+0000 7f8a2f59e700 1 -- 172.21.15.110:0/983212764 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f8a14060670 msgr2=0x7f8a14062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:31:12.404 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.396+0000 7f8a2f59e700 1 --2- 172.21.15.110:0/983212764 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f8a14060670 0x7f8a14062b20 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f8a30158bb0 tx=0x7f8a18007ec0 comp rx=0 tx=0).stop 2023-10-06T03:31:12.404 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.396+0000 7f8a2f59e700 1 -- 172.21.15.110:0/983212764 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8a30158c40 msgr2=0x7f8a3016cd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:31:12.405 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.396+0000 7f8a2f59e700 1 --2- 172.21.15.110:0/983212764 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8a30158c40 0x7f8a3016cd60 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f8a2000e190 tx=0x7f8a20009920 comp rx=0 tx=0).stop 2023-10-06T03:31:12.405 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.400+0000 7f8a2ed9d700 1 -- 172.21.15.110:0/983212764 reap_dead start 2023-10-06T03:31:12.405 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.400+0000 7f8a2f59e700 1 -- 172.21.15.110:0/983212764 shutdown_connections 2023-10-06T03:31:12.405 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.400+0000 7f8a2f59e700 1 -- 172.21.15.110:0/983212764 >> 172.21.15.110:0/983212764 conn(0x7f8a3006c760 msgr2=0x7f8a3015cb20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:31:12.405 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.400+0000 7f8a2f59e700 1 -- 172.21.15.110:0/983212764 shutdown_connections 2023-10-06T03:31:12.405 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.400+0000 7f8a2f59e700 1 -- 172.21.15.110:0/983212764 wait complete. 2023-10-06T03:31:12.415 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4 2023-10-06T03:31:12.529 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.524+0000 7facd5486700 1 Processor -- start 2023-10-06T03:31:12.530 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.524+0000 7facd5486700 1 -- start start 2023-10-06T03:31:12.531 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.524+0000 7facd5486700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7facd0155c90 0x7facd00c7300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:31:12.531 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.524+0000 7facd5486700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7facd006a830 con 0x7facd0156060 2023-10-06T03:31:12.531 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.524+0000 7facd5486700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7facd006a4e0 con 0x7facd0159a30 2023-10-06T03:31:12.531 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.524+0000 7facd5486700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7facd00c7c50 con 0x7facd0155c90 2023-10-06T03:31:12.531 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.524+0000 7facceffd700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7facd0155c90 0x7facd00c7300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:31:12.531 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.524+0000 7facc7fff700 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7facd0156060 0x7facd006acc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:48144/0 (socket says 172.21.15.110:48144) 2023-10-06T03:31:12.532 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.524+0000 7facc7fff700 1 -- 172.21.15.110:0/4176784118 learned_addr learned my addr 172.21.15.110:0/4176784118 (peer_addr_for_me v1:172.21.15.110:0/0) 2023-10-06T03:31:12.532 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.528+0000 7facceffd700 1 -- 172.21.15.110:0/4176784118 >> v1:172.21.15.192:6789/0 conn(0x7facd0159a30 legacy=0x7facd00c60e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:31:12.533 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.528+0000 7facceffd700 1 -- 172.21.15.110:0/4176784118 >> v1:172.21.15.110:6789/0 conn(0x7facd0156060 legacy=0x7facd006acc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:31:12.533 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.528+0000 7facceffd700 1 -- 172.21.15.110:0/4176784118 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facd00c7f30 con 0x7facd0155c90 2023-10-06T03:31:12.533 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.528+0000 7facceffd700 1 --2- 172.21.15.110:0/4176784118 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7facd0155c90 0x7facd00c7300 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7facc00098e0 tx=0x7facc00091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=786129b2046de72d server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:31:12.533 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.528+0000 7facce7fc700 1 -- 172.21.15.110:0/4176784118 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facc0009e80 con 0x7facd0155c90 2023-10-06T03:31:12.533 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.528+0000 7facce7fc700 1 -- 172.21.15.110:0/4176784118 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7facc000b860 con 0x7facd0155c90 2023-10-06T03:31:12.533 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.528+0000 7facce7fc700 1 -- 172.21.15.110:0/4176784118 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facc000bbe0 con 0x7facd0155c90 2023-10-06T03:31:12.534 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.528+0000 7facd5486700 1 -- 172.21.15.110:0/4176784118 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7facd0155c90 msgr2=0x7facd00c7300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:31:12.534 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.528+0000 7facd5486700 1 --2- 172.21.15.110:0/4176784118 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7facd0155c90 0x7facd00c7300 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7facc00098e0 tx=0x7facc00091a0 comp rx=0 tx=0).stop 2023-10-06T03:31:12.534 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.528+0000 7facd5486700 1 -- 172.21.15.110:0/4176784118 shutdown_connections 2023-10-06T03:31:12.534 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.528+0000 7facd5486700 1 --2- 172.21.15.110:0/4176784118 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7facd0155c90 0x7facd00c7300 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:12.535 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.528+0000 7facd5486700 1 -- 172.21.15.110:0/4176784118 >> 172.21.15.110:0/4176784118 conn(0x7facd0067040 msgr2=0x7facd0067410 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:31:12.535 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.528+0000 7facd5486700 1 -- 172.21.15.110:0/4176784118 shutdown_connections 2023-10-06T03:31:12.535 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.528+0000 7facd5486700 1 -- 172.21.15.110:0/4176784118 wait complete. 2023-10-06T03:31:12.535 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.532+0000 7facd5486700 1 Processor -- start 2023-10-06T03:31:12.536 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.532+0000 7facd5486700 1 -- start start 2023-10-06T03:31:12.538 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.532+0000 7facd5486700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7facd0155c90 0x7facd0157dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:31:12.538 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.532+0000 7facd5486700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7facd0159a30 0x7facd014bdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:31:12.539 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.532+0000 7facd5486700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7facd00b0cb0 0x7facd00b1130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:31:12.539 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.532+0000 7facd5486700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facd00b1670 con 0x7facd00b0cb0 2023-10-06T03:31:12.539 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.532+0000 7facd5486700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facd0169ad0 con 0x7facd0155c90 2023-10-06T03:31:12.539 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.532+0000 7facd5486700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7facd0158310 con 0x7facd0159a30 2023-10-06T03:31:12.539 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.532+0000 7faccf7fe700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7facd00b0cb0 0x7facd00b1130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:31:12.539 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.532+0000 7faccf7fe700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7facd00b0cb0 0x7facd00b1130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:54748/0 (socket says 172.21.15.110:54748) 2023-10-06T03:31:12.539 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.532+0000 7faccf7fe700 1 -- 172.21.15.110:0/1279167144 learned_addr learned my addr 172.21.15.110:0/1279167144 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:31:12.539 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.532+0000 7facceffd700 1 --2- 172.21.15.110:0/1279167144 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7facd0155c90 0x7facd0157dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:31:12.539 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.532+0000 7facc7fff700 1 --2- 172.21.15.110:0/1279167144 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7facd0159a30 0x7facd014bdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:31:12.540 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.532+0000 7faccf7fe700 1 -- 172.21.15.110:0/1279167144 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7facd0159a30 msgr2=0x7facd014bdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:31:12.540 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.532+0000 7faccf7fe700 1 --2- 172.21.15.110:0/1279167144 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7facd0159a30 0x7facd014bdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:12.540 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.532+0000 7faccf7fe700 1 -- 172.21.15.110:0/1279167144 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7facd0155c90 msgr2=0x7facd0157dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:31:12.540 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.532+0000 7faccf7fe700 1 --2- 172.21.15.110:0/1279167144 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7facd0155c90 0x7facd0157dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:12.540 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.532+0000 7faccf7fe700 1 -- 172.21.15.110:0/1279167144 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facc0009420 con 0x7facd00b0cb0 2023-10-06T03:31:12.540 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.532+0000 7faccf7fe700 1 --2- 172.21.15.110:0/1279167144 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7facd00b0cb0 0x7facd00b1130 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7facc800c810 tx=0x7facc8001650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:31:12.541 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.532+0000 7facc6ffd700 1 -- 172.21.15.110:0/1279167144 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facc8012630 con 0x7facd00b0cb0 2023-10-06T03:31:12.541 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.532+0000 7facc6ffd700 1 -- 172.21.15.110:0/1279167144 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7facc80127b0 con 0x7facd00b0cb0 2023-10-06T03:31:12.541 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.532+0000 7facd5486700 1 -- 172.21.15.110:0/1279167144 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7facd0168ec0 con 0x7facd00b0cb0 2023-10-06T03:31:12.541 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.532+0000 7facc6ffd700 1 -- 172.21.15.110:0/1279167144 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facc8012b70 con 0x7facd00b0cb0 2023-10-06T03:31:12.541 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.532+0000 7facd5486700 1 -- 172.21.15.110:0/1279167144 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7facd0169480 con 0x7facd00b0cb0 2023-10-06T03:31:12.542 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.536+0000 7facd5486700 1 -- 172.21.15.110:0/1279167144 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7facd00c07c0 con 0x7facd00b0cb0 2023-10-06T03:31:12.542 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.536+0000 7facc6ffd700 1 -- 172.21.15.110:0/1279167144 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 13) v1 ==== 80061+0+0 (secure 0 0 0) 0x7facc8003030 con 0x7facd00b0cb0 2023-10-06T03:31:12.542 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.536+0000 7facc6ffd700 1 --2- 172.21.15.110:0/1279167144 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7facb00606a0 0x7facb0062b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:31:12.543 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.536+0000 7facc6ffd700 1 -- 172.21.15.110:0/1279167144 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 6171+0+0 (secure 0 0 0) 0x7facc807c920 con 0x7facd00b0cb0 2023-10-06T03:31:12.543 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.536+0000 7facceffd700 1 --2- 172.21.15.110:0/1279167144 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7facb00606a0 0x7facb0062b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:31:12.543 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.540+0000 7facceffd700 1 --2- 172.21.15.110:0/1279167144 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7facb00606a0 0x7facb0062b50 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7facc00093f0 tx=0x7facc0019000 comp rx=0 tx=0).ready entity=mgr.5505 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:31:12.550 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.544+0000 7facc6ffd700 1 -- 172.21.15.110:0/1279167144 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7facd00c07c0 con 0x7facd00b0cb0 2023-10-06T03:31:12.714 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.708+0000 7facd5486700 1 -- 172.21.15.110:0/1279167144 --> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "4", "target": ["mon-mgr", ""]}) v1 -- 0x7facd00c07c0 con 0x7facb00606a0 2023-10-06T03:31:12.714 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.708+0000 7facc6ffd700 1 -- 172.21.15.110:0/1279167144 <== mgr.5505 v2:172.21.15.110:6832/33268 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 4 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7facd00c07c0 con 0x7facb00606a0 2023-10-06T03:31:12.716 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.712+0000 7facd5486700 1 -- 172.21.15.110:0/1279167144 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7facb00606a0 msgr2=0x7facb0062b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:31:12.717 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.712+0000 7facd5486700 1 --2- 172.21.15.110:0/1279167144 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7facb00606a0 0x7facb0062b50 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7facc00093f0 tx=0x7facc0019000 comp rx=0 tx=0).stop 2023-10-06T03:31:12.717 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.712+0000 7facd5486700 1 -- 172.21.15.110:0/1279167144 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7facd00b0cb0 msgr2=0x7facd00b1130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:31:12.717 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.712+0000 7facd5486700 1 --2- 172.21.15.110:0/1279167144 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7facd00b0cb0 0x7facd00b1130 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7facc800c810 tx=0x7facc8001650 comp rx=0 tx=0).stop 2023-10-06T03:31:12.717 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.712+0000 7facd5486700 1 -- 172.21.15.110:0/1279167144 shutdown_connections 2023-10-06T03:31:12.717 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.712+0000 7facd5486700 1 --2- 172.21.15.110:0/1279167144 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7facb00606a0 0x7facb0062b50 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:12.717 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.712+0000 7facd5486700 1 --2- 172.21.15.110:0/1279167144 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7facd00b0cb0 0x7facd00b1130 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:12.717 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.712+0000 7facd5486700 1 --2- 172.21.15.110:0/1279167144 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7facd0159a30 0x7facd014bdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:12.717 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.712+0000 7facd5486700 1 --2- 172.21.15.110:0/1279167144 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7facd0155c90 0x7facd0157dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:12.717 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.712+0000 7facd5486700 1 -- 172.21.15.110:0/1279167144 >> 172.21.15.110:0/1279167144 conn(0x7facd0067040 msgr2=0x7facd01540d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:31:12.717 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.712+0000 7facd5486700 1 -- 172.21.15.110:0/1279167144 shutdown_connections 2023-10-06T03:31:12.717 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.712+0000 7facd5486700 1 -- 172.21.15.110:0/1279167144 wait complete. 2023-10-06T03:31:12.718 INFO:teuthology.orchestra.run.smithi110.stderr:instructed osd(s) 4 to deep-scrub 2023-10-06T03:31:12.731 INFO:tasks.ceph:Scrubbing osd.5 2023-10-06T03:31:12.732 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 config set osd_debug_deep_scrub_sleep 0 2023-10-06T03:31:12.844 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.840+0000 7f4b5eda9700 1 Processor -- start 2023-10-06T03:31:12.844 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.840+0000 7f4b5eda9700 1 -- start start 2023-10-06T03:31:12.846 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.840+0000 7f4b5eda9700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f4b580b4eb0 0x7f4b580b5280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:31:12.846 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.840+0000 7f4b5eda9700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f4b58067a50 0x7f4b5815c940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:31:12.847 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.840+0000 7f4b5eda9700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b5806a3e0 con 0x7f4b5815ce80 2023-10-06T03:31:12.847 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.840+0000 7f4b5eda9700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4b5806a6a0 con 0x7f4b580b4eb0 2023-10-06T03:31:12.847 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.840+0000 7f4b5eda9700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4b5806a980 con 0x7f4b58067a50 2023-10-06T03:31:12.847 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.840+0000 7f4b5dda7700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f4b580b4eb0 0x7f4b580b5280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:31:12.847 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.840+0000 7f4b5dda7700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f4b580b4eb0 0x7f4b580b5280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:54750/0 (socket says 172.21.15.110:54750) 2023-10-06T03:31:12.847 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.840+0000 7f4b5dda7700 1 -- 172.21.15.110:0/779488889 learned_addr learned my addr 172.21.15.110:0/779488889 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:31:12.847 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.840+0000 7f4b5d5a6700 1 --2- 172.21.15.110:0/779488889 >> v2:172.21.15.192:3300/0 conn(0x7f4b58067a50 0x7f4b5815c940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:31:12.848 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.840+0000 7f4b5dda7700 1 -- 172.21.15.110:0/779488889 >> v2:172.21.15.192:3300/0 conn(0x7f4b58067a50 msgr2=0x7f4b5815c940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:31:12.848 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.840+0000 7f4b5dda7700 1 --2- 172.21.15.110:0/779488889 >> v2:172.21.15.192:3300/0 conn(0x7f4b58067a50 0x7f4b5815c940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:12.848 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.840+0000 7f4b5dda7700 1 -- 172.21.15.110:0/779488889 >> v1:172.21.15.192:6789/0 conn(0x7f4b5815ce80 legacy=0x7f4b58158730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:31:12.848 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.840+0000 7f4b5dda7700 1 -- 172.21.15.110:0/779488889 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b5806ac60 con 0x7f4b580b4eb0 2023-10-06T03:31:12.848 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.840+0000 7f4b5dda7700 1 --2- 172.21.15.110:0/779488889 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4b580b4eb0 0x7f4b580b5280 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f4b580b5b40 tx=0x7f4b54009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9dabeae60ce84768 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:31:12.848 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.840+0000 7f4b5cda5700 1 -- 172.21.15.110:0/779488889 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b54009e70 con 0x7f4b580b4eb0 2023-10-06T03:31:12.848 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.840+0000 7f4b5cda5700 1 -- 172.21.15.110:0/779488889 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b5400b860 con 0x7f4b580b4eb0 2023-10-06T03:31:12.849 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.840+0000 7f4b5cda5700 1 -- 172.21.15.110:0/779488889 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b5400bc00 con 0x7f4b580b4eb0 2023-10-06T03:31:12.850 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.844+0000 7f4b5eda9700 1 -- 172.21.15.110:0/779488889 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4b580b4eb0 msgr2=0x7f4b580b5280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:31:12.850 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.844+0000 7f4b5eda9700 1 --2- 172.21.15.110:0/779488889 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4b580b4eb0 0x7f4b580b5280 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f4b580b5b40 tx=0x7f4b54009250 comp rx=0 tx=0).stop 2023-10-06T03:31:12.850 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.844+0000 7f4b5eda9700 1 -- 172.21.15.110:0/779488889 shutdown_connections 2023-10-06T03:31:12.850 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.844+0000 7f4b5eda9700 1 --2- 172.21.15.110:0/779488889 >> v2:172.21.15.192:3300/0 conn(0x7f4b58067a50 0x7f4b5815c940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:12.850 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.844+0000 7f4b5eda9700 1 --2- 172.21.15.110:0/779488889 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4b580b4eb0 0x7f4b580b5280 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:12.850 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.844+0000 7f4b5eda9700 1 -- 172.21.15.110:0/779488889 >> 172.21.15.110:0/779488889 conn(0x7f4b5806c760 msgr2=0x7f4b5806cb60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:31:12.850 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.844+0000 7f4b5eda9700 1 -- 172.21.15.110:0/779488889 shutdown_connections 2023-10-06T03:31:12.850 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.844+0000 7f4b5eda9700 1 -- 172.21.15.110:0/779488889 wait complete. 2023-10-06T03:31:12.851 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.844+0000 7f4b5eda9700 1 Processor -- start 2023-10-06T03:31:12.851 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.844+0000 7f4b5eda9700 1 -- start start 2023-10-06T03:31:12.853 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.848+0000 7f4b5eda9700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4b58067a50 0x7f4b58150430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:31:12.853 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.848+0000 7f4b5eda9700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4b580b4eb0 0x7f4b5816d590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:31:12.853 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.848+0000 7f4b5eda9700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4b5815ce80 0x7f4b58154e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:31:12.853 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.848+0000 7f4b5eda9700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b5816a860 con 0x7f4b5815ce80 2023-10-06T03:31:12.853 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.848+0000 7f4b5eda9700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b5816dad0 con 0x7f4b58067a50 2023-10-06T03:31:12.854 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.848+0000 7f4b5eda9700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b58150aa0 con 0x7f4b580b4eb0 2023-10-06T03:31:12.854 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.848+0000 7f4b5e5a8700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4b5815ce80 0x7f4b58154e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:31:12.854 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.848+0000 7f4b5e5a8700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4b5815ce80 0x7f4b58154e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:54760/0 (socket says 172.21.15.110:54760) 2023-10-06T03:31:12.854 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.848+0000 7f4b5e5a8700 1 -- 172.21.15.110:0/3359180414 learned_addr learned my addr 172.21.15.110:0/3359180414 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:31:12.854 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.848+0000 7f4b5d5a6700 1 --2- 172.21.15.110:0/3359180414 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4b580b4eb0 0x7f4b5816d590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:31:12.854 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.848+0000 7f4b5dda7700 1 --2- 172.21.15.110:0/3359180414 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4b58067a50 0x7f4b58150430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:31:12.854 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.848+0000 7f4b5e5a8700 1 -- 172.21.15.110:0/3359180414 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4b580b4eb0 msgr2=0x7f4b5816d590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:31:12.854 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.848+0000 7f4b5e5a8700 1 --2- 172.21.15.110:0/3359180414 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4b580b4eb0 0x7f4b5816d590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:12.854 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.848+0000 7f4b5e5a8700 1 -- 172.21.15.110:0/3359180414 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4b58067a50 msgr2=0x7f4b58150430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:31:12.854 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.848+0000 7f4b5e5a8700 1 --2- 172.21.15.110:0/3359180414 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4b58067a50 0x7f4b58150430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:12.855 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.848+0000 7f4b5e5a8700 1 -- 172.21.15.110:0/3359180414 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b540094f0 con 0x7f4b5815ce80 2023-10-06T03:31:12.855 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.848+0000 7f4b5d5a6700 1 --2- 172.21.15.110:0/3359180414 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f4b580b4eb0 0x7f4b5816d590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-06T03:31:12.855 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.848+0000 7f4b5e5a8700 1 --2- 172.21.15.110:0/3359180414 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4b5815ce80 0x7f4b58154e90 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f4b5000c840 tx=0x7f4b50001670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:31:12.855 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.848+0000 7f4b4dffb700 1 -- 172.21.15.110:0/3359180414 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b50012630 con 0x7f4b5815ce80 2023-10-06T03:31:12.855 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.848+0000 7f4b4dffb700 1 -- 172.21.15.110:0/3359180414 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b500127b0 con 0x7f4b5815ce80 2023-10-06T03:31:12.855 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.848+0000 7f4b5eda9700 1 -- 172.21.15.110:0/3359180414 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b58150d80 con 0x7f4b5815ce80 2023-10-06T03:31:12.855 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.848+0000 7f4b4dffb700 1 -- 172.21.15.110:0/3359180414 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b50012b70 con 0x7f4b5815ce80 2023-10-06T03:31:12.855 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.848+0000 7f4b5eda9700 1 -- 172.21.15.110:0/3359180414 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b58151340 con 0x7f4b5815ce80 2023-10-06T03:31:12.856 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.848+0000 7f4b5eda9700 1 -- 172.21.15.110:0/3359180414 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f4b58069590 con 0x7f4b5815ce80 2023-10-06T03:31:12.856 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.852+0000 7f4b4dffb700 1 -- 172.21.15.110:0/3359180414 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 13) v1 ==== 80061+0+0 (secure 0 0 0) 0x7f4b50003030 con 0x7f4b5815ce80 2023-10-06T03:31:12.857 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.852+0000 7f4b4dffb700 1 --2- 172.21.15.110:0/3359180414 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f4b3c060670 0x7f4b3c062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:31:12.858 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.852+0000 7f4b4dffb700 1 -- 172.21.15.110:0/3359180414 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 6171+0+0 (secure 0 0 0) 0x7f4b5007d220 con 0x7f4b5815ce80 2023-10-06T03:31:12.858 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.852+0000 7f4b5dda7700 1 --2- 172.21.15.110:0/3359180414 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f4b3c060670 0x7f4b3c062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:31:12.858 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.852+0000 7f4b4dffb700 1 --2- 172.21.15.110:0/3359180414 >> [v2:172.21.15.192:6800/33684,v1:172.21.15.192:6801/33684] conn(0x7f4b3c067500 0x7f4b3c069930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:31:12.858 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.852+0000 7f4b4dffb700 1 -- 172.21.15.110:0/3359180414 --> [v2:172.21.15.192:6800/33684,v1:172.21.15.192:6801/33684] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4b3c06a000 con 0x7f4b3c067500 2023-10-06T03:31:12.858 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.852+0000 7f4b4dffb700 1 -- 172.21.15.110:0/3359180414 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_get_version_reply(handle=1 version=47) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4b5007d510 con 0x7f4b5815ce80 2023-10-06T03:31:12.858 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.852+0000 7f4b5d5a6700 1 --2- 172.21.15.110:0/3359180414 >> [v2:172.21.15.192:6800/33684,v1:172.21.15.192:6801/33684] conn(0x7f4b3c067500 0x7f4b3c069930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:31:12.859 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.852+0000 7f4b5dda7700 1 --2- 172.21.15.110:0/3359180414 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f4b3c060670 0x7f4b3c062b20 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f4b580b5b40 tx=0x7f4b54017000 comp rx=0 tx=0).ready entity=mgr.5505 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:31:12.859 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.852+0000 7f4b5d5a6700 1 --2- 172.21.15.110:0/3359180414 >> [v2:172.21.15.192:6800/33684,v1:172.21.15.192:6801/33684] conn(0x7f4b3c067500 0x7f4b3c069930 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:31:12.860 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.856+0000 7f4b4dffb700 1 -- 172.21.15.110:0/3359180414 <== osd.5 v2:172.21.15.192:6800/33684 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23999 (crc 0 0 0) 0x7f4b3c06a000 con 0x7f4b3c067500 2023-10-06T03:31:12.878 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.872+0000 7f4b5eda9700 1 -- 172.21.15.110:0/3359180414 --> [v2:172.21.15.192:6800/33684,v1:172.21.15.192:6801/33684] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f4b5806af40 con 0x7f4b3c067500 2023-10-06T03:31:12.879 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.872+0000 7f4b4dffb700 1 -- 172.21.15.110:0/3359180414 <== osd.5 v2:172.21.15.192:6800/33684 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7f4b5806af40 con 0x7f4b3c067500 2023-10-06T03:31:12.879 INFO:teuthology.orchestra.run.smithi110.stdout:{ 2023-10-06T03:31:12.879 INFO:teuthology.orchestra.run.smithi110.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-10-06T03:31:12.879 INFO:teuthology.orchestra.run.smithi110.stdout:} 2023-10-06T03:31:12.880 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.876+0000 7f4b5eda9700 1 -- 172.21.15.110:0/3359180414 >> [v2:172.21.15.192:6800/33684,v1:172.21.15.192:6801/33684] conn(0x7f4b3c067500 msgr2=0x7f4b3c069930 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:31:12.880 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.876+0000 7f4b5eda9700 1 --2- 172.21.15.110:0/3359180414 >> [v2:172.21.15.192:6800/33684,v1:172.21.15.192:6801/33684] conn(0x7f4b3c067500 0x7f4b3c069930 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:12.881 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.876+0000 7f4b5eda9700 1 -- 172.21.15.110:0/3359180414 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f4b3c060670 msgr2=0x7f4b3c062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:31:12.881 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.876+0000 7f4b5eda9700 1 --2- 172.21.15.110:0/3359180414 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f4b3c060670 0x7f4b3c062b20 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f4b580b5b40 tx=0x7f4b54017000 comp rx=0 tx=0).stop 2023-10-06T03:31:12.881 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.876+0000 7f4b5eda9700 1 -- 172.21.15.110:0/3359180414 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4b5815ce80 msgr2=0x7f4b58154e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:31:12.881 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.876+0000 7f4b5eda9700 1 --2- 172.21.15.110:0/3359180414 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4b5815ce80 0x7f4b58154e90 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f4b5000c840 tx=0x7f4b50001670 comp rx=0 tx=0).stop 2023-10-06T03:31:12.881 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.876+0000 7f4b5e5a8700 1 -- 172.21.15.110:0/3359180414 reap_dead start 2023-10-06T03:31:12.881 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.876+0000 7f4b5eda9700 1 -- 172.21.15.110:0/3359180414 shutdown_connections 2023-10-06T03:31:12.881 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.876+0000 7f4b5eda9700 1 -- 172.21.15.110:0/3359180414 >> 172.21.15.110:0/3359180414 conn(0x7f4b5806c760 msgr2=0x7f4b5814e920 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:31:12.881 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.876+0000 7f4b5eda9700 1 -- 172.21.15.110:0/3359180414 shutdown_connections 2023-10-06T03:31:12.881 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:12.876+0000 7f4b5eda9700 1 -- 172.21.15.110:0/3359180414 wait complete. 2023-10-06T03:31:12.892 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 5 2023-10-06T03:31:13.005 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.000+0000 7fea78dd0700 1 Processor -- start 2023-10-06T03:31:13.005 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.000+0000 7fea78dd0700 1 -- start start 2023-10-06T03:31:13.006 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.000+0000 7fea78dd0700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fea7415d100 0x7fea740c97a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:31:13.006 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.000+0000 7fea78dd0700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea740b08a0 con 0x7fea740ad890 2023-10-06T03:31:13.006 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.000+0000 7fea78dd0700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea740b0b80 con 0x7fea740accc0 2023-10-06T03:31:13.007 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.000+0000 7fea78dd0700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea740b1520 con 0x7fea7415d100 2023-10-06T03:31:13.007 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.000+0000 7fea73fff700 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7fea740ad890 0x7fea74160e80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:48154/0 (socket says 172.21.15.110:48154) 2023-10-06T03:31:13.007 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.000+0000 7fea73fff700 1 -- 172.21.15.110:0/2959917230 learned_addr learned my addr 172.21.15.110:0/2959917230 (peer_addr_for_me v1:172.21.15.110:0/0) 2023-10-06T03:31:13.007 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.000+0000 7fea737fe700 1 --2- 172.21.15.110:0/2959917230 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fea7415d100 0x7fea740c97a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:31:13.008 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.000+0000 7fea727fc700 1 -- 172.21.15.110:0/2959917230 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 367545840 0 0) 0x7fea740b08a0 con 0x7fea740ad890 2023-10-06T03:31:13.008 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.000+0000 7fea727fc700 1 -- 172.21.15.110:0/2959917230 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fea5c003570 con 0x7fea740ad890 2023-10-06T03:31:13.008 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.000+0000 7fea727fc700 1 -- 172.21.15.110:0/2959917230 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3289884255 0 0) 0x7fea740b0b80 con 0x7fea740accc0 2023-10-06T03:31:13.008 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.000+0000 7fea737fe700 1 -- 172.21.15.110:0/2959917230 >> v1:172.21.15.192:6789/0 conn(0x7fea740accc0 legacy=0x7fea740c7af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:31:13.008 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.000+0000 7fea737fe700 1 -- 172.21.15.110:0/2959917230 >> v1:172.21.15.110:6789/0 conn(0x7fea740ad890 legacy=0x7fea74160e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:31:13.009 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.000+0000 7fea737fe700 1 -- 172.21.15.110:0/2959917230 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea740b0e60 con 0x7fea7415d100 2023-10-06T03:31:13.009 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.004+0000 7fea737fe700 1 --2- 172.21.15.110:0/2959917230 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fea7415d100 0x7fea740c97a0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fea6c009e90 tx=0x7fea6c009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b4dfdd54d6fb404b server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:31:13.009 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.004+0000 7fea727fc700 1 -- 172.21.15.110:0/2959917230 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea6c00e070 con 0x7fea7415d100 2023-10-06T03:31:13.009 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.004+0000 7fea727fc700 1 -- 172.21.15.110:0/2959917230 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea6c00bc30 con 0x7fea7415d100 2023-10-06T03:31:13.009 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.004+0000 7fea727fc700 1 -- 172.21.15.110:0/2959917230 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea6c012430 con 0x7fea7415d100 2023-10-06T03:31:13.010 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.004+0000 7fea78dd0700 1 -- 172.21.15.110:0/2959917230 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fea7415d100 msgr2=0x7fea740c97a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:31:13.010 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.004+0000 7fea78dd0700 1 --2- 172.21.15.110:0/2959917230 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fea7415d100 0x7fea740c97a0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fea6c009e90 tx=0x7fea6c009780 comp rx=0 tx=0).stop 2023-10-06T03:31:13.010 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.004+0000 7fea78dd0700 1 -- 172.21.15.110:0/2959917230 shutdown_connections 2023-10-06T03:31:13.010 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.004+0000 7fea78dd0700 1 --2- 172.21.15.110:0/2959917230 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fea7415d100 0x7fea740c97a0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:13.010 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.004+0000 7fea78dd0700 1 -- 172.21.15.110:0/2959917230 >> 172.21.15.110:0/2959917230 conn(0x7fea7406c760 msgr2=0x7fea7406cb60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:31:13.010 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.004+0000 7fea78dd0700 1 -- 172.21.15.110:0/2959917230 shutdown_connections 2023-10-06T03:31:13.010 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.004+0000 7fea78dd0700 1 -- 172.21.15.110:0/2959917230 wait complete. 2023-10-06T03:31:13.011 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.004+0000 7fea78dd0700 1 Processor -- start 2023-10-06T03:31:13.011 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.008+0000 7fea78dd0700 1 -- start start 2023-10-06T03:31:13.013 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.008+0000 7fea78dd0700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fea740accc0 0x7fea740c06b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:31:13.013 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.008+0000 7fea78dd0700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fea740ad890 0x7fea74162000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:31:13.013 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.008+0000 7fea78dd0700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fea7415d100 0x7fea74162540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:31:13.014 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.008+0000 7fea78dd0700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea7415ed50 con 0x7fea740ad890 2023-10-06T03:31:13.014 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.008+0000 7fea78dd0700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea7415e140 con 0x7fea740accc0 2023-10-06T03:31:13.014 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.008+0000 7fea78dd0700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea7415e420 con 0x7fea7415d100 2023-10-06T03:31:13.014 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.008+0000 7fea72ffd700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fea740ad890 0x7fea74162000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:31:13.014 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.008+0000 7fea737fe700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fea740accc0 0x7fea740c06b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:31:13.014 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.008+0000 7fea72ffd700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fea740ad890 0x7fea74162000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:54768/0 (socket says 172.21.15.110:54768) 2023-10-06T03:31:13.014 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.008+0000 7fea72ffd700 1 -- 172.21.15.110:0/2052013007 learned_addr learned my addr 172.21.15.110:0/2052013007 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:31:13.014 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.008+0000 7fea73fff700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fea7415d100 0x7fea74162540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:31:13.015 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.008+0000 7fea72ffd700 1 -- 172.21.15.110:0/2052013007 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fea7415d100 msgr2=0x7fea74162540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:31:13.016 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.008+0000 7fea72ffd700 1 --2- 172.21.15.110:0/2052013007 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fea7415d100 0x7fea74162540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:13.016 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.008+0000 7fea72ffd700 1 -- 172.21.15.110:0/2052013007 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fea740accc0 msgr2=0x7fea740c06b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:31:13.016 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.008+0000 7fea72ffd700 1 --2- 172.21.15.110:0/2052013007 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fea740accc0 0x7fea740c06b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:13.016 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.008+0000 7fea72ffd700 1 -- 172.21.15.110:0/2052013007 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea6c009a20 con 0x7fea740ad890 2023-10-06T03:31:13.016 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.008+0000 7fea72ffd700 1 --2- 172.21.15.110:0/2052013007 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fea740ad890 0x7fea74162000 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7fea6800c5b0 tx=0x7fea68007f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:31:13.016 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.008+0000 7fea5b7fe700 1 -- 172.21.15.110:0/2052013007 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea68011590 con 0x7fea740ad890 2023-10-06T03:31:13.016 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.008+0000 7fea5b7fe700 1 -- 172.21.15.110:0/2052013007 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea68011710 con 0x7fea740ad890 2023-10-06T03:31:13.016 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.008+0000 7fea78dd0700 1 -- 172.21.15.110:0/2052013007 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea7415e700 con 0x7fea740ad890 2023-10-06T03:31:13.016 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.008+0000 7fea5b7fe700 1 -- 172.21.15.110:0/2052013007 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea68011ad0 con 0x7fea740ad890 2023-10-06T03:31:13.016 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.008+0000 7fea78dd0700 1 -- 172.21.15.110:0/2052013007 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea741574b0 con 0x7fea740ad890 2023-10-06T03:31:13.017 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.012+0000 7fea78dd0700 1 -- 172.21.15.110:0/2052013007 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea7415ef00 con 0x7fea740ad890 2023-10-06T03:31:13.018 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.012+0000 7fea5b7fe700 1 -- 172.21.15.110:0/2052013007 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 13) v1 ==== 80061+0+0 (secure 0 0 0) 0x7fea6801b3b0 con 0x7fea740ad890 2023-10-06T03:31:13.018 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.012+0000 7fea5b7fe700 1 --2- 172.21.15.110:0/2052013007 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fea60060670 0x7fea60062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:31:13.019 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.012+0000 7fea5b7fe700 1 -- 172.21.15.110:0/2052013007 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 6171+0+0 (secure 0 0 0) 0x7fea6807bf70 con 0x7fea740ad890 2023-10-06T03:31:13.019 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.012+0000 7fea737fe700 1 --2- 172.21.15.110:0/2052013007 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fea60060670 0x7fea60062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:31:13.019 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.016+0000 7fea737fe700 1 --2- 172.21.15.110:0/2052013007 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fea60060670 0x7fea60062b20 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fea6c009de0 tx=0x7fea6c019000 comp rx=0 tx=0).ready entity=mgr.5505 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:31:13.024 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.020+0000 7fea5b7fe700 1 -- 172.21.15.110:0/2052013007 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fea7415ef00 con 0x7fea740ad890 2023-10-06T03:31:13.186 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.180+0000 7fea78dd0700 1 -- 172.21.15.110:0/2052013007 --> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "5", "target": ["mon-mgr", ""]}) v1 -- 0x7fea74071020 con 0x7fea60060670 2023-10-06T03:31:13.187 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.180+0000 7fea5b7fe700 1 -- 172.21.15.110:0/2052013007 <== mgr.5505 v2:172.21.15.110:6832/33268 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 5 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7fea74071020 con 0x7fea60060670 2023-10-06T03:31:13.190 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.184+0000 7fea78dd0700 1 -- 172.21.15.110:0/2052013007 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fea60060670 msgr2=0x7fea60062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:31:13.191 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.184+0000 7fea78dd0700 1 --2- 172.21.15.110:0/2052013007 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fea60060670 0x7fea60062b20 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fea6c009de0 tx=0x7fea6c019000 comp rx=0 tx=0).stop 2023-10-06T03:31:13.191 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.184+0000 7fea78dd0700 1 -- 172.21.15.110:0/2052013007 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fea740ad890 msgr2=0x7fea74162000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:31:13.191 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.184+0000 7fea78dd0700 1 --2- 172.21.15.110:0/2052013007 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fea740ad890 0x7fea74162000 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7fea6800c5b0 tx=0x7fea68007f60 comp rx=0 tx=0).stop 2023-10-06T03:31:13.191 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.184+0000 7fea78dd0700 1 -- 172.21.15.110:0/2052013007 shutdown_connections 2023-10-06T03:31:13.191 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.184+0000 7fea78dd0700 1 --2- 172.21.15.110:0/2052013007 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fea60060670 0x7fea60062b20 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:13.191 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.184+0000 7fea78dd0700 1 --2- 172.21.15.110:0/2052013007 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fea7415d100 0x7fea74162540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:13.191 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.184+0000 7fea78dd0700 1 --2- 172.21.15.110:0/2052013007 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fea740ad890 0x7fea74162000 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:13.191 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.184+0000 7fea78dd0700 1 --2- 172.21.15.110:0/2052013007 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fea740accc0 0x7fea740c06b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:13.191 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.184+0000 7fea78dd0700 1 -- 172.21.15.110:0/2052013007 >> 172.21.15.110:0/2052013007 conn(0x7fea7406c760 msgr2=0x7fea740afa20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:31:13.191 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.184+0000 7fea78dd0700 1 -- 172.21.15.110:0/2052013007 shutdown_connections 2023-10-06T03:31:13.192 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.184+0000 7fea78dd0700 1 -- 172.21.15.110:0/2052013007 wait complete. 2023-10-06T03:31:13.192 INFO:teuthology.orchestra.run.smithi110.stderr:instructed osd(s) 5 to deep-scrub 2023-10-06T03:31:13.206 INFO:tasks.ceph:Scrubbing osd.6 2023-10-06T03:31:13.206 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 config set osd_debug_deep_scrub_sleep 0 2023-10-06T03:31:13.319 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.312+0000 7f28d52e6700 1 Processor -- start 2023-10-06T03:31:13.319 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.312+0000 7f28d52e6700 1 -- start start 2023-10-06T03:31:13.320 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.316+0000 7f28d52e6700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f28d0156d60 0x7f28d006ae50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:31:13.320 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.316+0000 7f28d52e6700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f28d006b390 0x7f28d00b1470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:31:13.320 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.316+0000 7f28d52e6700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28d00caf20 con 0x7f28d0156990 2023-10-06T03:31:13.320 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.316+0000 7f28d52e6700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f28d006b7d0 con 0x7f28d006b390 2023-10-06T03:31:13.320 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.316+0000 7f28d52e6700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28d00c8720 con 0x7f28d0156d60 2023-10-06T03:31:13.320 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.316+0000 7f28ceffd700 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f28d0156990 0x7f28d006c760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:48160/0 (socket says 172.21.15.110:48160) 2023-10-06T03:31:13.321 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.316+0000 7f28ceffd700 1 -- 172.21.15.110:0/1931319501 learned_addr learned my addr 172.21.15.110:0/1931319501 (peer_addr_for_me v1:172.21.15.110:0/0) 2023-10-06T03:31:13.321 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.316+0000 7f28cf7fe700 1 --2- 172.21.15.110:0/1931319501 >> v2:172.21.15.192:3300/0 conn(0x7f28d006b390 0x7f28d00b1470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:31:13.321 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.316+0000 7f28c67fc700 1 --2- 172.21.15.110:0/1931319501 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f28d0156d60 0x7f28d006ae50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:31:13.321 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.316+0000 7f28ce7fc700 1 -- 172.21.15.110:0/1931319501 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3083610780 0 0) 0x7f28d00caf20 con 0x7f28d0156990 2023-10-06T03:31:13.321 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.316+0000 7f28ce7fc700 1 -- 172.21.15.110:0/1931319501 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f28bc003570 con 0x7f28d0156990 2023-10-06T03:31:13.322 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.316+0000 7f28cf7fe700 1 -- 172.21.15.110:0/1931319501 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f28d0156d60 msgr2=0x7f28d006ae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:31:13.322 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.316+0000 7f28cf7fe700 1 --2- 172.21.15.110:0/1931319501 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f28d0156d60 0x7f28d006ae50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:13.322 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.316+0000 7f28cf7fe700 1 -- 172.21.15.110:0/1931319501 >> v1:172.21.15.110:6789/0 conn(0x7f28d0156990 legacy=0x7f28d006c760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:31:13.322 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.316+0000 7f28cf7fe700 1 -- 172.21.15.110:0/1931319501 --> v2:172.21.15.192:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28d006b9c0 con 0x7f28d006b390 2023-10-06T03:31:13.322 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.316+0000 7f28cf7fe700 1 --2- 172.21.15.110:0/1931319501 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f28d006b390 0x7f28d00b1470 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f28c8009e60 tx=0x7f28c8009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6e8b57cdc2374389 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:31:13.322 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.316+0000 7f28ce7fc700 1 -- 172.21.15.110:0/1931319501 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28c800e070 con 0x7f28d006b390 2023-10-06T03:31:13.322 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.316+0000 7f28ce7fc700 1 -- 172.21.15.110:0/1931319501 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28c800bab0 con 0x7f28d006b390 2023-10-06T03:31:13.322 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.316+0000 7f28ce7fc700 1 -- 172.21.15.110:0/1931319501 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28c800be30 con 0x7f28d006b390 2023-10-06T03:31:13.323 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.316+0000 7f28d52e6700 1 -- 172.21.15.110:0/1931319501 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f28d006b390 msgr2=0x7f28d00b1470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:31:13.323 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.316+0000 7f28d52e6700 1 --2- 172.21.15.110:0/1931319501 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f28d006b390 0x7f28d00b1470 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f28c8009e60 tx=0x7f28c8009780 comp rx=0 tx=0).stop 2023-10-06T03:31:13.323 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.316+0000 7f28d52e6700 1 -- 172.21.15.110:0/1931319501 shutdown_connections 2023-10-06T03:31:13.323 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.316+0000 7f28d52e6700 1 --2- 172.21.15.110:0/1931319501 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f28d006b390 0x7f28d00b1470 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:13.323 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.316+0000 7f28d52e6700 1 --2- 172.21.15.110:0/1931319501 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f28d0156d60 0x7f28d006ae50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:13.323 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.316+0000 7f28d52e6700 1 -- 172.21.15.110:0/1931319501 >> 172.21.15.110:0/1931319501 conn(0x7f28d00cc3a0 msgr2=0x7f28d00cc7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:31:13.324 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.316+0000 7f28d52e6700 1 -- 172.21.15.110:0/1931319501 shutdown_connections 2023-10-06T03:31:13.324 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.316+0000 7f28d52e6700 1 -- 172.21.15.110:0/1931319501 wait complete. 2023-10-06T03:31:13.324 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.320+0000 7f28d52e6700 1 Processor -- start 2023-10-06T03:31:13.324 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.320+0000 7f28d52e6700 1 -- start start 2023-10-06T03:31:13.327 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.320+0000 7f28d52e6700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f28d006b390 0x7f28d00c5760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:31:13.327 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.320+0000 7f28d52e6700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f28d0156990 0x7f28d00c1b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:31:13.327 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.320+0000 7f28d52e6700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f28d0156d60 0x7f28d0154b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:31:13.327 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.320+0000 7f28d52e6700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28d01546c0 con 0x7f28d0156990 2023-10-06T03:31:13.327 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.320+0000 7f28d52e6700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28d00c2530 con 0x7f28d0156d60 2023-10-06T03:31:13.327 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.320+0000 7f28d52e6700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28d0153d90 con 0x7f28d006b390 2023-10-06T03:31:13.327 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.320+0000 7f28c67fc700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f28d0156990 0x7f28d00c1b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:31:13.327 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.320+0000 7f28c67fc700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f28d0156990 0x7f28d00c1b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:54776/0 (socket says 172.21.15.110:54776) 2023-10-06T03:31:13.327 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.320+0000 7f28c67fc700 1 -- 172.21.15.110:0/3179512681 learned_addr learned my addr 172.21.15.110:0/3179512681 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:31:13.327 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.320+0000 7f28ceffd700 1 --2- 172.21.15.110:0/3179512681 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f28d006b390 0x7f28d00c5760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:31:13.327 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.320+0000 7f28cf7fe700 1 --2- 172.21.15.110:0/3179512681 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f28d0156d60 0x7f28d0154b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:31:13.328 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.320+0000 7f28c67fc700 1 -- 172.21.15.110:0/3179512681 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f28d006b390 msgr2=0x7f28d00c5760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:31:13.328 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.320+0000 7f28c67fc700 1 --2- 172.21.15.110:0/3179512681 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f28d006b390 0x7f28d00c5760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:13.328 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.320+0000 7f28c67fc700 1 -- 172.21.15.110:0/3179512681 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f28d0156d60 msgr2=0x7f28d0154b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:31:13.328 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.320+0000 7f28c67fc700 1 --2- 172.21.15.110:0/3179512681 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f28d0156d60 0x7f28d0154b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:13.328 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.320+0000 7f28c67fc700 1 -- 172.21.15.110:0/3179512681 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28c8009a20 con 0x7f28d0156990 2023-10-06T03:31:13.328 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.320+0000 7f28c67fc700 1 --2- 172.21.15.110:0/3179512681 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f28d0156990 0x7f28d00c1b50 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f28b800e930 tx=0x7f28b80091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:31:13.329 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.320+0000 7f28c77fe700 1 -- 172.21.15.110:0/3179512681 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28b8013680 con 0x7f28d0156990 2023-10-06T03:31:13.329 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.320+0000 7f28c77fe700 1 -- 172.21.15.110:0/3179512681 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28b80137e0 con 0x7f28d0156990 2023-10-06T03:31:13.329 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.320+0000 7f28c77fe700 1 -- 172.21.15.110:0/3179512681 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28b8013b80 con 0x7f28d0156990 2023-10-06T03:31:13.329 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.320+0000 7f28d52e6700 1 -- 172.21.15.110:0/3179512681 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28d0154070 con 0x7f28d0156990 2023-10-06T03:31:13.329 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.324+0000 7f28d52e6700 1 -- 172.21.15.110:0/3179512681 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28d0150d90 con 0x7f28d0156990 2023-10-06T03:31:13.329 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.324+0000 7f28d52e6700 1 -- 172.21.15.110:0/3179512681 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f28d0064770 con 0x7f28d0156990 2023-10-06T03:31:13.330 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.324+0000 7f28c77fe700 1 -- 172.21.15.110:0/3179512681 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 13) v1 ==== 80061+0+0 (secure 0 0 0) 0x7f28b800c3e0 con 0x7f28d0156990 2023-10-06T03:31:13.330 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.324+0000 7f28c77fe700 1 --2- 172.21.15.110:0/3179512681 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f28b0060670 0x7f28b0062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:31:13.331 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.324+0000 7f28c77fe700 1 -- 172.21.15.110:0/3179512681 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 6171+0+0 (secure 0 0 0) 0x7f28b807cc20 con 0x7f28d0156990 2023-10-06T03:31:13.331 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.324+0000 7f28ceffd700 1 --2- 172.21.15.110:0/3179512681 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f28b0060670 0x7f28b0062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:31:13.331 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.324+0000 7f28c77fe700 1 --2- 172.21.15.110:0/3179512681 >> [v2:172.21.15.192:6824/33686,v1:172.21.15.192:6825/33686] conn(0x7f28b0067480 0x7f28b00698b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:31:13.331 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.324+0000 7f28c77fe700 1 -- 172.21.15.110:0/3179512681 --> [v2:172.21.15.192:6824/33686,v1:172.21.15.192:6825/33686] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f28b0069f80 con 0x7f28b0067480 2023-10-06T03:31:13.331 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.324+0000 7f28c77fe700 1 -- 172.21.15.110:0/3179512681 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_get_version_reply(handle=1 version=47) v2 ==== 24+0+0 (secure 0 0 0) 0x7f28b807f260 con 0x7f28d0156990 2023-10-06T03:31:13.332 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.324+0000 7f28cf7fe700 1 --2- 172.21.15.110:0/3179512681 >> [v2:172.21.15.192:6824/33686,v1:172.21.15.192:6825/33686] conn(0x7f28b0067480 0x7f28b00698b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:31:13.332 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.328+0000 7f28ceffd700 1 --2- 172.21.15.110:0/3179512681 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f28b0060670 0x7f28b0062b20 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f28c0007fd0 tx=0x7f28c0009040 comp rx=0 tx=0).ready entity=mgr.5505 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:31:13.332 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.328+0000 7f28cf7fe700 1 --2- 172.21.15.110:0/3179512681 >> [v2:172.21.15.192:6824/33686,v1:172.21.15.192:6825/33686] conn(0x7f28b0067480 0x7f28b00698b0 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:31:13.333 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.328+0000 7f28c77fe700 1 -- 172.21.15.110:0/3179512681 <== osd.6 v2:172.21.15.192:6824/33686 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23999 (crc 0 0 0) 0x7f28b0069f80 con 0x7f28b0067480 2023-10-06T03:31:13.351 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.344+0000 7f28d52e6700 1 -- 172.21.15.110:0/3179512681 --> [v2:172.21.15.192:6824/33686,v1:172.21.15.192:6825/33686] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f28d006bca0 con 0x7f28b0067480 2023-10-06T03:31:13.351 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.348+0000 7f28c77fe700 1 -- 172.21.15.110:0/3179512681 <== osd.6 v2:172.21.15.192:6824/33686 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7f28d006bca0 con 0x7f28b0067480 2023-10-06T03:31:13.352 INFO:teuthology.orchestra.run.smithi110.stdout:{ 2023-10-06T03:31:13.352 INFO:teuthology.orchestra.run.smithi110.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-10-06T03:31:13.352 INFO:teuthology.orchestra.run.smithi110.stdout:} 2023-10-06T03:31:13.353 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.348+0000 7f28d52e6700 1 -- 172.21.15.110:0/3179512681 >> [v2:172.21.15.192:6824/33686,v1:172.21.15.192:6825/33686] conn(0x7f28b0067480 msgr2=0x7f28b00698b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:31:13.353 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.348+0000 7f28d52e6700 1 --2- 172.21.15.110:0/3179512681 >> [v2:172.21.15.192:6824/33686,v1:172.21.15.192:6825/33686] conn(0x7f28b0067480 0x7f28b00698b0 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:13.353 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.348+0000 7f28d52e6700 1 -- 172.21.15.110:0/3179512681 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f28b0060670 msgr2=0x7f28b0062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:31:13.353 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.348+0000 7f28d52e6700 1 --2- 172.21.15.110:0/3179512681 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f28b0060670 0x7f28b0062b20 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f28c0007fd0 tx=0x7f28c0009040 comp rx=0 tx=0).stop 2023-10-06T03:31:13.353 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.348+0000 7f28d52e6700 1 -- 172.21.15.110:0/3179512681 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f28d0156990 msgr2=0x7f28d00c1b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:31:13.353 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.348+0000 7f28d52e6700 1 --2- 172.21.15.110:0/3179512681 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f28d0156990 0x7f28d00c1b50 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f28b800e930 tx=0x7f28b80091a0 comp rx=0 tx=0).stop 2023-10-06T03:31:13.353 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.348+0000 7f28cf7fe700 1 -- 172.21.15.110:0/3179512681 reap_dead start 2023-10-06T03:31:13.353 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.348+0000 7f28d52e6700 1 -- 172.21.15.110:0/3179512681 shutdown_connections 2023-10-06T03:31:13.353 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.348+0000 7f28d52e6700 1 -- 172.21.15.110:0/3179512681 >> 172.21.15.110:0/3179512681 conn(0x7f28d00cc3a0 msgr2=0x7f28d00cb3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:31:13.354 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.348+0000 7f28d52e6700 1 -- 172.21.15.110:0/3179512681 shutdown_connections 2023-10-06T03:31:13.354 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.348+0000 7f28d52e6700 1 -- 172.21.15.110:0/3179512681 wait complete. 2023-10-06T03:31:13.364 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 6 2023-10-06T03:31:13.475 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.468+0000 7f7e01dba700 1 Processor -- start 2023-10-06T03:31:13.475 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.468+0000 7f7e01dba700 1 -- start start 2023-10-06T03:31:13.476 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.472+0000 7f7e01dba700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7dfc155c90 0x7f7dfc0c7300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:31:13.476 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.472+0000 7f7e01dba700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7dfc06a830 con 0x7f7dfc159a30 2023-10-06T03:31:13.477 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.472+0000 7f7e01dba700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7dfc06a4e0 con 0x7f7dfc156060 2023-10-06T03:31:13.477 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.472+0000 7f7e01dba700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7dfc0c7c50 con 0x7f7dfc155c90 2023-10-06T03:31:13.477 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.472+0000 7f7dfbfff700 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f7dfc159a30 0x7f7dfc0c60e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:48174/0 (socket says 172.21.15.110:48174) 2023-10-06T03:31:13.477 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.472+0000 7f7dfbfff700 1 -- 172.21.15.110:0/3506010692 learned_addr learned my addr 172.21.15.110:0/3506010692 (peer_addr_for_me v1:172.21.15.110:0/0) 2023-10-06T03:31:13.477 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.472+0000 7f7dfb7fe700 1 --2- 172.21.15.110:0/3506010692 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7dfc155c90 0x7f7dfc0c7300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:31:13.478 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.472+0000 7f7dfa7fc700 1 -- 172.21.15.110:0/3506010692 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 256218147 0 0) 0x7f7dfc06a830 con 0x7f7dfc159a30 2023-10-06T03:31:13.478 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.472+0000 7f7dfa7fc700 1 -- 172.21.15.110:0/3506010692 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7de4003570 con 0x7f7dfc159a30 2023-10-06T03:31:13.479 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.472+0000 7f7dfa7fc700 1 -- 172.21.15.110:0/3506010692 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 753193706 0 0) 0x7f7dfc06a4e0 con 0x7f7dfc156060 2023-10-06T03:31:13.479 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.472+0000 7f7dfa7fc700 1 -- 172.21.15.110:0/3506010692 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7dfc06a830 con 0x7f7dfc156060 2023-10-06T03:31:13.479 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.472+0000 7f7dfb7fe700 1 -- 172.21.15.110:0/3506010692 >> v1:172.21.15.192:6789/0 conn(0x7f7dfc156060 legacy=0x7f7dfc06acc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:31:13.479 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.472+0000 7f7dfb7fe700 1 -- 172.21.15.110:0/3506010692 >> v1:172.21.15.110:6789/0 conn(0x7f7dfc159a30 legacy=0x7f7dfc0c60e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:31:13.479 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.472+0000 7f7dfb7fe700 1 -- 172.21.15.110:0/3506010692 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7dfc0c7f30 con 0x7f7dfc155c90 2023-10-06T03:31:13.479 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.472+0000 7f7dfb7fe700 1 --2- 172.21.15.110:0/3506010692 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7dfc155c90 0x7f7dfc0c7300 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f7df4009e90 tx=0x7f7df4009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3963d8652e993c02 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:31:13.479 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.472+0000 7f7dfa7fc700 1 -- 172.21.15.110:0/3506010692 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7df400e070 con 0x7f7dfc155c90 2023-10-06T03:31:13.479 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.472+0000 7f7dfa7fc700 1 -- 172.21.15.110:0/3506010692 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7df400bb80 con 0x7f7dfc155c90 2023-10-06T03:31:13.479 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.472+0000 7f7dfa7fc700 1 -- 172.21.15.110:0/3506010692 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7df4012430 con 0x7f7dfc155c90 2023-10-06T03:31:13.480 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.476+0000 7f7e01dba700 1 -- 172.21.15.110:0/3506010692 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7dfc155c90 msgr2=0x7f7dfc0c7300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:31:13.480 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.476+0000 7f7e01dba700 1 --2- 172.21.15.110:0/3506010692 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7dfc155c90 0x7f7dfc0c7300 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f7df4009e90 tx=0x7f7df4009780 comp rx=0 tx=0).stop 2023-10-06T03:31:13.480 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.476+0000 7f7e01dba700 1 -- 172.21.15.110:0/3506010692 shutdown_connections 2023-10-06T03:31:13.480 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.476+0000 7f7e01dba700 1 --2- 172.21.15.110:0/3506010692 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7dfc155c90 0x7f7dfc0c7300 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:13.480 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.476+0000 7f7e01dba700 1 -- 172.21.15.110:0/3506010692 >> 172.21.15.110:0/3506010692 conn(0x7f7dfc067040 msgr2=0x7f7dfc067410 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:31:13.480 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.476+0000 7f7e01dba700 1 -- 172.21.15.110:0/3506010692 shutdown_connections 2023-10-06T03:31:13.481 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.476+0000 7f7e01dba700 1 -- 172.21.15.110:0/3506010692 wait complete. 2023-10-06T03:31:13.481 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.476+0000 7f7e01dba700 1 Processor -- start 2023-10-06T03:31:13.482 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.476+0000 7f7e01dba700 1 -- start start 2023-10-06T03:31:13.486 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.480+0000 7f7e01dba700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7dfc155c90 0x7f7dfc151f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:31:13.486 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.480+0000 7f7e01dba700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7dfc159a30 0x7f7dfc0c5590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:31:13.486 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.480+0000 7f7e01dba700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7dfc0c5ad0 0x7f7dfc0ae9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:31:13.487 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.480+0000 7f7e01dba700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7dfc0c9e00 con 0x7f7dfc159a30 2023-10-06T03:31:13.487 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.480+0000 7f7e01dba700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7dfc0ca0e0 con 0x7f7dfc155c90 2023-10-06T03:31:13.487 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.480+0000 7f7e01dba700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7dfc0ca3c0 con 0x7f7dfc0c5ad0 2023-10-06T03:31:13.487 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.480+0000 7f7dfb7fe700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7dfc155c90 0x7f7dfc151f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:31:13.487 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.480+0000 7f7dfaffd700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7dfc159a30 0x7f7dfc0c5590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:31:13.487 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.480+0000 7f7dfb7fe700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7dfc155c90 0x7f7dfc151f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.110:47496/0 (socket says 172.21.15.110:47496) 2023-10-06T03:31:13.487 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.480+0000 7f7dfb7fe700 1 -- 172.21.15.110:0/2654789860 learned_addr learned my addr 172.21.15.110:0/2654789860 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:31:13.487 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.480+0000 7f7dfbfff700 1 --2- 172.21.15.110:0/2654789860 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7dfc0c5ad0 0x7f7dfc0ae9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:31:13.487 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.480+0000 7f7dfaffd700 1 -- 172.21.15.110:0/2654789860 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7dfc0c5ad0 msgr2=0x7f7dfc0ae9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:31:13.487 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.480+0000 7f7dfaffd700 1 --2- 172.21.15.110:0/2654789860 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7dfc0c5ad0 0x7f7dfc0ae9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:13.487 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.480+0000 7f7dfaffd700 1 -- 172.21.15.110:0/2654789860 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7dfc155c90 msgr2=0x7f7dfc151f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:31:13.488 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.480+0000 7f7dfaffd700 1 --2- 172.21.15.110:0/2654789860 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7dfc155c90 0x7f7dfc151f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:13.488 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.480+0000 7f7dfaffd700 1 -- 172.21.15.110:0/2654789860 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7df4009a20 con 0x7f7dfc159a30 2023-10-06T03:31:13.488 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.480+0000 7f7dfaffd700 1 --2- 172.21.15.110:0/2654789860 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7dfc159a30 0x7f7dfc0c5590 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f7df000c560 tx=0x7f7df0007d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:31:13.488 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.480+0000 7f7de37fe700 1 -- 172.21.15.110:0/2654789860 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7df0011590 con 0x7f7dfc159a30 2023-10-06T03:31:13.488 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.480+0000 7f7de37fe700 1 -- 172.21.15.110:0/2654789860 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7df00116f0 con 0x7f7dfc159a30 2023-10-06T03:31:13.488 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.480+0000 7f7e01dba700 1 -- 172.21.15.110:0/2654789860 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7dfc0ca6a0 con 0x7f7dfc159a30 2023-10-06T03:31:13.488 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.480+0000 7f7de37fe700 1 -- 172.21.15.110:0/2654789860 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7df0011ab0 con 0x7f7dfc159a30 2023-10-06T03:31:13.488 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.480+0000 7f7e01dba700 1 -- 172.21.15.110:0/2654789860 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7dfc0c2d00 con 0x7f7dfc159a30 2023-10-06T03:31:13.489 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.480+0000 7f7e01dba700 1 -- 172.21.15.110:0/2654789860 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7dfc064770 con 0x7f7dfc159a30 2023-10-06T03:31:13.489 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.484+0000 7f7de37fe700 1 -- 172.21.15.110:0/2654789860 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 13) v1 ==== 80061+0+0 (secure 0 0 0) 0x7f7df001c460 con 0x7f7dfc159a30 2023-10-06T03:31:13.489 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.484+0000 7f7de37fe700 1 --2- 172.21.15.110:0/2654789860 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f7de80606a0 0x7f7de8062b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:31:13.489 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.484+0000 7f7dfb7fe700 1 --2- 172.21.15.110:0/2654789860 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f7de80606a0 0x7f7de8062b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:31:13.490 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.484+0000 7f7de37fe700 1 -- 172.21.15.110:0/2654789860 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 6171+0+0 (secure 0 0 0) 0x7f7df007c710 con 0x7f7dfc159a30 2023-10-06T03:31:13.490 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.484+0000 7f7dfb7fe700 1 --2- 172.21.15.110:0/2654789860 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f7de80606a0 0x7f7de8062b50 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f7df4012560 tx=0x7f7df4019000 comp rx=0 tx=0).ready entity=mgr.5505 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:31:13.495 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.492+0000 7f7de37fe700 1 -- 172.21.15.110:0/2654789860 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f7df004ca20 con 0x7f7dfc159a30 2023-10-06T03:31:13.655 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.652+0000 7f7e01dba700 1 -- 172.21.15.110:0/2654789860 --> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "6", "target": ["mon-mgr", ""]}) v1 -- 0x7f7dfc0c8210 con 0x7f7de80606a0 2023-10-06T03:31:13.656 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.652+0000 7f7de37fe700 1 -- 172.21.15.110:0/2654789860 <== mgr.5505 v2:172.21.15.110:6832/33268 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 6 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f7dfc0c8210 con 0x7f7de80606a0 2023-10-06T03:31:13.659 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.652+0000 7f7e01dba700 1 -- 172.21.15.110:0/2654789860 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f7de80606a0 msgr2=0x7f7de8062b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:31:13.659 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.652+0000 7f7e01dba700 1 --2- 172.21.15.110:0/2654789860 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f7de80606a0 0x7f7de8062b50 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f7df4012560 tx=0x7f7df4019000 comp rx=0 tx=0).stop 2023-10-06T03:31:13.659 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.652+0000 7f7e01dba700 1 -- 172.21.15.110:0/2654789860 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7dfc159a30 msgr2=0x7f7dfc0c5590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:31:13.659 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.652+0000 7f7e01dba700 1 --2- 172.21.15.110:0/2654789860 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7dfc159a30 0x7f7dfc0c5590 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f7df000c560 tx=0x7f7df0007d50 comp rx=0 tx=0).stop 2023-10-06T03:31:13.659 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.652+0000 7f7e01dba700 1 -- 172.21.15.110:0/2654789860 shutdown_connections 2023-10-06T03:31:13.659 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.652+0000 7f7e01dba700 1 --2- 172.21.15.110:0/2654789860 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f7de80606a0 0x7f7de8062b50 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:13.659 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.652+0000 7f7e01dba700 1 --2- 172.21.15.110:0/2654789860 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f7dfc0c5ad0 0x7f7dfc0ae9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:13.659 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.652+0000 7f7e01dba700 1 --2- 172.21.15.110:0/2654789860 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7dfc159a30 0x7f7dfc0c5590 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:13.660 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.652+0000 7f7e01dba700 1 --2- 172.21.15.110:0/2654789860 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7dfc155c90 0x7f7dfc151f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:13.660 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.652+0000 7f7e01dba700 1 -- 172.21.15.110:0/2654789860 >> 172.21.15.110:0/2654789860 conn(0x7f7dfc067040 msgr2=0x7f7dfc153320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:31:13.660 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.652+0000 7f7e01dba700 1 -- 172.21.15.110:0/2654789860 shutdown_connections 2023-10-06T03:31:13.660 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.652+0000 7f7e01dba700 1 -- 172.21.15.110:0/2654789860 wait complete. 2023-10-06T03:31:13.660 INFO:teuthology.orchestra.run.smithi110.stderr:instructed osd(s) 6 to deep-scrub 2023-10-06T03:31:13.670 INFO:tasks.ceph:Scrubbing osd.7 2023-10-06T03:31:13.670 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 config set osd_debug_deep_scrub_sleep 0 2023-10-06T03:31:13.780 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.776+0000 7ff7fd84c700 1 Processor -- start 2023-10-06T03:31:13.780 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.776+0000 7ff7fd84c700 1 -- start start 2023-10-06T03:31:13.782 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.776+0000 7ff7fd84c700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7ff7f8152be0 0x7ff7f80c6cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:31:13.782 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.776+0000 7ff7fd84c700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff7f8152fb0 0x7ff7f80ca570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:31:13.782 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.776+0000 7ff7fd84c700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7f806ae20 con 0x7ff7f8159af0 2023-10-06T03:31:13.782 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.776+0000 7ff7fd84c700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff7f80c8f40 con 0x7ff7f8152be0 2023-10-06T03:31:13.782 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.776+0000 7ff7fd84c700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7f80c7200 con 0x7ff7f8152fb0 2023-10-06T03:31:13.782 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.776+0000 7ff7fc84a700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7ff7f8152be0 0x7ff7f80c6cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:31:13.782 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.776+0000 7ff7fc84a700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7ff7f8152be0 0x7ff7f80c6cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:54794/0 (socket says 172.21.15.110:54794) 2023-10-06T03:31:13.783 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.776+0000 7ff7fc84a700 1 -- 172.21.15.110:0/3788975129 learned_addr learned my addr 172.21.15.110:0/3788975129 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:31:13.783 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.776+0000 7ff7fd04b700 1 --2- 172.21.15.110:0/3788975129 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff7f8152fb0 0x7ff7f80ca570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:31:13.783 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.776+0000 7ff7fc84a700 1 -- 172.21.15.110:0/3788975129 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff7f8152fb0 msgr2=0x7ff7f80ca570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:31:13.783 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.776+0000 7ff7fc84a700 1 --2- 172.21.15.110:0/3788975129 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff7f8152fb0 0x7ff7f80ca570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:13.783 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.776+0000 7ff7fc84a700 1 -- 172.21.15.110:0/3788975129 >> v1:172.21.15.192:6789/0 conn(0x7ff7f8159af0 legacy=0x7ff7f8155ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:31:13.783 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.776+0000 7ff7fc84a700 1 -- 172.21.15.110:0/3788975129 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7f806a4f0 con 0x7ff7f8152be0 2023-10-06T03:31:13.783 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.776+0000 7ff7fc84a700 1 --2- 172.21.15.110:0/3788975129 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff7f8152be0 0x7ff7f80c6cc0 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7ff7ec0099b0 tx=0x7ff7ec009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e99667737c7da9e server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:31:13.783 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.776+0000 7ff7f77fe700 1 -- 172.21.15.110:0/3788975129 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7ec00b860 con 0x7ff7f8152be0 2023-10-06T03:31:13.783 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.776+0000 7ff7f77fe700 1 -- 172.21.15.110:0/3788975129 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff7ec00b9c0 con 0x7ff7f8152be0 2023-10-06T03:31:13.783 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.776+0000 7ff7f77fe700 1 -- 172.21.15.110:0/3788975129 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7ec00bd60 con 0x7ff7f8152be0 2023-10-06T03:31:13.784 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.776+0000 7ff7fd84c700 1 -- 172.21.15.110:0/3788975129 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff7f8152be0 msgr2=0x7ff7f80c6cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:31:13.784 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.776+0000 7ff7fd84c700 1 --2- 172.21.15.110:0/3788975129 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff7f8152be0 0x7ff7f80c6cc0 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7ff7ec0099b0 tx=0x7ff7ec009250 comp rx=0 tx=0).stop 2023-10-06T03:31:13.784 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.776+0000 7ff7fd84c700 1 -- 172.21.15.110:0/3788975129 shutdown_connections 2023-10-06T03:31:13.784 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.776+0000 7ff7fd84c700 1 --2- 172.21.15.110:0/3788975129 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff7f8152fb0 0x7ff7f80ca570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:13.785 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.776+0000 7ff7fd84c700 1 --2- 172.21.15.110:0/3788975129 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff7f8152be0 0x7ff7f80c6cc0 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:13.785 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.776+0000 7ff7fd84c700 1 -- 172.21.15.110:0/3788975129 >> 172.21.15.110:0/3788975129 conn(0x7ff7f806c760 msgr2=0x7ff7f806cb60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:31:13.785 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.776+0000 7ff7fd84c700 1 -- 172.21.15.110:0/3788975129 shutdown_connections 2023-10-06T03:31:13.785 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.776+0000 7ff7fd84c700 1 -- 172.21.15.110:0/3788975129 wait complete. 2023-10-06T03:31:13.785 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.780+0000 7ff7fd84c700 1 Processor -- start 2023-10-06T03:31:13.785 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.780+0000 7ff7fd84c700 1 -- start start 2023-10-06T03:31:13.787 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.780+0000 7ff7fd84c700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff7f8152be0 0x7ff7f8152400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:31:13.788 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.780+0000 7ff7fd84c700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff7f8152fb0 0x7ff7f80af1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:31:13.788 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.780+0000 7ff7fd84c700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff7f8159af0 0x7ff7f80abdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:31:13.788 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.780+0000 7ff7fd84c700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7f80a8cd0 con 0x7ff7f8152fb0 2023-10-06T03:31:13.788 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.780+0000 7ff7fd84c700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7f80b00c0 con 0x7ff7f8159af0 2023-10-06T03:31:13.788 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.780+0000 7ff7fd84c700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7f80b03a0 con 0x7ff7f8152be0 2023-10-06T03:31:13.788 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.780+0000 7ff7f7fff700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff7f8152fb0 0x7ff7f80af1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:31:13.788 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.780+0000 7ff7f7fff700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff7f8152fb0 0x7ff7f80af1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:54806/0 (socket says 172.21.15.110:54806) 2023-10-06T03:31:13.788 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.780+0000 7ff7f7fff700 1 -- 172.21.15.110:0/234588845 learned_addr learned my addr 172.21.15.110:0/234588845 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:31:13.788 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.780+0000 7ff7fd04b700 1 --2- 172.21.15.110:0/234588845 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff7f8159af0 0x7ff7f80abdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:31:13.788 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.780+0000 7ff7fc84a700 1 --2- 172.21.15.110:0/234588845 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff7f8152be0 0x7ff7f8152400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:31:13.788 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.780+0000 7ff7f7fff700 1 -- 172.21.15.110:0/234588845 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff7f8152be0 msgr2=0x7ff7f8152400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:31:13.789 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.780+0000 7ff7f7fff700 1 --2- 172.21.15.110:0/234588845 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff7f8152be0 0x7ff7f8152400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:13.789 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.780+0000 7ff7f7fff700 1 -- 172.21.15.110:0/234588845 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff7f8159af0 msgr2=0x7ff7f80abdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:31:13.789 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.780+0000 7ff7f7fff700 1 --2- 172.21.15.110:0/234588845 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff7f8159af0 0x7ff7f80abdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:13.789 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.780+0000 7ff7f7fff700 1 -- 172.21.15.110:0/234588845 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7ec0094f0 con 0x7ff7f8152fb0 2023-10-06T03:31:13.789 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.780+0000 7ff7fc84a700 1 --2- 172.21.15.110:0/234588845 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7ff7f8152be0 0x7ff7f8152400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-06T03:31:13.789 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.780+0000 7ff7f7fff700 1 --2- 172.21.15.110:0/234588845 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff7f8152fb0 0x7ff7f80af1b0 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7ff7e400c4e0 tx=0x7ff7e400c7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:31:13.789 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.784+0000 7ff7f4ff9700 1 -- 172.21.15.110:0/234588845 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7e4011590 con 0x7ff7f8152fb0 2023-10-06T03:31:13.789 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.784+0000 7ff7f4ff9700 1 -- 172.21.15.110:0/234588845 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff7e4011710 con 0x7ff7f8152fb0 2023-10-06T03:31:13.789 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.784+0000 7ff7fd84c700 1 -- 172.21.15.110:0/234588845 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff7f80b0680 con 0x7ff7f8152fb0 2023-10-06T03:31:13.789 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.784+0000 7ff7f4ff9700 1 -- 172.21.15.110:0/234588845 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7e4011ad0 con 0x7ff7f8152fb0 2023-10-06T03:31:13.789 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.784+0000 7ff7fd84c700 1 -- 172.21.15.110:0/234588845 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7f8152940 con 0x7ff7f8152fb0 2023-10-06T03:31:13.790 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.784+0000 7ff7fd84c700 1 -- 172.21.15.110:0/234588845 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7ff7f8064770 con 0x7ff7f8152fb0 2023-10-06T03:31:13.790 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.784+0000 7ff7f4ff9700 1 -- 172.21.15.110:0/234588845 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 13) v1 ==== 80061+0+0 (secure 0 0 0) 0x7ff7e401b3b0 con 0x7ff7f8152fb0 2023-10-06T03:31:13.791 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.784+0000 7ff7f4ff9700 1 --2- 172.21.15.110:0/234588845 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7ff7dc0606a0 0x7ff7dc062b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:31:13.791 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.784+0000 7ff7f4ff9700 1 -- 172.21.15.110:0/234588845 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 6171+0+0 (secure 0 0 0) 0x7ff7e407bf10 con 0x7ff7f8152fb0 2023-10-06T03:31:13.791 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.784+0000 7ff7fc84a700 1 --2- 172.21.15.110:0/234588845 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7ff7dc0606a0 0x7ff7dc062b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:31:13.791 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.784+0000 7ff7f4ff9700 1 --2- 172.21.15.110:0/234588845 >> [v2:172.21.15.192:6808/33685,v1:172.21.15.192:6809/33685] conn(0x7ff7dc067530 0x7ff7dc069960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:31:13.791 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.784+0000 7ff7f4ff9700 1 -- 172.21.15.110:0/234588845 --> [v2:172.21.15.192:6808/33685,v1:172.21.15.192:6809/33685] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff7dc06a030 con 0x7ff7dc067530 2023-10-06T03:31:13.792 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.784+0000 7ff7f4ff9700 1 -- 172.21.15.110:0/234588845 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_get_version_reply(handle=1 version=47) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff7e407e550 con 0x7ff7f8152fb0 2023-10-06T03:31:13.792 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.788+0000 7ff7fd04b700 1 --2- 172.21.15.110:0/234588845 >> [v2:172.21.15.192:6808/33685,v1:172.21.15.192:6809/33685] conn(0x7ff7dc067530 0x7ff7dc069960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:31:13.792 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.788+0000 7ff7fc84a700 1 --2- 172.21.15.110:0/234588845 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7ff7dc0606a0 0x7ff7dc062b50 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7ff7ec0099b0 tx=0x7ff7ec017000 comp rx=0 tx=0).ready entity=mgr.5505 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:31:13.792 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.788+0000 7ff7fd04b700 1 --2- 172.21.15.110:0/234588845 >> [v2:172.21.15.192:6808/33685,v1:172.21.15.192:6809/33685] conn(0x7ff7dc067530 0x7ff7dc069960 crc :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:31:13.793 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.788+0000 7ff7f4ff9700 1 -- 172.21.15.110:0/234588845 <== osd.7 v2:172.21.15.192:6808/33685 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23999 (crc 0 0 0) 0x7ff7dc06a030 con 0x7ff7dc067530 2023-10-06T03:31:13.811 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.808+0000 7ff7fd84c700 1 -- 172.21.15.110:0/234588845 --> [v2:172.21.15.192:6808/33685,v1:172.21.15.192:6809/33685] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7ff7f80cc530 con 0x7ff7dc067530 2023-10-06T03:31:13.812 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.808+0000 7ff7f4ff9700 1 -- 172.21.15.110:0/234588845 <== osd.7 v2:172.21.15.192:6808/33685 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7ff7f80cc530 con 0x7ff7dc067530 2023-10-06T03:31:13.812 INFO:teuthology.orchestra.run.smithi110.stdout:{ 2023-10-06T03:31:13.813 INFO:teuthology.orchestra.run.smithi110.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-10-06T03:31:13.813 INFO:teuthology.orchestra.run.smithi110.stdout:} 2023-10-06T03:31:13.814 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.808+0000 7ff7fd84c700 1 -- 172.21.15.110:0/234588845 >> [v2:172.21.15.192:6808/33685,v1:172.21.15.192:6809/33685] conn(0x7ff7dc067530 msgr2=0x7ff7dc069960 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:31:13.814 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.808+0000 7ff7fd84c700 1 --2- 172.21.15.110:0/234588845 >> [v2:172.21.15.192:6808/33685,v1:172.21.15.192:6809/33685] conn(0x7ff7dc067530 0x7ff7dc069960 crc :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:13.814 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.808+0000 7ff7fd84c700 1 -- 172.21.15.110:0/234588845 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7ff7dc0606a0 msgr2=0x7ff7dc062b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:31:13.814 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.808+0000 7ff7fd84c700 1 --2- 172.21.15.110:0/234588845 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7ff7dc0606a0 0x7ff7dc062b50 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7ff7ec0099b0 tx=0x7ff7ec017000 comp rx=0 tx=0).stop 2023-10-06T03:31:13.814 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.808+0000 7ff7fd84c700 1 -- 172.21.15.110:0/234588845 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff7f8152fb0 msgr2=0x7ff7f80af1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:31:13.814 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.808+0000 7ff7fd84c700 1 --2- 172.21.15.110:0/234588845 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff7f8152fb0 0x7ff7f80af1b0 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7ff7e400c4e0 tx=0x7ff7e400c7f0 comp rx=0 tx=0).stop 2023-10-06T03:31:13.814 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.808+0000 7ff7fd04b700 1 -- 172.21.15.110:0/234588845 reap_dead start 2023-10-06T03:31:13.814 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.808+0000 7ff7fd84c700 1 -- 172.21.15.110:0/234588845 shutdown_connections 2023-10-06T03:31:13.815 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.808+0000 7ff7fd84c700 1 -- 172.21.15.110:0/234588845 >> 172.21.15.110:0/234588845 conn(0x7ff7f806c760 msgr2=0x7ff7f80c0770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:31:13.815 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.808+0000 7ff7fd84c700 1 -- 172.21.15.110:0/234588845 shutdown_connections 2023-10-06T03:31:13.815 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.808+0000 7ff7fd84c700 1 -- 172.21.15.110:0/234588845 wait complete. 2023-10-06T03:31:13.825 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 7 2023-10-06T03:31:13.938 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.932+0000 7fab9190c700 1 Processor -- start 2023-10-06T03:31:13.938 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.932+0000 7fab9190c700 1 -- start start 2023-10-06T03:31:13.939 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.932+0000 7fab9190c700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fab8c158d00 0x7fab8c0b1d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:31:13.939 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.932+0000 7fab9190c700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab8c0aef80 con 0x7fab8c06de80 2023-10-06T03:31:13.939 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.932+0000 7fab9190c700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab8c0af540 con 0x7fab8c1590d0 2023-10-06T03:31:13.939 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.932+0000 7fab9190c700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fab8c15d510 con 0x7fab8c158d00 2023-10-06T03:31:13.939 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.932+0000 7fab9090a700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fab8c158d00 0x7fab8c0b1d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:31:13.939 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.932+0000 7fab9090a700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fab8c158d00 0x7fab8c0b1d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:54820/0 (socket says 172.21.15.110:54820) 2023-10-06T03:31:13.939 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.932+0000 7fab9090a700 1 -- 172.21.15.110:0/3301890167 learned_addr learned my addr 172.21.15.110:0/3301890167 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:31:13.939 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.932+0000 7fab9110b700 1 --1- 172.21.15.110:0/3301890167 >> v1:172.21.15.110:6789/0 conn(0x7fab8c06de80 0x7fab8c0af9d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:48190/0 (socket says 172.21.15.110:48190) 2023-10-06T03:31:13.940 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.936+0000 7fab9090a700 1 -- 172.21.15.110:0/3301890167 >> v1:172.21.15.192:6789/0 conn(0x7fab8c1590d0 legacy=0x7fab8c06d770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:31:13.940 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.936+0000 7fab9090a700 1 -- 172.21.15.110:0/3301890167 >> v1:172.21.15.110:6789/0 conn(0x7fab8c06de80 legacy=0x7fab8c0af9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:31:13.941 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.936+0000 7fab9090a700 1 -- 172.21.15.110:0/3301890167 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab8c15d7f0 con 0x7fab8c158d00 2023-10-06T03:31:13.941 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.936+0000 7fab9090a700 1 --2- 172.21.15.110:0/3301890167 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fab8c158d00 0x7fab8c0b1d20 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7fab7c0099b0 tx=0x7fab7c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=22deddea1ffe1b4 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:31:13.941 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.936+0000 7fab8b7fe700 1 -- 172.21.15.110:0/3301890167 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab7c00b860 con 0x7fab8c158d00 2023-10-06T03:31:13.941 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.936+0000 7fab8b7fe700 1 -- 172.21.15.110:0/3301890167 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab7c00b9c0 con 0x7fab8c158d00 2023-10-06T03:31:13.941 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.936+0000 7fab8b7fe700 1 -- 172.21.15.110:0/3301890167 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab7c00bd60 con 0x7fab8c158d00 2023-10-06T03:31:13.942 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.936+0000 7fab9190c700 1 -- 172.21.15.110:0/3301890167 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fab8c158d00 msgr2=0x7fab8c0b1d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:31:13.942 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.936+0000 7fab9190c700 1 --2- 172.21.15.110:0/3301890167 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fab8c158d00 0x7fab8c0b1d20 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7fab7c0099b0 tx=0x7fab7c009250 comp rx=0 tx=0).stop 2023-10-06T03:31:13.942 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.936+0000 7fab9190c700 1 -- 172.21.15.110:0/3301890167 shutdown_connections 2023-10-06T03:31:13.942 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.936+0000 7fab9190c700 1 --2- 172.21.15.110:0/3301890167 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fab8c158d00 0x7fab8c0b1d20 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:13.942 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.936+0000 7fab9190c700 1 -- 172.21.15.110:0/3301890167 >> 172.21.15.110:0/3301890167 conn(0x7fab8c06b6e0 msgr2=0x7fab8c0adbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:31:13.942 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.936+0000 7fab9190c700 1 -- 172.21.15.110:0/3301890167 shutdown_connections 2023-10-06T03:31:13.942 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.936+0000 7fab9190c700 1 -- 172.21.15.110:0/3301890167 wait complete. 2023-10-06T03:31:13.944 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.940+0000 7fab9190c700 1 Processor -- start 2023-10-06T03:31:13.944 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.940+0000 7fab9190c700 1 -- start start 2023-10-06T03:31:13.947 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.940+0000 7fab9190c700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fab8c06de80 0x7fab8c140c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:31:13.947 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.940+0000 7fab9190c700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fab8c158d00 0x7fab8c141150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:31:13.947 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.940+0000 7fab9190c700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fab8c1590d0 0x7fab8c0a8c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:31:13.947 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.940+0000 7fab9190c700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab8c141690 con 0x7fab8c158d00 2023-10-06T03:31:13.947 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.940+0000 7fab9190c700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab8c0c91d0 con 0x7fab8c06de80 2023-10-06T03:31:13.948 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.940+0000 7fab9190c700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab8c15c2a0 con 0x7fab8c1590d0 2023-10-06T03:31:13.948 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.940+0000 7fab8bfff700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fab8c158d00 0x7fab8c141150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:31:13.948 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.940+0000 7fab8bfff700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fab8c158d00 0x7fab8c141150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:54826/0 (socket says 172.21.15.110:54826) 2023-10-06T03:31:13.948 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.940+0000 7fab8bfff700 1 -- 172.21.15.110:0/2479414262 learned_addr learned my addr 172.21.15.110:0/2479414262 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:31:13.948 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.940+0000 7fab9090a700 1 --2- 172.21.15.110:0/2479414262 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fab8c06de80 0x7fab8c140c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:31:13.948 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.940+0000 7fab9110b700 1 --2- 172.21.15.110:0/2479414262 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fab8c1590d0 0x7fab8c0a8c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:31:13.948 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.940+0000 7fab8bfff700 1 -- 172.21.15.110:0/2479414262 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fab8c1590d0 msgr2=0x7fab8c0a8c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:31:13.948 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.940+0000 7fab8bfff700 1 --2- 172.21.15.110:0/2479414262 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fab8c1590d0 0x7fab8c0a8c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:13.948 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.940+0000 7fab8bfff700 1 -- 172.21.15.110:0/2479414262 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fab8c06de80 msgr2=0x7fab8c140c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:31:13.948 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.940+0000 7fab8bfff700 1 --2- 172.21.15.110:0/2479414262 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fab8c06de80 0x7fab8c140c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:13.948 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.940+0000 7fab8bfff700 1 -- 172.21.15.110:0/2479414262 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab7c0094f0 con 0x7fab8c158d00 2023-10-06T03:31:13.949 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.940+0000 7fab9090a700 1 --2- 172.21.15.110:0/2479414262 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fab8c06de80 0x7fab8c140c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-06T03:31:13.949 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.940+0000 7fab9110b700 1 --2- 172.21.15.110:0/2479414262 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fab8c1590d0 0x7fab8c0a8c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-06T03:31:13.949 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.940+0000 7fab8bfff700 1 --2- 172.21.15.110:0/2479414262 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fab8c158d00 0x7fab8c141150 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7fab7800c510 tx=0x7fab78007f50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:31:13.949 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.940+0000 7fab88ff9700 1 -- 172.21.15.110:0/2479414262 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab78011590 con 0x7fab8c158d00 2023-10-06T03:31:13.949 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.940+0000 7fab88ff9700 1 -- 172.21.15.110:0/2479414262 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab78011710 con 0x7fab8c158d00 2023-10-06T03:31:13.949 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.940+0000 7fab9190c700 1 -- 172.21.15.110:0/2479414262 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab8c0c88a0 con 0x7fab8c158d00 2023-10-06T03:31:13.949 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.940+0000 7fab88ff9700 1 -- 172.21.15.110:0/2479414262 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab78011ad0 con 0x7fab8c158d00 2023-10-06T03:31:13.949 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.940+0000 7fab9190c700 1 -- 172.21.15.110:0/2479414262 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab8c0c8e60 con 0x7fab8c158d00 2023-10-06T03:31:13.950 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.944+0000 7fab9190c700 1 -- 172.21.15.110:0/2479414262 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab8c071020 con 0x7fab8c158d00 2023-10-06T03:31:13.950 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.944+0000 7fab88ff9700 1 -- 172.21.15.110:0/2479414262 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 13) v1 ==== 80061+0+0 (secure 0 0 0) 0x7fab7801b3b0 con 0x7fab8c158d00 2023-10-06T03:31:13.950 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.944+0000 7fab88ff9700 1 --2- 172.21.15.110:0/2479414262 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fab74060670 0x7fab74062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:31:13.950 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.944+0000 7fab9090a700 1 --2- 172.21.15.110:0/2479414262 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fab74060670 0x7fab74062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:31:13.951 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.944+0000 7fab88ff9700 1 -- 172.21.15.110:0/2479414262 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 6171+0+0 (secure 0 0 0) 0x7fab7807c0d0 con 0x7fab8c158d00 2023-10-06T03:31:13.951 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.944+0000 7fab9090a700 1 --2- 172.21.15.110:0/2479414262 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fab74060670 0x7fab74062b20 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fab7c0099b0 tx=0x7fab7c017000 comp rx=0 tx=0).ready entity=mgr.5505 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:31:13.955 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:13.952+0000 7fab88ff9700 1 -- 172.21.15.110:0/2479414262 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fab7804c340 con 0x7fab8c158d00 2023-10-06T03:31:14.120 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:14.112+0000 7fab9190c700 1 -- 172.21.15.110:0/2479414262 --> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "7", "target": ["mon-mgr", ""]}) v1 -- 0x7fab8c0c9310 con 0x7fab74060670 2023-10-06T03:31:14.120 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:14.116+0000 7fab88ff9700 1 -- 172.21.15.110:0/2479414262 <== mgr.5505 v2:172.21.15.110:6832/33268 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 7 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7fab8c0c9310 con 0x7fab74060670 2023-10-06T03:31:14.122 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:14.116+0000 7fab9190c700 1 -- 172.21.15.110:0/2479414262 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fab74060670 msgr2=0x7fab74062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:31:14.122 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:14.116+0000 7fab9190c700 1 --2- 172.21.15.110:0/2479414262 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fab74060670 0x7fab74062b20 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fab7c0099b0 tx=0x7fab7c017000 comp rx=0 tx=0).stop 2023-10-06T03:31:14.122 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:14.116+0000 7fab9190c700 1 -- 172.21.15.110:0/2479414262 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fab8c158d00 msgr2=0x7fab8c141150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:31:14.122 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:14.116+0000 7fab9190c700 1 --2- 172.21.15.110:0/2479414262 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fab8c158d00 0x7fab8c141150 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7fab7800c510 tx=0x7fab78007f50 comp rx=0 tx=0).stop 2023-10-06T03:31:14.123 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:14.116+0000 7fab9190c700 1 -- 172.21.15.110:0/2479414262 shutdown_connections 2023-10-06T03:31:14.123 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:14.116+0000 7fab9190c700 1 --2- 172.21.15.110:0/2479414262 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fab74060670 0x7fab74062b20 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:14.123 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:14.116+0000 7fab9190c700 1 --2- 172.21.15.110:0/2479414262 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fab8c1590d0 0x7fab8c0a8c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:14.123 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:14.116+0000 7fab9190c700 1 --2- 172.21.15.110:0/2479414262 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fab8c158d00 0x7fab8c141150 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:14.123 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:14.116+0000 7fab9190c700 1 --2- 172.21.15.110:0/2479414262 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fab8c06de80 0x7fab8c140c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:14.123 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:14.116+0000 7fab9190c700 1 -- 172.21.15.110:0/2479414262 >> 172.21.15.110:0/2479414262 conn(0x7fab8c06b6e0 msgr2=0x7fab8c0ad0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:31:14.123 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:14.116+0000 7fab9190c700 1 -- 172.21.15.110:0/2479414262 shutdown_connections 2023-10-06T03:31:14.123 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:14.116+0000 7fab9190c700 1 -- 172.21.15.110:0/2479414262 wait complete. 2023-10-06T03:31:14.124 INFO:teuthology.orchestra.run.smithi110.stderr:instructed osd(s) 7 to deep-scrub 2023-10-06T03:31:14.133 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-06T03:31:14.236 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:14.232+0000 7fdcf3846700 1 Processor -- start 2023-10-06T03:31:14.236 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:14.232+0000 7fdcf3846700 1 -- start start 2023-10-06T03:31:14.237 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:14.232+0000 7fdcf3846700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fdcec1590a0 0x7fdcec0cafc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:31:14.237 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:14.232+0000 7fdcf3846700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fdcec0cb500 0x7fdcec0aff10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:31:14.237 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:14.232+0000 7fdcf3846700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdcec0cab30 con 0x7fdcec158cd0 2023-10-06T03:31:14.237 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:14.232+0000 7fdcf3846700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdcec1594f0 con 0x7fdcec0cb500 2023-10-06T03:31:14.237 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:14.232+0000 7fdcf3846700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdcec0cef30 con 0x7fdcec1590a0 2023-10-06T03:31:14.238 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:14.232+0000 7fdcf1de3700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fdcec0cb500 0x7fdcec0aff10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:31:14.239 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:14.232+0000 7fdcf15e2700 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7fdcec158cd0 0x7fdcec0ce490 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:48192/0 (socket says 172.21.15.110:48192) 2023-10-06T03:31:14.239 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:14.232+0000 7fdcf1de3700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fdcec0cb500 0x7fdcec0aff10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:54838/0 (socket says 172.21.15.110:54838) 2023-10-06T03:31:14.239 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:14.232+0000 7fdcf15e2700 1 -- 172.21.15.110:0/2513455079 learned_addr learned my addr 172.21.15.110:0/2513455079 (peer_addr_for_me v1:172.21.15.110:0/0) 2023-10-06T03:31:14.239 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:14.232+0000 7fdcf0de1700 1 --2- 172.21.15.110:0/2513455079 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fdcec1590a0 0x7fdcec0cafc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:31:14.239 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:14.232+0000 7fdcf1de3700 1 -- 172.21.15.110:0/2513455079 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fdcec1590a0 msgr2=0x7fdcec0cafc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:31:14.239 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:14.232+0000 7fdcf1de3700 1 --2- 172.21.15.110:0/2513455079 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fdcec1590a0 0x7fdcec0cafc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:14.239 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:14.232+0000 7fdcf1de3700 1 -- 172.21.15.110:0/2513455079 >> v1:172.21.15.110:6789/0 conn(0x7fdcec158cd0 legacy=0x7fdcec0ce490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:31:14.239 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:14.232+0000 7fdce3fff700 1 -- 172.21.15.110:0/2513455079 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2250022742 0 0) 0x7fdcec0cab30 con 0x7fdcec158cd0 2023-10-06T03:31:14.239 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:14.232+0000 7fdcf1de3700 1 -- 172.21.15.110:0/2513455079 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdcec0cf210 con 0x7fdcec0cb500 2023-10-06T03:31:14.240 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:14.232+0000 7fdcf0de1700 1 --2- 172.21.15.110:0/2513455079 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fdcec1590a0 0x7fdcec0cafc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-06T03:31:14.240 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:14.232+0000 7fdcf1de3700 1 --2- 172.21.15.110:0/2513455079 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdcec0cb500 0x7fdcec0aff10 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7fdce40098e0 tx=0x7fdce40091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f739faa6a209775b server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:31:14.240 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:14.232+0000 7fdce3fff700 1 -- 172.21.15.110:0/2513455079 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdce4009e80 con 0x7fdcec0cb500 2023-10-06T03:31:14.240 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:14.232+0000 7fdce3fff700 1 -- 172.21.15.110:0/2513455079 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdce400b860 con 0x7fdcec0cb500 2023-10-06T03:31:14.240 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:14.232+0000 7fdce3fff700 1 -- 172.21.15.110:0/2513455079 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdce400bbe0 con 0x7fdcec0cb500 2023-10-06T03:31:14.241 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:14.236+0000 7fdcf3846700 1 -- 172.21.15.110:0/2513455079 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdcec0cb500 msgr2=0x7fdcec0aff10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:31:14.241 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:14.236+0000 7fdcf3846700 1 --2- 172.21.15.110:0/2513455079 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdcec0cb500 0x7fdcec0aff10 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7fdce40098e0 tx=0x7fdce40091a0 comp rx=0 tx=0).stop 2023-10-06T03:31:14.241 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:14.236+0000 7fdcf3846700 1 -- 172.21.15.110:0/2513455079 shutdown_connections 2023-10-06T03:31:14.241 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:14.236+0000 7fdcf3846700 1 --2- 172.21.15.110:0/2513455079 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdcec0cb500 0x7fdcec0aff10 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:14.241 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:14.236+0000 7fdcf3846700 1 --2- 172.21.15.110:0/2513455079 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fdcec1590a0 0x7fdcec0cafc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:14.241 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:14.236+0000 7fdcf3846700 1 -- 172.21.15.110:0/2513455079 >> 172.21.15.110:0/2513455079 conn(0x7fdcec067040 msgr2=0x7fdcec067440 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:31:14.241 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:14.236+0000 7fdcf3846700 1 -- 172.21.15.110:0/2513455079 shutdown_connections 2023-10-06T03:31:14.241 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:14.236+0000 7fdcf3846700 1 -- 172.21.15.110:0/2513455079 wait complete. 2023-10-06T03:31:14.242 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:14.236+0000 7fdcf3846700 1 Processor -- start 2023-10-06T03:31:14.242 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:14.236+0000 7fdcf3846700 1 -- start start 2023-10-06T03:31:14.245 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:14.240+0000 7fdcf3846700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fdcec0cb500 0x7fdcec0b2d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:31:14.245 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:14.240+0000 7fdcf3846700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdcec158cd0 0x7fdcec1514d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:31:14.245 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:14.240+0000 7fdcf3846700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdcec1590a0 0x7fdcec15b890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:31:14.246 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:14.240+0000 7fdcf3846700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcec15c650 con 0x7fdcec158cd0 2023-10-06T03:31:14.246 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:14.240+0000 7fdcf3846700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcec0c57a0 con 0x7fdcec1590a0 2023-10-06T03:31:14.246 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:14.240+0000 7fdcf3846700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdcec15a5e0 con 0x7fdcec0cb500 2023-10-06T03:31:14.246 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:14.240+0000 7fdcf0de1700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdcec158cd0 0x7fdcec1514d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:31:14.246 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:14.240+0000 7fdcf0de1700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdcec158cd0 0x7fdcec1514d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:54846/0 (socket says 172.21.15.110:54846) 2023-10-06T03:31:14.246 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:14.240+0000 7fdcf0de1700 1 -- 172.21.15.110:0/3158396920 learned_addr learned my addr 172.21.15.110:0/3158396920 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:31:14.246 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:14.240+0000 7fdcf15e2700 1 --2- 172.21.15.110:0/3158396920 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fdcec0cb500 0x7fdcec0b2d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:31:14.246 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:14.240+0000 7fdcf1de3700 1 --2- 172.21.15.110:0/3158396920 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdcec1590a0 0x7fdcec15b890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:31:14.246 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:14.240+0000 7fdcf0de1700 1 -- 172.21.15.110:0/3158396920 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fdcec0cb500 msgr2=0x7fdcec0b2d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:31:14.246 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:14.240+0000 7fdcf0de1700 1 --2- 172.21.15.110:0/3158396920 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fdcec0cb500 0x7fdcec0b2d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:14.247 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:14.240+0000 7fdcf0de1700 1 -- 172.21.15.110:0/3158396920 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdcec1590a0 msgr2=0x7fdcec15b890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:31:14.247 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:14.240+0000 7fdcf0de1700 1 --2- 172.21.15.110:0/3158396920 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdcec1590a0 0x7fdcec15b890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:14.247 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:14.240+0000 7fdcf0de1700 1 -- 172.21.15.110:0/3158396920 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdce4009420 con 0x7fdcec158cd0 2023-10-06T03:31:14.247 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:14.240+0000 7fdcf0de1700 1 --2- 172.21.15.110:0/3158396920 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdcec158cd0 0x7fdcec1514d0 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7fdce800d180 tx=0x7fdce80099f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:31:14.247 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:14.240+0000 7fdce17fa700 1 -- 172.21.15.110:0/3158396920 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdce800dcc0 con 0x7fdcec158cd0 2023-10-06T03:31:14.247 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:14.240+0000 7fdcf3846700 1 -- 172.21.15.110:0/3158396920 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdcec15a8c0 con 0x7fdcec158cd0 2023-10-06T03:31:14.247 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:14.240+0000 7fdce17fa700 1 -- 172.21.15.110:0/3158396920 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdce800f040 con 0x7fdcec158cd0 2023-10-06T03:31:14.247 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:14.240+0000 7fdcf3846700 1 -- 172.21.15.110:0/3158396920 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdcec0c5430 con 0x7fdcec158cd0 2023-10-06T03:31:14.247 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:14.240+0000 7fdce17fa700 1 -- 172.21.15.110:0/3158396920 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdce800de20 con 0x7fdcec158cd0 2023-10-06T03:31:14.248 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:14.240+0000 7fdce17fa700 1 -- 172.21.15.110:0/3158396920 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 13) v1 ==== 80061+0+0 (secure 0 0 0) 0x7fdce8015860 con 0x7fdcec158cd0 2023-10-06T03:31:14.248 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:14.240+0000 7fdcf3846700 1 -- 172.21.15.110:0/3158396920 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdcec0c5910 con 0x7fdcec158cd0 2023-10-06T03:31:14.248 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:14.244+0000 7fdce17fa700 1 --2- 172.21.15.110:0/3158396920 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fdcd8060670 0x7fdcd8062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:31:14.248 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:14.244+0000 7fdcf15e2700 1 --2- 172.21.15.110:0/3158396920 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fdcd8060670 0x7fdcd8062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:31:14.248 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:14.244+0000 7fdce17fa700 1 -- 172.21.15.110:0/3158396920 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 6171+0+0 (secure 0 0 0) 0x7fdce807cfe0 con 0x7fdcec158cd0 2023-10-06T03:31:14.249 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:14.244+0000 7fdcf15e2700 1 --2- 172.21.15.110:0/3158396920 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fdcd8060670 0x7fdcd8062b20 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fdcdc001360 tx=0x7fdcdc009040 comp rx=0 tx=0).ready entity=mgr.5505 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:31:14.254 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:14.248+0000 7fdce17fa700 1 -- 172.21.15.110:0/3158396920 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fdce804d1f0 con 0x7fdcec158cd0 2023-10-06T03:31:14.413 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:14.408+0000 7fdcf3846700 1 -- 172.21.15.110:0/3158396920 --> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fdcec064770 con 0x7fdcd8060670 2023-10-06T03:31:14.416 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:14.412+0000 7fdce17fa700 1 -- 172.21.15.110:0/3158396920 <== mgr.5505 v2:172.21.15.110:6832/33268 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+70875 (secure 0 0 0) 0x7fdcec064770 con 0x7fdcd8060670 2023-10-06T03:31:14.417 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-10-06T03:31:14.419 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:14.412+0000 7fdcf3846700 1 -- 172.21.15.110:0/3158396920 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fdcd8060670 msgr2=0x7fdcd8062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:31:14.419 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:14.412+0000 7fdcf3846700 1 --2- 172.21.15.110:0/3158396920 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fdcd8060670 0x7fdcd8062b20 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fdcdc001360 tx=0x7fdcdc009040 comp rx=0 tx=0).stop 2023-10-06T03:31:14.419 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:14.412+0000 7fdcf3846700 1 -- 172.21.15.110:0/3158396920 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdcec158cd0 msgr2=0x7fdcec1514d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:31:14.420 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:14.412+0000 7fdcf3846700 1 --2- 172.21.15.110:0/3158396920 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdcec158cd0 0x7fdcec1514d0 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7fdce800d180 tx=0x7fdce80099f0 comp rx=0 tx=0).stop 2023-10-06T03:31:14.420 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:14.412+0000 7fdcf3846700 1 -- 172.21.15.110:0/3158396920 shutdown_connections 2023-10-06T03:31:14.420 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:14.412+0000 7fdcf3846700 1 --2- 172.21.15.110:0/3158396920 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fdcd8060670 0x7fdcd8062b20 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:14.420 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:14.412+0000 7fdcf3846700 1 --2- 172.21.15.110:0/3158396920 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdcec1590a0 0x7fdcec15b890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:14.420 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:14.412+0000 7fdcf3846700 1 --2- 172.21.15.110:0/3158396920 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdcec158cd0 0x7fdcec1514d0 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:14.420 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:14.412+0000 7fdcf3846700 1 --2- 172.21.15.110:0/3158396920 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fdcec0cb500 0x7fdcec0b2d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:14.421 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:14.412+0000 7fdcf3846700 1 -- 172.21.15.110:0/3158396920 >> 172.21.15.110:0/3158396920 conn(0x7fdcec067040 msgr2=0x7fdcec151f10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:31:14.421 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:14.412+0000 7fdcf3846700 1 -- 172.21.15.110:0/3158396920 shutdown_connections 2023-10-06T03:31:14.421 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:14.412+0000 7fdcf3846700 1 -- 172.21.15.110:0/3158396920 wait complete. 2023-10-06T03:31:14.421 INFO:teuthology.orchestra.run.smithi110.stderr:dumped all 2023-10-06T03:31:14.431 INFO:teuthology.orchestra.run.smithi110.stdout:{"pg_ready":true,"pg_map":{"version":12,"stamp":"2023-10-06T03:31:13.948456+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":308,"num_read_kb":512,"num_write":199,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":139,"ondisk_log_size":139,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":7,"kb":754974720,"kb_used":2400104,"kb_used_data":5416,"kb_used_omap":0,"kb_used_meta":2394624,"kb_avail":752574616,"statfs":{"total":773094113280,"available":770636406784,"internally_reserved":0,"allocated":5545984,"data_stored":3168592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2452094976},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.559089"},"pg_stats":[{"pgid":"1.0","version":"44'139","reported_seq":264,"reported_epoch":46,"state":"active+clean","last_fresh":"2023-10-06T03:31:04.872804+0000","last_change":"2023-10-06T03:29:21.978311+0000","last_active":"2023-10-06T03:31:04.872804+0000","last_peered":"2023-10-06T03:31:04.872804+0000","last_clean":"2023-10-06T03:31:04.872804+0000","last_became_active":"2023-10-06T03:29:21.978017+0000","last_became_peered":"2023-10-06T03:29:21.978017+0000","last_unstale":"2023-10-06T03:31:04.872804+0000","last_undegraded":"2023-10-06T03:31:04.872804+0000","last_fullsized":"2023-10-06T03:31:04.872804+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:20.957955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:20.957955+0000","last_clean_scrub_stamp":"2023-10-06T03:29:20.957955+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T07:33:10.429084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":308,"num_read_kb":512,"num_write":199,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":33,"reported_epoch":46,"state":"active+clean","last_fresh":"2023-10-06T03:31:05.363600+0000","last_change":"2023-10-06T03:29:29.386120+0000","last_active":"2023-10-06T03:31:05.363600+0000","last_peered":"2023-10-06T03:31:05.363600+0000","last_clean":"2023-10-06T03:31:05.363600+0000","last_became_active":"2023-10-06T03:29:29.385692+0000","last_became_peered":"2023-10-06T03:29:29.385692+0000","last_unstale":"2023-10-06T03:31:05.363600+0000","last_undegraded":"2023-10-06T03:31:05.363600+0000","last_fullsized":"2023-10-06T03:31:05.363600+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:28.357506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:28.357506+0000","last_clean_scrub_stamp":"2023-10-06T03:29:28.357506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T10:21:35.864895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":33,"reported_epoch":46,"state":"active+clean","last_fresh":"2023-10-06T03:31:04.872618+0000","last_change":"2023-10-06T03:29:29.384353+0000","last_active":"2023-10-06T03:31:04.872618+0000","last_peered":"2023-10-06T03:31:04.872618+0000","last_clean":"2023-10-06T03:31:04.872618+0000","last_became_active":"2023-10-06T03:29:29.383884+0000","last_became_peered":"2023-10-06T03:29:29.383884+0000","last_unstale":"2023-10-06T03:31:04.872618+0000","last_undegraded":"2023-10-06T03:31:04.872618+0000","last_fullsized":"2023-10-06T03:31:04.872618+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:28.357506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:28.357506+0000","last_clean_scrub_stamp":"2023-10-06T03:29:28.357506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T14:43:09.107140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":45,"reported_epoch":47,"state":"active+clean","last_fresh":"2023-10-06T03:31:12.508603+0000","last_change":"2023-10-06T03:31:12.508580+0000","last_active":"2023-10-06T03:31:12.508603+0000","last_peered":"2023-10-06T03:31:12.508603+0000","last_clean":"2023-10-06T03:31:12.508603+0000","last_became_active":"2023-10-06T03:29:29.605489+0000","last_became_peered":"2023-10-06T03:29:29.605489+0000","last_unstale":"2023-10-06T03:31:12.508603+0000","last_undegraded":"2023-10-06T03:31:12.508603+0000","last_fullsized":"2023-10-06T03:31:12.508603+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:31:12.508497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:31:12.508497+0000","last_clean_scrub_stamp":"2023-10-06T03:31:12.508497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T14:49:34.037015+0000","scrub_duration":0.002597813,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":33,"reported_epoch":46,"state":"active+clean","last_fresh":"2023-10-06T03:31:05.362954+0000","last_change":"2023-10-06T03:29:29.384415+0000","last_active":"2023-10-06T03:31:05.362954+0000","last_peered":"2023-10-06T03:31:05.362954+0000","last_clean":"2023-10-06T03:31:05.362954+0000","last_became_active":"2023-10-06T03:29:29.383894+0000","last_became_peered":"2023-10-06T03:29:29.383894+0000","last_unstale":"2023-10-06T03:31:05.362954+0000","last_undegraded":"2023-10-06T03:31:05.362954+0000","last_fullsized":"2023-10-06T03:31:05.362954+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:28.357506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:28.357506+0000","last_clean_scrub_stamp":"2023-10-06T03:29:28.357506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T13:09:00.657107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":34,"reported_epoch":47,"state":"active+clean","last_fresh":"2023-10-06T03:31:07.224848+0000","last_change":"2023-10-06T03:29:29.382183+0000","last_active":"2023-10-06T03:31:07.224848+0000","last_peered":"2023-10-06T03:31:07.224848+0000","last_clean":"2023-10-06T03:31:07.224848+0000","last_became_active":"2023-10-06T03:29:29.381918+0000","last_became_peered":"2023-10-06T03:29:29.381918+0000","last_unstale":"2023-10-06T03:31:07.224848+0000","last_undegraded":"2023-10-06T03:31:07.224848+0000","last_fullsized":"2023-10-06T03:31:07.224848+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:28.357506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:28.357506+0000","last_clean_scrub_stamp":"2023-10-06T03:29:28.357506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":33,"reported_epoch":46,"state":"active+clean","last_fresh":"2023-10-06T03:31:04.875983+0000","last_change":"2023-10-06T03:29:29.384279+0000","last_active":"2023-10-06T03:31:04.875983+0000","last_peered":"2023-10-06T03:31:04.875983+0000","last_clean":"2023-10-06T03:31:04.875983+0000","last_became_active":"2023-10-06T03:29:29.383533+0000","last_became_peered":"2023-10-06T03:29:29.383533+0000","last_unstale":"2023-10-06T03:31:04.875983+0000","last_undegraded":"2023-10-06T03:31:04.875983+0000","last_fullsized":"2023-10-06T03:31:04.875983+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:28.357506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:28.357506+0000","last_clean_scrub_stamp":"2023-10-06T03:29:28.357506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T08:34:19.641525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":34,"reported_epoch":47,"state":"active+clean","last_fresh":"2023-10-06T03:31:07.226011+0000","last_change":"2023-10-06T03:29:29.386238+0000","last_active":"2023-10-06T03:31:07.226011+0000","last_peered":"2023-10-06T03:31:07.226011+0000","last_clean":"2023-10-06T03:31:07.226011+0000","last_became_active":"2023-10-06T03:29:29.385865+0000","last_became_peered":"2023-10-06T03:29:29.385865+0000","last_unstale":"2023-10-06T03:31:07.226011+0000","last_undegraded":"2023-10-06T03:31:07.226011+0000","last_fullsized":"2023-10-06T03:31:07.226011+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:28.357506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:28.357506+0000","last_clean_scrub_stamp":"2023-10-06T03:29:28.357506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":32,"reported_epoch":46,"state":"active+clean","last_fresh":"2023-10-06T03:31:05.367899+0000","last_change":"2023-10-06T03:29:29.387402+0000","last_active":"2023-10-06T03:31:05.367899+0000","last_peered":"2023-10-06T03:31:05.367899+0000","last_clean":"2023-10-06T03:31:05.367899+0000","last_became_active":"2023-10-06T03:29:29.386812+0000","last_became_peered":"2023-10-06T03:29:29.386812+0000","last_unstale":"2023-10-06T03:31:05.367899+0000","last_undegraded":"2023-10-06T03:31:05.367899+0000","last_fullsized":"2023-10-06T03:31:05.367899+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:29:28.357506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:29:28.357506+0000","last_clean_scrub_stamp":"2023-10-06T03:29:28.357506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T12:09:43.895810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":308,"num_read_kb":512,"num_write":199,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":139,"ondisk_log_size":139,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509466,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298400,"kb_used_data":1112,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94073440,"statfs":{"total":96636764160,"available":96331202560,"internally_reserved":0,"allocated":1138688,"data_stored":838850,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Oct 6 03:30:26 2023","interfaces":[{"interface":"back","average":{"1min":0.878,"5min":0.878,"15min":0.878},"min":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"max":{"1min":1.228,"5min":1.228,"15min":1.228},"last":1.052},{"interface":"front","average":{"1min":0.878,"5min":0.878,"15min":0.878},"min":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":1.181,"5min":1.181,"15min":1.181},"last":0.81000000000000005}]},{"osd":1,"last update":"Fri Oct 6 03:30:26 2023","interfaces":[{"interface":"back","average":{"1min":0.93000000000000005,"5min":0.93000000000000005,"15min":0.93000000000000005},"min":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"max":{"1min":1.2889999999999999,"5min":1.2889999999999999,"15min":1.2889999999999999},"last":0.70799999999999996},{"interface":"front","average":{"1min":0.94299999999999995,"5min":0.94299999999999995,"15min":0.94299999999999995},"min":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"max":{"1min":1.341,"5min":1.341,"15min":1.341},"last":0.56000000000000005}]},{"osd":2,"last update":"Fri Oct 6 03:30:26 2023","interfaces":[{"interface":"back","average":{"1min":0.95299999999999996,"5min":0.95299999999999996,"15min":0.95299999999999996},"min":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.72599999999999998},"max":{"1min":1.2050000000000001,"5min":1.2050000000000001,"15min":1.2050000000000001},"last":0.66600000000000004},{"interface":"front","average":{"1min":1.006,"5min":1.006,"15min":1.006},"min":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"max":{"1min":1.409,"5min":1.409,"15min":1.409},"last":0.879}]},{"osd":3,"last update":"Fri Oct 6 03:30:26 2023","interfaces":[{"interface":"back","average":{"1min":0.86299999999999999,"5min":0.86299999999999999,"15min":0.86299999999999999},"min":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"max":{"1min":1.1339999999999999,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":1.0149999999999999},{"interface":"front","average":{"1min":0.96899999999999997,"5min":0.96899999999999997,"15min":0.96899999999999997},"min":{"1min":0.63400000000000001,"5min":0.63400000000000001,"15min":0.63400000000000001},"max":{"1min":1.256,"5min":1.256,"15min":1.256},"last":0.97499999999999998}]},{"osd":4,"last update":"Fri Oct 6 03:30:26 2023","interfaces":[{"interface":"back","average":{"1min":0.91900000000000004,"5min":0.91900000000000004,"15min":0.91900000000000004},"min":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"max":{"1min":1.2230000000000001,"5min":1.2230000000000001,"15min":1.2230000000000001},"last":0.84899999999999998},{"interface":"front","average":{"1min":1.0529999999999999,"5min":1.0529999999999999,"15min":1.0529999999999999},"min":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"max":{"1min":1.4750000000000001,"5min":1.4750000000000001,"15min":1.4750000000000001},"last":1.2569999999999999}]},{"osd":5,"last update":"Fri Oct 6 03:30:26 2023","interfaces":[{"interface":"back","average":{"1min":1.0229999999999999,"5min":1.0229999999999999,"15min":1.0229999999999999},"min":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":0.77900000000000003},"max":{"1min":1.431,"5min":1.431,"15min":1.431},"last":1.121},{"interface":"front","average":{"1min":1.105,"5min":1.105,"15min":1.105},"min":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"max":{"1min":1.407,"5min":1.407,"15min":1.407},"last":1.1699999999999999}]},{"osd":6,"last update":"Fri Oct 6 03:30:26 2023","interfaces":[{"interface":"back","average":{"1min":1.1399999999999999,"5min":1.1399999999999999,"15min":1.1399999999999999},"min":{"1min":0.86799999999999999,"5min":0.86799999999999999,"15min":0.86799999999999999},"max":{"1min":1.526,"5min":1.526,"15min":1.526},"last":1.147},{"interface":"front","average":{"1min":1.198,"5min":1.198,"15min":1.198},"min":{"1min":0.98399999999999999,"5min":0.98399999999999999,"15min":0.98399999999999999},"max":{"1min":1.6719999999999999,"5min":1.6719999999999999,"15min":1.6719999999999999},"last":1.2190000000000001}]}]},{"osd":6,"up_from":16,"seq":68719476763,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297820,"kb_used_data":532,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074020,"statfs":{"total":96636764160,"available":96331796480,"internally_reserved":0,"allocated":544768,"data_stored":248482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Oct 6 03:30:26 2023","interfaces":[{"interface":"back","average":{"1min":0.876,"5min":0.876,"15min":0.876},"min":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"max":{"1min":1.149,"5min":1.149,"15min":1.149},"last":0.80600000000000005},{"interface":"front","average":{"1min":0.90400000000000003,"5min":0.90400000000000003,"15min":0.90400000000000003},"min":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"max":{"1min":1.254,"5min":1.254,"15min":1.254},"last":0.73999999999999999}]},{"osd":1,"last update":"Fri Oct 6 03:30:26 2023","interfaces":[{"interface":"back","average":{"1min":0.81299999999999994,"5min":0.81299999999999994,"15min":0.81299999999999994},"min":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"max":{"1min":1.4730000000000001,"5min":1.4730000000000001,"15min":1.4730000000000001},"last":0.77200000000000002},{"interface":"front","average":{"1min":0.97099999999999997,"5min":0.97099999999999997,"15min":0.97099999999999997},"min":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"max":{"1min":1.3440000000000001,"5min":1.3440000000000001,"15min":1.3440000000000001},"last":1.052}]},{"osd":2,"last update":"Fri Oct 6 03:30:26 2023","interfaces":[{"interface":"back","average":{"1min":0.94099999999999995,"5min":0.94099999999999995,"15min":0.94099999999999995},"min":{"1min":0.629,"5min":0.629,"15min":0.629},"max":{"1min":1.236,"5min":1.236,"15min":1.236},"last":1.206},{"interface":"front","average":{"1min":0.95599999999999996,"5min":0.95599999999999996,"15min":0.95599999999999996},"min":{"1min":0.623,"5min":0.623,"15min":0.623},"max":{"1min":1.3660000000000001,"5min":1.3660000000000001,"15min":1.3660000000000001},"last":0.63300000000000001}]},{"osd":3,"last update":"Fri Oct 6 03:30:26 2023","interfaces":[{"interface":"back","average":{"1min":0.99299999999999999,"5min":0.99299999999999999,"15min":0.99299999999999999},"min":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":1.3660000000000001,"5min":1.3660000000000001,"15min":1.3660000000000001},"last":0.88800000000000001},{"interface":"front","average":{"1min":0.95599999999999996,"5min":0.95599999999999996,"15min":0.95599999999999996},"min":{"1min":0.76300000000000001,"5min":0.76300000000000001,"15min":0.76300000000000001},"max":{"1min":1.2609999999999999,"5min":1.2609999999999999,"15min":1.2609999999999999},"last":0.91600000000000004}]},{"osd":4,"last update":"Fri Oct 6 03:30:26 2023","interfaces":[{"interface":"back","average":{"1min":1.1040000000000001,"5min":1.1040000000000001,"15min":1.1040000000000001},"min":{"1min":0.875,"5min":0.875,"15min":0.875},"max":{"1min":1.2869999999999999,"5min":1.2869999999999999,"15min":1.2869999999999999},"last":0.95299999999999996},{"interface":"front","average":{"1min":1.1140000000000001,"5min":1.1140000000000001,"15min":1.1140000000000001},"min":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.73399999999999999},"max":{"1min":1.4450000000000001,"5min":1.4450000000000001,"15min":1.4450000000000001},"last":1.113}]},{"osd":5,"last update":"Fri Oct 6 03:30:26 2023","interfaces":[{"interface":"back","average":{"1min":1.105,"5min":1.105,"15min":1.105},"min":{"1min":0.86799999999999999,"5min":0.86799999999999999,"15min":0.86799999999999999},"max":{"1min":1.613,"5min":1.613,"15min":1.613},"last":1.01},{"interface":"front","average":{"1min":1.1699999999999999,"5min":1.1699999999999999,"15min":1.1699999999999999},"min":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"max":{"1min":1.351,"5min":1.351,"15min":1.351},"last":1.155}]},{"osd":7,"last update":"Fri Oct 6 03:30:26 2023","interfaces":[{"interface":"back","average":{"1min":1.2370000000000001,"5min":1.2370000000000001,"15min":1.2370000000000001},"min":{"1min":1.012,"5min":1.012,"15min":1.012},"max":{"1min":1.5409999999999999,"5min":1.5409999999999999,"15min":1.5409999999999999},"last":0.98599999999999999},{"interface":"front","average":{"1min":1.212,"5min":1.212,"15min":1.212},"min":{"1min":0.96099999999999997,"5min":0.96099999999999997,"15min":0.96099999999999997},"max":{"1min":1.704,"5min":1.704,"15min":1.704},"last":1.085}]}]},{"osd":5,"up_from":16,"seq":68719476763,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297820,"kb_used_data":532,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074020,"statfs":{"total":96636764160,"available":96331796480,"internally_reserved":0,"allocated":544768,"data_stored":248482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Oct 6 03:30:23 2023","interfaces":[{"interface":"back","average":{"1min":0.86399999999999999,"5min":0.86399999999999999,"15min":0.86399999999999999},"min":{"1min":0.505,"5min":0.505,"15min":0.505},"max":{"1min":1.4079999999999999,"5min":1.4079999999999999,"15min":1.4079999999999999},"last":1.0329999999999999},{"interface":"front","average":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":0.89800000000000002},"min":{"1min":0.64500000000000002,"5min":0.64500000000000002,"15min":0.64500000000000002},"max":{"1min":1.3779999999999999,"5min":1.3779999999999999,"15min":1.3779999999999999},"last":0.78300000000000003}]},{"osd":1,"last update":"Fri Oct 6 03:30:23 2023","interfaces":[{"interface":"back","average":{"1min":0.97799999999999998,"5min":0.97799999999999998,"15min":0.97799999999999998},"min":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":1.2829999999999999,"5min":1.2829999999999999,"15min":1.2829999999999999},"last":0.91300000000000003},{"interface":"front","average":{"1min":0.85999999999999999,"5min":0.85999999999999999,"15min":0.85999999999999999},"min":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"max":{"1min":1.581,"5min":1.581,"15min":1.581},"last":0.753}]},{"osd":2,"last update":"Fri Oct 6 03:30:23 2023","interfaces":[{"interface":"back","average":{"1min":1,"5min":1,"15min":1},"min":{"1min":0.67300000000000004,"5min":0.67300000000000004,"15min":0.67300000000000004},"max":{"1min":1.258,"5min":1.258,"15min":1.258},"last":1.0629999999999999},{"interface":"front","average":{"1min":0.90600000000000003,"5min":0.90600000000000003,"15min":0.90600000000000003},"min":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.67100000000000004},"max":{"1min":1.367,"5min":1.367,"15min":1.367},"last":0.82399999999999995}]},{"osd":3,"last update":"Fri Oct 6 03:30:23 2023","interfaces":[{"interface":"back","average":{"1min":1.034,"5min":1.034,"15min":1.034},"min":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.77200000000000002},"max":{"1min":1.4079999999999999,"5min":1.4079999999999999,"15min":1.4079999999999999},"last":0.59399999999999997},{"interface":"front","average":{"1min":1.034,"5min":1.034,"15min":1.034},"min":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"max":{"1min":1.244,"5min":1.244,"15min":1.244},"last":0.872}]},{"osd":4,"last update":"Fri Oct 6 03:30:23 2023","interfaces":[{"interface":"back","average":{"1min":1.0920000000000001,"5min":1.0920000000000001,"15min":1.0920000000000001},"min":{"1min":0.752,"5min":0.752,"15min":0.752},"max":{"1min":2.2050000000000001,"5min":2.2050000000000001,"15min":2.2050000000000001},"last":0.96999999999999997},{"interface":"front","average":{"1min":1.1870000000000001,"5min":1.1870000000000001,"15min":1.1870000000000001},"min":{"1min":0.90900000000000003,"5min":0.90900000000000003,"15min":0.90900000000000003},"max":{"1min":1.518,"5min":1.518,"15min":1.518},"last":1.0069999999999999}]},{"osd":6,"last update":"Fri Oct 6 03:30:23 2023","interfaces":[{"interface":"back","average":{"1min":1.1040000000000001,"5min":1.1040000000000001,"15min":1.1040000000000001},"min":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"max":{"1min":1.3169999999999999,"5min":1.3169999999999999,"15min":1.3169999999999999},"last":0.95099999999999996},{"interface":"front","average":{"1min":1.2370000000000001,"5min":1.2370000000000001,"15min":1.2370000000000001},"min":{"1min":0.94599999999999995,"5min":0.94599999999999995,"15min":0.94599999999999995},"max":{"1min":2.3809999999999998,"5min":2.3809999999999998,"15min":2.3809999999999998},"last":1.1220000000000001}]},{"osd":7,"last update":"Fri Oct 6 03:30:23 2023","interfaces":[{"interface":"back","average":{"1min":1.2370000000000001,"5min":1.2370000000000001,"15min":1.2370000000000001},"min":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.79500000000000004},"max":{"1min":1.637,"5min":1.637,"15min":1.637},"last":1.095},{"interface":"front","average":{"1min":1.2929999999999999,"5min":1.2929999999999999,"15min":1.2929999999999999},"min":{"1min":1.0429999999999999,"5min":1.0429999999999999,"15min":1.0429999999999999},"max":{"1min":2.3330000000000002,"5min":2.3330000000000002,"15min":2.3330000000000002},"last":1.155}]}]},{"osd":4,"up_from":16,"seq":68719476762,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":297820,"kb_used_data":532,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074020,"statfs":{"total":96636764160,"available":96331796480,"internally_reserved":0,"allocated":544768,"data_stored":248482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Oct 6 03:30:25 2023","interfaces":[{"interface":"back","average":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"min":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"max":{"1min":1.129,"5min":1.129,"15min":1.129},"last":0.77300000000000002},{"interface":"front","average":{"1min":0.96299999999999997,"5min":0.96299999999999997,"15min":0.96299999999999997},"min":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"max":{"1min":1.629,"5min":1.629,"15min":1.629},"last":1.1479999999999999}]},{"osd":1,"last update":"Fri Oct 6 03:30:25 2023","interfaces":[{"interface":"back","average":{"1min":0.89300000000000002,"5min":0.89300000000000002,"15min":0.89300000000000002},"min":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"max":{"1min":1.23,"5min":1.23,"15min":1.23},"last":1.105},{"interface":"front","average":{"1min":0.86799999999999999,"5min":0.86799999999999999,"15min":0.86799999999999999},"min":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"max":{"1min":1.3680000000000001,"5min":1.3680000000000001,"15min":1.3680000000000001},"last":1.0249999999999999}]},{"osd":2,"last update":"Fri Oct 6 03:30:25 2023","interfaces":[{"interface":"back","average":{"1min":0.98999999999999999,"5min":0.98999999999999999,"15min":0.98999999999999999},"min":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":1.5549999999999999,"5min":1.5549999999999999,"15min":1.5549999999999999},"last":0.88400000000000001},{"interface":"front","average":{"1min":0.92100000000000004,"5min":0.92100000000000004,"15min":0.92100000000000004},"min":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.60799999999999998},"max":{"1min":1.306,"5min":1.306,"15min":1.306},"last":0.96999999999999997}]},{"osd":3,"last update":"Fri Oct 6 03:30:25 2023","interfaces":[{"interface":"back","average":{"1min":1.1040000000000001,"5min":1.1040000000000001,"15min":1.1040000000000001},"min":{"1min":0.84499999999999997,"5min":0.84499999999999997,"15min":0.84499999999999997},"max":{"1min":1.5580000000000001,"5min":1.5580000000000001,"15min":1.5580000000000001},"last":0.82099999999999995},{"interface":"front","average":{"1min":1.04,"5min":1.04,"15min":1.04},"min":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.67500000000000004},"max":{"1min":1.6699999999999999,"5min":1.6699999999999999,"15min":1.6699999999999999},"last":1.179}]},{"osd":5,"last update":"Fri Oct 6 03:30:25 2023","interfaces":[{"interface":"back","average":{"1min":1.044,"5min":1.044,"15min":1.044},"min":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.73399999999999999},"max":{"1min":1.5169999999999999,"5min":1.5169999999999999,"15min":1.5169999999999999},"last":0.85999999999999999},{"interface":"front","average":{"1min":1.266,"5min":1.266,"15min":1.266},"min":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":0.82799999999999996},"max":{"1min":2.2029999999999998,"5min":2.2029999999999998,"15min":2.2029999999999998},"last":1.0620000000000001}]},{"osd":6,"last update":"Fri Oct 6 03:30:25 2023","interfaces":[{"interface":"back","average":{"1min":1.181,"5min":1.181,"15min":1.181},"min":{"1min":0.79900000000000004,"5min":0.79900000000000004,"15min":0.79900000000000004},"max":{"1min":2.0209999999999999,"5min":2.0209999999999999,"15min":2.0209999999999999},"last":1.298},{"interface":"front","average":{"1min":1.179,"5min":1.179,"15min":1.179},"min":{"1min":0.71399999999999997,"5min":0.71399999999999997,"15min":0.71399999999999997},"max":{"1min":2.1440000000000001,"5min":2.1440000000000001,"15min":2.1440000000000001},"last":1.1279999999999999}]},{"osd":7,"last update":"Fri Oct 6 03:30:25 2023","interfaces":[{"interface":"back","average":{"1min":1.337,"5min":1.337,"15min":1.337},"min":{"1min":0.94399999999999995,"5min":0.94399999999999995,"15min":0.94399999999999995},"max":{"1min":2.0939999999999999,"5min":2.0939999999999999,"15min":2.0939999999999999},"last":1.2070000000000001},{"interface":"front","average":{"1min":1.3089999999999999,"5min":1.3089999999999999,"15min":1.3089999999999999},"min":{"1min":0.95899999999999996,"5min":0.95899999999999996,"15min":0.95899999999999996},"max":{"1min":1.968,"5min":1.968,"15min":1.968},"last":1.244}]}]},{"osd":3,"up_from":14,"seq":60129542170,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":301916,"kb_used_data":532,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94069924,"statfs":{"total":96636764160,"available":96327602176,"internally_reserved":0,"allocated":544768,"data_stored":248482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Oct 6 03:30:26 2023","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.81899999999999995,"5min":0.81899999999999995,"15min":0.81899999999999995},"last":0.76700000000000002},{"interface":"front","average":{"1min":0.754,"5min":0.754,"15min":0.754},"min":{"1min":0.505,"5min":0.505,"15min":0.505},"max":{"1min":1.0920000000000001,"5min":1.0920000000000001,"15min":1.0920000000000001},"last":0.745}]},{"osd":1,"last update":"Fri Oct 6 03:30:20 2023","interfaces":[{"interface":"back","average":{"1min":0.83199999999999996,"5min":0.83199999999999996,"15min":0.83199999999999996},"min":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":1.2210000000000001,"5min":1.2210000000000001,"15min":1.2210000000000001},"last":0.48199999999999998},{"interface":"front","average":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":1.321,"5min":1.321,"15min":1.321},"last":0.44500000000000001}]},{"osd":2,"last update":"Fri Oct 6 03:30:20 2023","interfaces":[{"interface":"back","average":{"1min":0.86799999999999999,"5min":0.86799999999999999,"15min":0.86799999999999999},"min":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.61199999999999999},"max":{"1min":1.228,"5min":1.228,"15min":1.228},"last":0.96099999999999997},{"interface":"front","average":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"min":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":1.3680000000000001,"5min":1.3680000000000001,"15min":1.3680000000000001},"last":0.65000000000000002}]},{"osd":4,"last update":"Fri Oct 6 03:30:26 2023","interfaces":[{"interface":"back","average":{"1min":1.0129999999999999,"5min":1.0129999999999999,"15min":1.0129999999999999},"min":{"1min":0.76800000000000002,"5min":0.76800000000000002,"15min":0.76800000000000002},"max":{"1min":1.6359999999999999,"5min":1.6359999999999999,"15min":1.6359999999999999},"last":0.91400000000000003},{"interface":"front","average":{"1min":0.96699999999999997,"5min":0.96699999999999997,"15min":0.96699999999999997},"min":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"max":{"1min":1.409,"5min":1.409,"15min":1.409},"last":1.034}]},{"osd":5,"last update":"Fri Oct 6 03:30:26 2023","interfaces":[{"interface":"back","average":{"1min":0.98999999999999999,"5min":0.98999999999999999,"15min":0.98999999999999999},"min":{"1min":0.746,"5min":0.746,"15min":0.746},"max":{"1min":1.387,"5min":1.387,"15min":1.387},"last":0.92800000000000005},{"interface":"front","average":{"1min":1.1359999999999999,"5min":1.1359999999999999,"15min":1.1359999999999999},"min":{"1min":0.89000000000000001,"5min":0.89000000000000001,"15min":0.89000000000000001},"max":{"1min":1.615,"5min":1.615,"15min":1.615},"last":0.85199999999999998}]},{"osd":6,"last update":"Fri Oct 6 03:30:26 2023","interfaces":[{"interface":"back","average":{"1min":1.079,"5min":1.079,"15min":1.079},"min":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"max":{"1min":1.643,"5min":1.643,"15min":1.643},"last":0.876},{"interface":"front","average":{"1min":1.105,"5min":1.105,"15min":1.105},"min":{"1min":0.74399999999999999,"5min":0.74399999999999999,"15min":0.74399999999999999},"max":{"1min":1.6819999999999999,"5min":1.6819999999999999,"15min":1.6819999999999999},"last":0.89100000000000001}]},{"osd":7,"last update":"Fri Oct 6 03:30:26 2023","interfaces":[{"interface":"back","average":{"1min":1.1850000000000001,"5min":1.1850000000000001,"15min":1.1850000000000001},"min":{"1min":0.92000000000000004,"5min":0.92000000000000004,"15min":0.92000000000000004},"max":{"1min":1.597,"5min":1.597,"15min":1.597},"last":1.012},{"interface":"front","average":{"1min":1.111,"5min":1.111,"15min":1.111},"min":{"1min":0.86599999999999999,"5min":0.86599999999999999,"15min":0.86599999999999999},"max":{"1min":1.446,"5min":1.446,"15min":1.446},"last":0.83199999999999996}]}]},{"osd":2,"up_from":14,"seq":60129542169,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":301916,"kb_used_data":532,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94069924,"statfs":{"total":96636764160,"available":96327602176,"internally_reserved":0,"allocated":544768,"data_stored":248482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Oct 6 03:30:21 2023","interfaces":[{"interface":"back","average":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":1.1240000000000001,"5min":1.1240000000000001,"15min":1.1240000000000001},"last":0.82299999999999995},{"interface":"front","average":{"1min":0.70399999999999996,"5min":0.70399999999999996,"15min":0.70399999999999996},"min":{"1min":0.495,"5min":0.495,"15min":0.495},"max":{"1min":1.1559999999999999,"5min":1.1559999999999999,"15min":1.1559999999999999},"last":0.64200000000000002}]},{"osd":1,"last update":"Fri Oct 6 03:30:21 2023","interfaces":[{"interface":"back","average":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"min":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":1.1060000000000001,"5min":1.1060000000000001,"15min":1.1060000000000001},"last":0.95799999999999996},{"interface":"front","average":{"1min":0.73499999999999999,"5min":0.73499999999999999,"15min":0.73499999999999999},"min":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":1.2390000000000001,"5min":1.2390000000000001,"15min":1.2390000000000001},"last":0.70499999999999996}]},{"osd":3,"last update":"Fri Oct 6 03:30:21 2023","interfaces":[{"interface":"back","average":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":0.78500000000000003},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":1.165,"5min":1.165,"15min":1.165},"last":1.0309999999999999},{"interface":"front","average":{"1min":0.873,"5min":0.873,"15min":0.873},"min":{"1min":0.626,"5min":0.626,"15min":0.626},"max":{"1min":1.2949999999999999,"5min":1.2949999999999999,"15min":1.2949999999999999},"last":0.77900000000000003}]},{"osd":4,"last update":"Fri Oct 6 03:30:23 2023","interfaces":[{"interface":"back","average":{"1min":1.095,"5min":1.095,"15min":1.095},"min":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"max":{"1min":2.3450000000000002,"5min":2.3450000000000002,"15min":2.3450000000000002},"last":1.0880000000000001},{"interface":"front","average":{"1min":0.95599999999999996,"5min":0.95599999999999996,"15min":0.95599999999999996},"min":{"1min":0.74399999999999999,"5min":0.74399999999999999,"15min":0.74399999999999999},"max":{"1min":1.1950000000000001,"5min":1.1950000000000001,"15min":1.1950000000000001},"last":0.91000000000000003}]},{"osd":5,"last update":"Fri Oct 6 03:30:23 2023","interfaces":[{"interface":"back","average":{"1min":1.139,"5min":1.139,"15min":1.139},"min":{"1min":0.79900000000000004,"5min":0.79900000000000004,"15min":0.79900000000000004},"max":{"1min":2.5169999999999999,"5min":2.5169999999999999,"15min":2.5169999999999999},"last":1.1799999999999999},{"interface":"front","average":{"1min":1.107,"5min":1.107,"15min":1.107},"min":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":0.81799999999999995},"max":{"1min":1.7250000000000001,"5min":1.7250000000000001,"15min":1.7250000000000001},"last":0.88700000000000001}]},{"osd":6,"last update":"Fri Oct 6 03:30:23 2023","interfaces":[{"interface":"back","average":{"1min":1.014,"5min":1.014,"15min":1.014},"min":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"max":{"1min":1.218,"5min":1.218,"15min":1.218},"last":0.97799999999999998},{"interface":"front","average":{"1min":1.1899999999999999,"5min":1.1899999999999999,"15min":1.1899999999999999},"min":{"1min":0.85899999999999999,"5min":0.85899999999999999,"15min":0.85899999999999999},"max":{"1min":2.2450000000000001,"5min":2.2450000000000001,"15min":2.2450000000000001},"last":1.4390000000000001}]},{"osd":7,"last update":"Fri Oct 6 03:30:21 2023","interfaces":[{"interface":"back","average":{"1min":1.1639999999999999,"5min":1.1639999999999999,"15min":1.1639999999999999},"min":{"1min":0.81599999999999995,"5min":0.81599999999999995,"15min":0.81599999999999995},"max":{"1min":1.8140000000000001,"5min":1.8140000000000001,"15min":1.8140000000000001},"last":1.0029999999999999},{"interface":"front","average":{"1min":1.0860000000000001,"5min":1.0860000000000001,"15min":1.0860000000000001},"min":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.79500000000000004},"max":{"1min":1.2749999999999999,"5min":1.2749999999999999,"15min":1.2749999999999999},"last":1.05}]}]},{"osd":1,"up_from":14,"seq":60129542169,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":301916,"kb_used_data":532,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94069924,"statfs":{"total":96636764160,"available":96327602176,"internally_reserved":0,"allocated":544768,"data_stored":248482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Oct 6 03:30:26 2023","interfaces":[{"interface":"back","average":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.60799999999999998},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.65900000000000003},{"interface":"front","average":{"1min":0.71599999999999997,"5min":0.71599999999999997,"15min":0.71599999999999997},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":1.5329999999999999,"5min":1.5329999999999999,"15min":1.5329999999999999},"last":0.71899999999999997}]},{"osd":2,"last update":"Fri Oct 6 03:30:20 2023","interfaces":[{"interface":"back","average":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.93899999999999995,"5min":0.93899999999999995,"15min":0.93899999999999995},"last":0.83099999999999996},{"interface":"front","average":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"min":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"max":{"1min":1.278,"5min":1.278,"15min":1.278},"last":0.68500000000000005}]},{"osd":3,"last update":"Fri Oct 6 03:30:20 2023","interfaces":[{"interface":"back","average":{"1min":0.72899999999999998,"5min":0.72899999999999998,"15min":0.72899999999999998},"min":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"max":{"1min":0.90500000000000003,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.94499999999999995},{"interface":"front","average":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.3700000000000001,"5min":1.3700000000000001,"15min":1.3700000000000001},"last":0.98599999999999999}]},{"osd":4,"last update":"Fri Oct 6 03:30:26 2023","interfaces":[{"interface":"back","average":{"1min":0.94299999999999995,"5min":0.94299999999999995,"15min":0.94299999999999995},"min":{"1min":0.66400000000000003,"5min":0.66400000000000003,"15min":0.66400000000000003},"max":{"1min":1.7549999999999999,"5min":1.7549999999999999,"15min":1.7549999999999999},"last":1.2989999999999999},{"interface":"front","average":{"1min":0.97499999999999998,"5min":0.97499999999999998,"15min":0.97499999999999998},"min":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":1.415,"5min":1.415,"15min":1.415},"last":1.222}]},{"osd":5,"last update":"Fri Oct 6 03:30:26 2023","interfaces":[{"interface":"back","average":{"1min":0.93999999999999995,"5min":0.93999999999999995,"15min":0.93999999999999995},"min":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"max":{"1min":1.4910000000000001,"5min":1.4910000000000001,"15min":1.4910000000000001},"last":0.90600000000000003},{"interface":"front","average":{"1min":1.0049999999999999,"5min":1.0049999999999999,"15min":1.0049999999999999},"min":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":0.70499999999999996},"max":{"1min":1.4370000000000001,"5min":1.4370000000000001,"15min":1.4370000000000001},"last":1.242}]},{"osd":6,"last update":"Fri Oct 6 03:30:26 2023","interfaces":[{"interface":"back","average":{"1min":1.022,"5min":1.022,"15min":1.022},"min":{"1min":0.69199999999999995,"5min":0.69199999999999995,"15min":0.69199999999999995},"max":{"1min":1.603,"5min":1.603,"15min":1.603},"last":1.5309999999999999},{"interface":"front","average":{"1min":1.1080000000000001,"5min":1.1080000000000001,"15min":1.1080000000000001},"min":{"1min":0.69199999999999995,"5min":0.69199999999999995,"15min":0.69199999999999995},"max":{"1min":1.5069999999999999,"5min":1.5069999999999999,"15min":1.5069999999999999},"last":1.581}]},{"osd":7,"last update":"Fri Oct 6 03:30:26 2023","interfaces":[{"interface":"back","average":{"1min":1.04,"5min":1.04,"15min":1.04},"min":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.60799999999999998},"max":{"1min":1.6160000000000001,"5min":1.6160000000000001,"15min":1.6160000000000001},"last":1.0960000000000001},{"interface":"front","average":{"1min":1.071,"5min":1.071,"15min":1.071},"min":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.67500000000000004},"max":{"1min":1.5700000000000001,"5min":1.5700000000000001,"15min":1.5700000000000001},"last":1.131}]}]},{"osd":0,"up_from":15,"seq":64424509467,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302496,"kb_used_data":1112,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94069344,"statfs":{"total":96636764160,"available":96327008256,"internally_reserved":0,"allocated":1138688,"data_stored":838850,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Oct 6 03:30:22 2023","interfaces":[{"interface":"back","average":{"1min":0.629,"5min":0.629,"15min":0.629},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":1.0029999999999999,"5min":1.0029999999999999,"15min":1.0029999999999999},"last":0.58699999999999997},{"interface":"front","average":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.74199999999999999},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":1.179,"5min":1.179,"15min":1.179},"last":0.70199999999999996}]},{"osd":2,"last update":"Fri Oct 6 03:30:22 2023","interfaces":[{"interface":"back","average":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":1.1879999999999999,"5min":1.1879999999999999,"15min":1.1879999999999999},"last":0.81699999999999995},{"interface":"front","average":{"1min":0.72499999999999998,"5min":0.72499999999999998,"15min":0.72499999999999998},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":1.3879999999999999,"5min":1.3879999999999999,"15min":1.3879999999999999},"last":0.56599999999999995}]},{"osd":3,"last update":"Fri Oct 6 03:30:22 2023","interfaces":[{"interface":"back","average":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":1.1930000000000001,"5min":1.1930000000000001,"15min":1.1930000000000001},"last":0.65000000000000002},{"interface":"front","average":{"1min":0.86399999999999999,"5min":0.86399999999999999,"15min":0.86399999999999999},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":1.1970000000000001,"5min":1.1970000000000001,"15min":1.1970000000000001},"last":0.78200000000000003}]},{"osd":4,"last update":"Fri Oct 6 03:30:22 2023","interfaces":[{"interface":"back","average":{"1min":1.0189999999999999,"5min":1.0189999999999999,"15min":1.0189999999999999},"min":{"1min":0.72899999999999998,"5min":0.72899999999999998,"15min":0.72899999999999998},"max":{"1min":1.55,"5min":1.55,"15min":1.55},"last":0.86299999999999999},{"interface":"front","average":{"1min":0.99099999999999999,"5min":0.99099999999999999,"15min":0.99099999999999999},"min":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"max":{"1min":1.5269999999999999,"5min":1.5269999999999999,"15min":1.5269999999999999},"last":0.79800000000000004}]},{"osd":5,"last update":"Fri Oct 6 03:30:22 2023","interfaces":[{"interface":"back","average":{"1min":0.98499999999999999,"5min":0.98499999999999999,"15min":0.98499999999999999},"min":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"max":{"1min":1.5409999999999999,"5min":1.5409999999999999,"15min":1.5409999999999999},"last":0.82999999999999996},{"interface":"front","average":{"1min":1.087,"5min":1.087,"15min":1.087},"min":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"max":{"1min":1.353,"5min":1.353,"15min":1.353},"last":1.0580000000000001}]},{"osd":6,"last update":"Fri Oct 6 03:30:22 2023","interfaces":[{"interface":"back","average":{"1min":1.145,"5min":1.145,"15min":1.145},"min":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"max":{"1min":1.502,"5min":1.502,"15min":1.502},"last":1.026},{"interface":"front","average":{"1min":1.1299999999999999,"5min":1.1299999999999999,"15min":1.1299999999999999},"min":{"1min":0.64900000000000002,"5min":0.64900000000000002,"15min":0.64900000000000002},"max":{"1min":1.6439999999999999,"5min":1.6439999999999999,"15min":1.6439999999999999},"last":1.1599999999999999}]},{"osd":7,"last update":"Fri Oct 6 03:30:22 2023","interfaces":[{"interface":"back","average":{"1min":1.099,"5min":1.099,"15min":1.099},"min":{"1min":0.874,"5min":0.874,"15min":0.874},"max":{"1min":1.369,"5min":1.369,"15min":1.369},"last":1.0740000000000001},{"interface":"front","average":{"1min":1.2030000000000001,"5min":1.2030000000000001,"15min":1.2030000000000001},"min":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"max":{"1min":1.573,"5min":1.573,"15min":1.573},"last":1.131}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-06T03:31:14.436 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2023-10-06T03:29:20.957955+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=6, tm_hour=3, tm_min=29, tm_sec=20, tm_wday=4, tm_yday=279, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=6, tm_hour=3, tm_min=31, tm_sec=9, tm_wday=4, tm_yday=279, tm_isdst=0) 2023-10-06T03:31:14.437 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2023-10-06T03:29:28.357506+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=6, tm_hour=3, tm_min=29, tm_sec=28, tm_wday=4, tm_yday=279, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=6, tm_hour=3, tm_min=31, tm_sec=9, tm_wday=4, tm_yday=279, tm_isdst=0) 2023-10-06T03:31:14.437 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2023-10-06T03:29:28.357506+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=6, tm_hour=3, tm_min=29, tm_sec=28, tm_wday=4, tm_yday=279, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=6, tm_hour=3, tm_min=31, tm_sec=9, tm_wday=4, tm_yday=279, tm_isdst=0) 2023-10-06T03:31:14.437 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2023-10-06T03:29:28.357506+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=6, tm_hour=3, tm_min=29, tm_sec=28, tm_wday=4, tm_yday=279, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=6, tm_hour=3, tm_min=31, tm_sec=9, tm_wday=4, tm_yday=279, tm_isdst=0) 2023-10-06T03:31:14.437 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2023-10-06T03:29:28.357506+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=6, tm_hour=3, tm_min=29, tm_sec=28, tm_wday=4, tm_yday=279, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=6, tm_hour=3, tm_min=31, tm_sec=9, tm_wday=4, tm_yday=279, tm_isdst=0) 2023-10-06T03:31:14.437 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-10-06T03:29:28.357506+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=6, tm_hour=3, tm_min=29, tm_sec=28, tm_wday=4, tm_yday=279, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=6, tm_hour=3, tm_min=31, tm_sec=9, tm_wday=4, tm_yday=279, tm_isdst=0) 2023-10-06T03:31:14.437 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2023-10-06T03:29:28.357506+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=6, tm_hour=3, tm_min=29, tm_sec=28, tm_wday=4, tm_yday=279, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=6, tm_hour=3, tm_min=31, tm_sec=9, tm_wday=4, tm_yday=279, tm_isdst=0) 2023-10-06T03:31:14.437 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2023-10-06T03:29:28.357506+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=6, tm_hour=3, tm_min=29, tm_sec=28, tm_wday=4, tm_yday=279, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=6, tm_hour=3, tm_min=31, tm_sec=9, tm_wday=4, tm_yday=279, tm_isdst=0) 2023-10-06T03:31:14.438 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-06T03:31:34.439 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-06T03:31:34.557 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:34.551+0000 7f1690907700 1 Processor -- start 2023-10-06T03:31:34.557 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:34.551+0000 7f1690907700 1 -- start start 2023-10-06T03:31:34.558 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:34.551+0000 7f1690907700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f168c15cf70 0x7f168c06bbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:31:34.558 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:34.551+0000 7f1690907700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f168c0c9e00 0x7f168c0ca240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:31:34.558 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:34.551+0000 7f1690907700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f168c06b750 con 0x7f168c15d340 2023-10-06T03:31:34.558 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:34.551+0000 7f1690907700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f168c0ca780 con 0x7f168c0c9e00 2023-10-06T03:31:34.558 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:34.551+0000 7f1690907700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f168c0cc740 con 0x7f168c15cf70 2023-10-06T03:31:34.558 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:34.551+0000 7f168b59e700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f168c0c9e00 0x7f168c0ca240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:31:34.558 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:34.551+0000 7f168b59e700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f168c0c9e00 0x7f168c0ca240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:43874/0 (socket says 172.21.15.110:43874) 2023-10-06T03:31:34.559 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:34.551+0000 7f168b59e700 1 -- 172.21.15.110:0/1348589783 learned_addr learned my addr 172.21.15.110:0/1348589783 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:31:34.559 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:34.551+0000 7f168ad9d700 1 --2- 172.21.15.110:0/1348589783 >> v2:172.21.15.192:3300/0 conn(0x7f168c15cf70 0x7f168c06bbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:31:34.560 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:34.551+0000 7f168b59e700 1 -- 172.21.15.110:0/1348589783 >> v2:172.21.15.192:3300/0 conn(0x7f168c15cf70 msgr2=0x7f168c06bbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:31:34.560 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:34.551+0000 7f168b59e700 1 --2- 172.21.15.110:0/1348589783 >> v2:172.21.15.192:3300/0 conn(0x7f168c15cf70 0x7f168c06bbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:34.560 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:34.551+0000 7f168b59e700 1 -- 172.21.15.110:0/1348589783 >> v1:172.21.15.192:6789/0 conn(0x7f168c15d340 legacy=0x7f168c0cc030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:31:34.560 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:34.551+0000 7f168b59e700 1 -- 172.21.15.110:0/1348589783 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f168c06ab40 con 0x7f168c0c9e00 2023-10-06T03:31:34.560 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:34.551+0000 7f168b59e700 1 --2- 172.21.15.110:0/1348589783 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f168c0c9e00 0x7f168c0ca240 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f168c069730 tx=0x7f1684009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a4259701af26207f server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:31:34.560 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:34.551+0000 7f1689d9b700 1 -- 172.21.15.110:0/1348589783 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f168400b860 con 0x7f168c0c9e00 2023-10-06T03:31:34.561 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:34.551+0000 7f1689d9b700 1 -- 172.21.15.110:0/1348589783 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f168400b9c0 con 0x7f168c0c9e00 2023-10-06T03:31:34.561 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:34.551+0000 7f1689d9b700 1 -- 172.21.15.110:0/1348589783 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f168400bd60 con 0x7f168c0c9e00 2023-10-06T03:31:34.561 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:34.555+0000 7f1690907700 1 -- 172.21.15.110:0/1348589783 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f168c0c9e00 msgr2=0x7f168c0ca240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:31:34.561 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:34.555+0000 7f1690907700 1 --2- 172.21.15.110:0/1348589783 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f168c0c9e00 0x7f168c0ca240 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f168c069730 tx=0x7f1684009250 comp rx=0 tx=0).stop 2023-10-06T03:31:34.561 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:34.555+0000 7f1690907700 1 -- 172.21.15.110:0/1348589783 shutdown_connections 2023-10-06T03:31:34.561 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:34.555+0000 7f1690907700 1 --2- 172.21.15.110:0/1348589783 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f168c0c9e00 0x7f168c0ca240 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:34.561 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:34.555+0000 7f1690907700 1 --2- 172.21.15.110:0/1348589783 >> v2:172.21.15.192:3300/0 conn(0x7f168c15cf70 0x7f168c06bbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:34.561 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:34.555+0000 7f1690907700 1 -- 172.21.15.110:0/1348589783 >> 172.21.15.110:0/1348589783 conn(0x7f168c06c760 msgr2=0x7f168c06cb60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:31:34.561 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:34.555+0000 7f1690907700 1 -- 172.21.15.110:0/1348589783 shutdown_connections 2023-10-06T03:31:34.561 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:34.555+0000 7f1690907700 1 -- 172.21.15.110:0/1348589783 wait complete. 2023-10-06T03:31:34.563 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:34.555+0000 7f1690907700 1 Processor -- start 2023-10-06T03:31:34.563 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:34.555+0000 7f1690907700 1 -- start start 2023-10-06T03:31:34.564 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:34.555+0000 7f1690907700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f168c0c9e00 0x7f168c155350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:31:34.564 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:34.555+0000 7f1690907700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f168c15cf70 0x7f168c15fb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:31:34.564 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:34.555+0000 7f1690907700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f168c15d340 0x7f168c1600d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:31:34.564 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:34.555+0000 7f1690907700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f168c0c3a30 con 0x7f168c15cf70 2023-10-06T03:31:34.564 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:34.555+0000 7f1690907700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f168c160610 con 0x7f168c0c9e00 2023-10-06T03:31:34.564 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:34.555+0000 7f1690907700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f168c15ab60 con 0x7f168c15d340 2023-10-06T03:31:34.564 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:34.555+0000 7f168a59c700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f168c15cf70 0x7f168c15fb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:31:34.564 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:34.555+0000 7f168a59c700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f168c15cf70 0x7f168c15fb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:43876/0 (socket says 172.21.15.110:43876) 2023-10-06T03:31:34.564 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:34.555+0000 7f168a59c700 1 -- 172.21.15.110:0/2632447696 learned_addr learned my addr 172.21.15.110:0/2632447696 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:31:34.564 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:34.555+0000 7f168ad9d700 1 --2- 172.21.15.110:0/2632447696 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f168c0c9e00 0x7f168c155350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:31:34.564 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:34.555+0000 7f168b59e700 1 --2- 172.21.15.110:0/2632447696 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f168c15d340 0x7f168c1600d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:31:34.565 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:34.555+0000 7f168a59c700 1 -- 172.21.15.110:0/2632447696 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f168c15d340 msgr2=0x7f168c1600d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:31:34.565 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:34.555+0000 7f168a59c700 1 --2- 172.21.15.110:0/2632447696 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f168c15d340 0x7f168c1600d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:34.565 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:34.555+0000 7f168a59c700 1 -- 172.21.15.110:0/2632447696 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f168c0c9e00 msgr2=0x7f168c155350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:31:34.565 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:34.555+0000 7f168a59c700 1 --2- 172.21.15.110:0/2632447696 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f168c0c9e00 0x7f168c155350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:34.565 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:34.555+0000 7f168a59c700 1 -- 172.21.15.110:0/2632447696 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16840094f0 con 0x7f168c15cf70 2023-10-06T03:31:34.565 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:34.555+0000 7f168a59c700 1 --2- 172.21.15.110:0/2632447696 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f168c15cf70 0x7f168c15fb90 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f167400c8e0 tx=0x7f1674001710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:31:34.565 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:34.555+0000 7f1682ffd700 1 -- 172.21.15.110:0/2632447696 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1674012630 con 0x7f168c15cf70 2023-10-06T03:31:34.565 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:34.555+0000 7f1682ffd700 1 -- 172.21.15.110:0/2632447696 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16740127b0 con 0x7f168c15cf70 2023-10-06T03:31:34.565 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:34.555+0000 7f1690907700 1 -- 172.21.15.110:0/2632447696 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f168c15ae40 con 0x7f168c15cf70 2023-10-06T03:31:34.565 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:34.555+0000 7f1682ffd700 1 -- 172.21.15.110:0/2632447696 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1674012b70 con 0x7f168c15cf70 2023-10-06T03:31:34.565 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:34.559+0000 7f1690907700 1 -- 172.21.15.110:0/2632447696 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f168c15b400 con 0x7f168c15cf70 2023-10-06T03:31:34.566 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:34.559+0000 7f1690907700 1 -- 172.21.15.110:0/2632447696 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f168c0c3ba0 con 0x7f168c15cf70 2023-10-06T03:31:34.566 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:34.559+0000 7f1682ffd700 1 -- 172.21.15.110:0/2632447696 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 13) v1 ==== 80061+0+0 (secure 0 0 0) 0x7f1674003030 con 0x7f168c15cf70 2023-10-06T03:31:34.566 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:34.559+0000 7f1682ffd700 1 --2- 172.21.15.110:0/2632447696 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f166c060670 0x7f166c062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:31:34.567 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:34.559+0000 7f1682ffd700 1 -- 172.21.15.110:0/2632447696 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 6171+0+0 (secure 0 0 0) 0x7f167407c9e0 con 0x7f168c15cf70 2023-10-06T03:31:34.567 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:34.559+0000 7f168ad9d700 1 --2- 172.21.15.110:0/2632447696 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f166c060670 0x7f166c062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:31:34.567 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:34.563+0000 7f168ad9d700 1 --2- 172.21.15.110:0/2632447696 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f166c060670 0x7f166c062b20 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f167c006fd0 tx=0x7f167c008040 comp rx=0 tx=0).ready entity=mgr.5505 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:31:34.573 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:34.567+0000 7f1682ffd700 1 -- 172.21.15.110:0/2632447696 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f1674050780 con 0x7f168c15cf70 2023-10-06T03:31:34.730 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:34.723+0000 7f1690907700 1 -- 172.21.15.110:0/2632447696 --> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f168c064770 con 0x7f166c060670 2023-10-06T03:31:34.737 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:34.731+0000 7f1682ffd700 1 -- 172.21.15.110:0/2632447696 <== mgr.5505 v2:172.21.15.110:6832/33268 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+71568 (secure 0 0 0) 0x7f168c064770 con 0x7f166c060670 2023-10-06T03:31:34.738 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-10-06T03:31:34.740 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:34.735+0000 7f1690907700 1 -- 172.21.15.110:0/2632447696 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f166c060670 msgr2=0x7f166c062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:31:34.740 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:34.735+0000 7f1690907700 1 --2- 172.21.15.110:0/2632447696 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f166c060670 0x7f166c062b20 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f167c006fd0 tx=0x7f167c008040 comp rx=0 tx=0).stop 2023-10-06T03:31:34.740 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:34.735+0000 7f1690907700 1 -- 172.21.15.110:0/2632447696 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f168c15cf70 msgr2=0x7f168c15fb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:31:34.740 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:34.735+0000 7f1690907700 1 --2- 172.21.15.110:0/2632447696 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f168c15cf70 0x7f168c15fb90 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f167400c8e0 tx=0x7f1674001710 comp rx=0 tx=0).stop 2023-10-06T03:31:34.740 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:34.735+0000 7f1690907700 1 -- 172.21.15.110:0/2632447696 shutdown_connections 2023-10-06T03:31:34.740 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:34.735+0000 7f1690907700 1 --2- 172.21.15.110:0/2632447696 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f166c060670 0x7f166c062b20 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:34.741 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:34.735+0000 7f1690907700 1 --2- 172.21.15.110:0/2632447696 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f168c15d340 0x7f168c1600d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:34.741 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:34.735+0000 7f1690907700 1 --2- 172.21.15.110:0/2632447696 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f168c15cf70 0x7f168c15fb90 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:34.741 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:34.735+0000 7f1690907700 1 --2- 172.21.15.110:0/2632447696 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f168c0c9e00 0x7f168c155350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:34.741 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:34.735+0000 7f1690907700 1 -- 172.21.15.110:0/2632447696 >> 172.21.15.110:0/2632447696 conn(0x7f168c06c760 msgr2=0x7f168c0c87d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:31:34.741 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:34.735+0000 7f1690907700 1 -- 172.21.15.110:0/2632447696 shutdown_connections 2023-10-06T03:31:34.741 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:34.735+0000 7f1690907700 1 -- 172.21.15.110:0/2632447696 wait complete. 2023-10-06T03:31:34.741 INFO:teuthology.orchestra.run.smithi110.stderr:dumped all 2023-10-06T03:31:34.752 INFO:teuthology.orchestra.run.smithi110.stdout:{"pg_ready":true,"pg_map":{"version":22,"stamp":"2023-10-06T03:31:33.954118+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":308,"num_read_kb":512,"num_write":199,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":140,"ondisk_log_size":140,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":7,"kb":754974720,"kb_used":2400104,"kb_used_data":5416,"kb_used_omap":0,"kb_used_meta":2394624,"kb_avail":752574616,"statfs":{"total":773094113280,"available":770636406784,"internally_reserved":0,"allocated":5545984,"data_stored":3168592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2452094976},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003450"},"pg_stats":[{"pgid":"1.0","version":"47'140","reported_seq":276,"reported_epoch":47,"state":"active+clean","last_fresh":"2023-10-06T03:31:14.294834+0000","last_change":"2023-10-06T03:31:14.294750+0000","last_active":"2023-10-06T03:31:14.294834+0000","last_peered":"2023-10-06T03:31:14.294834+0000","last_clean":"2023-10-06T03:31:14.294834+0000","last_became_active":"2023-10-06T03:29:21.978017+0000","last_became_peered":"2023-10-06T03:29:21.978017+0000","last_unstale":"2023-10-06T03:31:14.294834+0000","last_undegraded":"2023-10-06T03:31:14.294834+0000","last_fullsized":"2023-10-06T03:31:14.294834+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"47'140","last_scrub_stamp":"2023-10-06T03:31:14.294533+0000","last_deep_scrub":"47'140","last_deep_scrub_stamp":"2023-10-06T03:31:14.294533+0000","last_clean_scrub_stamp":"2023-10-06T03:31:14.294533+0000","objects_scrubbed":2,"log_size":140,"log_dups_size":0,"ondisk_log_size":140,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T03:51:00.602804+0000","scrub_duration":0.0086854129999999995,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":308,"num_read_kb":512,"num_write":199,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":44,"reported_epoch":47,"state":"active+clean","last_fresh":"2023-10-06T03:31:13.586115+0000","last_change":"2023-10-06T03:31:13.586049+0000","last_active":"2023-10-06T03:31:13.586115+0000","last_peered":"2023-10-06T03:31:13.586115+0000","last_clean":"2023-10-06T03:31:13.586115+0000","last_became_active":"2023-10-06T03:29:29.385692+0000","last_became_peered":"2023-10-06T03:29:29.385692+0000","last_unstale":"2023-10-06T03:31:13.586115+0000","last_undegraded":"2023-10-06T03:31:13.586115+0000","last_fullsized":"2023-10-06T03:31:13.586115+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:31:13.585811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:31:13.585811+0000","last_clean_scrub_stamp":"2023-10-06T03:31:13.585811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T09:12:07.349572+0000","scrub_duration":0.0042278669999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":44,"reported_epoch":47,"state":"active+clean","last_fresh":"2023-10-06T03:31:16.274436+0000","last_change":"2023-10-06T03:31:16.274370+0000","last_active":"2023-10-06T03:31:16.274436+0000","last_peered":"2023-10-06T03:31:16.274436+0000","last_clean":"2023-10-06T03:31:16.274436+0000","last_became_active":"2023-10-06T03:29:29.383884+0000","last_became_peered":"2023-10-06T03:29:29.383884+0000","last_unstale":"2023-10-06T03:31:16.274436+0000","last_undegraded":"2023-10-06T03:31:16.274436+0000","last_fullsized":"2023-10-06T03:31:16.274436+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:31:16.274141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:31:16.274141+0000","last_clean_scrub_stamp":"2023-10-06T03:31:16.274141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T03:45:35.709753+0000","scrub_duration":0.0048744829999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":45,"reported_epoch":47,"state":"active+clean","last_fresh":"2023-10-06T03:31:12.508603+0000","last_change":"2023-10-06T03:31:12.508580+0000","last_active":"2023-10-06T03:31:12.508603+0000","last_peered":"2023-10-06T03:31:12.508603+0000","last_clean":"2023-10-06T03:31:12.508603+0000","last_became_active":"2023-10-06T03:29:29.605489+0000","last_became_peered":"2023-10-06T03:29:29.605489+0000","last_unstale":"2023-10-06T03:31:12.508603+0000","last_undegraded":"2023-10-06T03:31:12.508603+0000","last_fullsized":"2023-10-06T03:31:12.508603+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:31:12.508497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:31:12.508497+0000","last_clean_scrub_stamp":"2023-10-06T03:31:12.508497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T14:49:34.037015+0000","scrub_duration":0.002597813,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":44,"reported_epoch":47,"state":"active+clean","last_fresh":"2023-10-06T03:31:14.622984+0000","last_change":"2023-10-06T03:31:14.622920+0000","last_active":"2023-10-06T03:31:14.622984+0000","last_peered":"2023-10-06T03:31:14.622984+0000","last_clean":"2023-10-06T03:31:14.622984+0000","last_became_active":"2023-10-06T03:29:29.383894+0000","last_became_peered":"2023-10-06T03:29:29.383894+0000","last_unstale":"2023-10-06T03:31:14.622984+0000","last_undegraded":"2023-10-06T03:31:14.622984+0000","last_fullsized":"2023-10-06T03:31:14.622984+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:31:14.622682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:31:14.622682+0000","last_clean_scrub_stamp":"2023-10-06T03:31:14.622682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T03:45:34.058294+0000","scrub_duration":0.0045302169999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":44,"reported_epoch":47,"state":"active+clean","last_fresh":"2023-10-06T03:31:15.272463+0000","last_change":"2023-10-06T03:31:15.272395+0000","last_active":"2023-10-06T03:31:15.272463+0000","last_peered":"2023-10-06T03:31:15.272463+0000","last_clean":"2023-10-06T03:31:15.272463+0000","last_became_active":"2023-10-06T03:29:29.381918+0000","last_became_peered":"2023-10-06T03:29:29.381918+0000","last_unstale":"2023-10-06T03:31:15.272463+0000","last_undegraded":"2023-10-06T03:31:15.272463+0000","last_fullsized":"2023-10-06T03:31:15.272463+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:31:15.272176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:31:15.272176+0000","last_clean_scrub_stamp":"2023-10-06T03:31:15.272176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T07:57:16.994115+0000","scrub_duration":0.0037153920000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":44,"reported_epoch":47,"state":"active+clean","last_fresh":"2023-10-06T03:31:15.257774+0000","last_change":"2023-10-06T03:31:15.257705+0000","last_active":"2023-10-06T03:31:15.257774+0000","last_peered":"2023-10-06T03:31:15.257774+0000","last_clean":"2023-10-06T03:31:15.257774+0000","last_became_active":"2023-10-06T03:29:29.383533+0000","last_became_peered":"2023-10-06T03:29:29.383533+0000","last_unstale":"2023-10-06T03:31:15.257774+0000","last_undegraded":"2023-10-06T03:31:15.257774+0000","last_fullsized":"2023-10-06T03:31:15.257774+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:31:15.257440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:31:15.257440+0000","last_clean_scrub_stamp":"2023-10-06T03:31:15.257440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T09:12:09.021201+0000","scrub_duration":0.0045948930000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":44,"reported_epoch":47,"state":"active+clean","last_fresh":"2023-10-06T03:31:16.301514+0000","last_change":"2023-10-06T03:31:16.301455+0000","last_active":"2023-10-06T03:31:16.301514+0000","last_peered":"2023-10-06T03:31:16.301514+0000","last_clean":"2023-10-06T03:31:16.301514+0000","last_became_active":"2023-10-06T03:29:29.385865+0000","last_became_peered":"2023-10-06T03:29:29.385865+0000","last_unstale":"2023-10-06T03:31:16.301514+0000","last_undegraded":"2023-10-06T03:31:16.301514+0000","last_fullsized":"2023-10-06T03:31:16.301514+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:31:16.301234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:31:16.301234+0000","last_clean_scrub_stamp":"2023-10-06T03:31:16.301234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T05:50:49.837252+0000","scrub_duration":0.004261188,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":43,"reported_epoch":47,"state":"active+clean","last_fresh":"2023-10-06T03:31:15.040773+0000","last_change":"2023-10-06T03:31:15.040718+0000","last_active":"2023-10-06T03:31:15.040773+0000","last_peered":"2023-10-06T03:31:15.040773+0000","last_clean":"2023-10-06T03:31:15.040773+0000","last_became_active":"2023-10-06T03:29:29.386812+0000","last_became_peered":"2023-10-06T03:29:29.386812+0000","last_unstale":"2023-10-06T03:31:15.040773+0000","last_undegraded":"2023-10-06T03:31:15.040773+0000","last_fullsized":"2023-10-06T03:31:15.040773+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T03:31:15.040485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T03:31:15.040485+0000","last_clean_scrub_stamp":"2023-10-06T03:31:15.040485+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T14:49:36.569002+0000","scrub_duration":0.0035756899999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":308,"num_read_kb":512,"num_write":199,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":140,"ondisk_log_size":140,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509470,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298400,"kb_used_data":1112,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94073440,"statfs":{"total":96636764160,"available":96331202560,"internally_reserved":0,"allocated":1138688,"data_stored":838850,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Oct 6 03:31:30 2023","interfaces":[{"interface":"back","average":{"1min":0.88600000000000001,"5min":0.879,"15min":0.878},"min":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"max":{"1min":1.2370000000000001,"5min":1.2370000000000001,"15min":1.2370000000000001},"last":1.0920000000000001},{"interface":"front","average":{"1min":0.92900000000000005,"5min":0.88800000000000001,"15min":0.88100000000000001},"min":{"1min":0.66200000000000003,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":1.3140000000000001,"5min":1.3140000000000001,"15min":1.3140000000000001},"last":1.1559999999999999}]},{"osd":1,"last update":"Fri Oct 6 03:31:30 2023","interfaces":[{"interface":"back","average":{"1min":0.91600000000000004,"5min":0.92700000000000005,"15min":0.92900000000000005},"min":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"max":{"1min":1.242,"5min":1.2889999999999999,"15min":1.2889999999999999},"last":1.242},{"interface":"front","average":{"1min":0.92200000000000004,"5min":0.93799999999999994,"15min":0.94099999999999995},"min":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"max":{"1min":1.2589999999999999,"5min":1.341,"15min":1.341},"last":0.83999999999999997}]},{"osd":2,"last update":"Fri Oct 6 03:31:30 2023","interfaces":[{"interface":"back","average":{"1min":0.89700000000000002,"5min":0.94099999999999995,"15min":0.94899999999999995},"min":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"max":{"1min":1.163,"5min":1.2050000000000001,"15min":1.2050000000000001},"last":0.86499999999999999},{"interface":"front","average":{"1min":0.98899999999999999,"5min":1.002,"15min":1.004},"min":{"1min":0.73199999999999998,"5min":0.71199999999999997,"15min":0.71199999999999997},"max":{"1min":1.232,"5min":1.409,"15min":1.409},"last":0.91300000000000003}]},{"osd":3,"last update":"Fri Oct 6 03:31:30 2023","interfaces":[{"interface":"back","average":{"1min":0.93700000000000006,"5min":0.877,"15min":0.86699999999999999},"min":{"1min":0.67500000000000004,"5min":0.56200000000000006,"15min":0.56200000000000006},"max":{"1min":1.2829999999999999,"5min":1.2829999999999999,"15min":1.2829999999999999},"last":0.751},{"interface":"front","average":{"1min":0.95799999999999996,"5min":0.96599999999999997,"15min":0.96799999999999997},"min":{"1min":0.73699999999999999,"5min":0.63400000000000001,"15min":0.63400000000000001},"max":{"1min":1.2789999999999999,"5min":1.2789999999999999,"15min":1.2789999999999999},"last":1.0449999999999999}]},{"osd":4,"last update":"Fri Oct 6 03:31:30 2023","interfaces":[{"interface":"back","average":{"1min":0.92700000000000005,"5min":0.92000000000000004,"15min":0.91900000000000004},"min":{"1min":0.747,"5min":0.58999999999999997,"15min":0.58999999999999997},"max":{"1min":1.179,"5min":1.2230000000000001,"15min":1.2230000000000001},"last":1.1279999999999999},{"interface":"front","average":{"1min":0.99399999999999999,"5min":1.0409999999999999,"15min":1.0489999999999999},"min":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"max":{"1min":1.3460000000000001,"5min":1.4750000000000001,"15min":1.4750000000000001},"last":0.79700000000000004}]},{"osd":5,"last update":"Fri Oct 6 03:31:30 2023","interfaces":[{"interface":"back","average":{"1min":1.0289999999999999,"5min":1.024,"15min":1.0229999999999999},"min":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":0.66700000000000004},"max":{"1min":1.3240000000000001,"5min":1.431,"15min":1.431},"last":0.95899999999999996},{"interface":"front","average":{"1min":1.0529999999999999,"5min":1.0940000000000001,"15min":1.101},"min":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"max":{"1min":1.2909999999999999,"5min":1.407,"15min":1.407},"last":1.2070000000000001}]},{"osd":6,"last update":"Fri Oct 6 03:31:30 2023","interfaces":[{"interface":"back","average":{"1min":1.153,"5min":1.1419999999999999,"15min":1.1399999999999999},"min":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"max":{"1min":1.3280000000000001,"5min":1.526,"15min":1.526},"last":1.3280000000000001},{"interface":"front","average":{"1min":1.2150000000000001,"5min":1.2010000000000001,"15min":1.1990000000000001},"min":{"1min":1.073,"5min":0.98399999999999999,"15min":0.98399999999999999},"max":{"1min":1.383,"5min":1.6719999999999999,"15min":1.6719999999999999},"last":1.278}]}]},{"osd":6,"up_from":16,"seq":68719476767,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297820,"kb_used_data":532,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074020,"statfs":{"total":96636764160,"available":96331796480,"internally_reserved":0,"allocated":544768,"data_stored":248482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Oct 6 03:31:27 2023","interfaces":[{"interface":"back","average":{"1min":0.94999999999999996,"5min":0.89000000000000001,"15min":0.88},"min":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"max":{"1min":1.2529999999999999,"5min":1.2529999999999999,"15min":1.2529999999999999},"last":1.3500000000000001},{"interface":"front","average":{"1min":0.84199999999999997,"5min":0.89100000000000001,"15min":0.89900000000000002},"min":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"max":{"1min":1.141,"5min":1.254,"15min":1.254},"last":0.70899999999999996}]},{"osd":1,"last update":"Fri Oct 6 03:31:27 2023","interfaces":[{"interface":"back","average":{"1min":0.84999999999999998,"5min":0.81999999999999995,"15min":0.81499999999999995},"min":{"1min":0.60299999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"max":{"1min":1.1499999999999999,"5min":1.4730000000000001,"15min":1.4730000000000001},"last":0.78900000000000003},{"interface":"front","average":{"1min":0.96099999999999997,"5min":0.96899999999999997,"15min":0.96999999999999997},"min":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"max":{"1min":1.208,"5min":1.3440000000000001,"15min":1.3440000000000001},"last":1.2869999999999999}]},{"osd":2,"last update":"Fri Oct 6 03:31:27 2023","interfaces":[{"interface":"back","average":{"1min":0.92300000000000004,"5min":0.93700000000000006,"15min":0.93899999999999995},"min":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":1.206,"5min":1.236,"15min":1.236},"last":1.167},{"interface":"front","average":{"1min":0.90900000000000003,"5min":0.94599999999999995,"15min":0.95199999999999996},"min":{"1min":0.63300000000000001,"5min":0.623,"15min":0.623},"max":{"1min":1.248,"5min":1.3660000000000001,"15min":1.3660000000000001},"last":1.0620000000000001}]},{"osd":3,"last update":"Fri Oct 6 03:31:27 2023","interfaces":[{"interface":"back","average":{"1min":0.92200000000000004,"5min":0.97799999999999998,"15min":0.98799999999999999},"min":{"1min":0.629,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":1.1839999999999999,"5min":1.3660000000000001,"15min":1.3660000000000001},"last":0.88600000000000001},{"interface":"front","average":{"1min":0.96499999999999997,"5min":0.95699999999999996,"15min":0.95599999999999996},"min":{"1min":0.753,"5min":0.753,"15min":0.753},"max":{"1min":1.119,"5min":1.2609999999999999,"15min":1.2609999999999999},"last":1.1100000000000001}]},{"osd":4,"last update":"Fri Oct 6 03:31:27 2023","interfaces":[{"interface":"back","average":{"1min":0.92900000000000005,"5min":1.069,"15min":1.0920000000000001},"min":{"1min":0.63900000000000001,"5min":0.63900000000000001,"15min":0.63900000000000001},"max":{"1min":1.3009999999999999,"5min":1.3009999999999999,"15min":1.3009999999999999},"last":0.84299999999999997},{"interface":"front","average":{"1min":1.0740000000000001,"5min":1.1060000000000001,"15min":1.111},"min":{"1min":0.68700000000000006,"5min":0.68700000000000006,"15min":0.68700000000000006},"max":{"1min":1.2470000000000001,"5min":1.4450000000000001,"15min":1.4450000000000001},"last":1.218}]},{"osd":5,"last update":"Fri Oct 6 03:31:27 2023","interfaces":[{"interface":"back","average":{"1min":0.97199999999999998,"5min":1.0780000000000001,"15min":1.0960000000000001},"min":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.70999999999999996},"max":{"1min":1.5620000000000001,"5min":1.613,"15min":1.613},"last":0.94699999999999995},{"interface":"front","average":{"1min":1.109,"5min":1.157,"15min":1.165},"min":{"1min":0.92400000000000004,"5min":0.83499999999999996,"15min":0.83499999999999996},"max":{"1min":1.395,"5min":1.395,"15min":1.395},"last":1.3999999999999999}]},{"osd":7,"last update":"Fri Oct 6 03:31:27 2023","interfaces":[{"interface":"back","average":{"1min":1.1000000000000001,"5min":1.2090000000000001,"15min":1.2270000000000001},"min":{"1min":0.76300000000000001,"5min":0.76300000000000001,"15min":0.76300000000000001},"max":{"1min":1.319,"5min":1.5409999999999999,"15min":1.5409999999999999},"last":1.3220000000000001},{"interface":"front","average":{"1min":1.139,"5min":1.1970000000000001,"15min":1.2070000000000001},"min":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":0.91800000000000004},"max":{"1min":1.6539999999999999,"5min":1.704,"15min":1.704},"last":1.25}]}]},{"osd":5,"up_from":16,"seq":68719476767,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297820,"kb_used_data":532,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074020,"statfs":{"total":96636764160,"available":96331796480,"internally_reserved":0,"allocated":544768,"data_stored":248482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Oct 6 03:31:24 2023","interfaces":[{"interface":"back","average":{"1min":0.93300000000000005,"5min":0.877,"15min":0.86799999999999999},"min":{"1min":0.64700000000000002,"5min":0.505,"15min":0.505},"max":{"1min":1.2749999999999999,"5min":1.4079999999999999,"15min":1.4079999999999999},"last":0.85999999999999999},{"interface":"front","average":{"1min":0.90500000000000003,"5min":0.89900000000000002,"15min":0.89800000000000002},"min":{"1min":0.66200000000000003,"5min":0.64500000000000002,"15min":0.64500000000000002},"max":{"1min":1.341,"5min":1.3779999999999999,"15min":1.3779999999999999},"last":0.72399999999999998}]},{"osd":1,"last update":"Fri Oct 6 03:31:24 2023","interfaces":[{"interface":"back","average":{"1min":0.91900000000000004,"5min":0.96599999999999997,"15min":0.97399999999999998},"min":{"1min":0.57099999999999995,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":1.2210000000000001,"5min":1.2829999999999999,"15min":1.2829999999999999},"last":0.81999999999999995},{"interface":"front","average":{"1min":0.88900000000000001,"5min":0.86499999999999999,"15min":0.86099999999999999},"min":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"max":{"1min":1.2669999999999999,"5min":1.581,"15min":1.581},"last":0.97799999999999998}]},{"osd":2,"last update":"Fri Oct 6 03:31:24 2023","interfaces":[{"interface":"back","average":{"1min":0.93999999999999995,"5min":0.98799999999999999,"15min":0.996},"min":{"1min":0.71499999999999997,"5min":0.67300000000000004,"15min":0.67300000000000004},"max":{"1min":1.244,"5min":1.258,"15min":1.258},"last":1.1899999999999999},{"interface":"front","average":{"1min":0.84699999999999998,"5min":0.89400000000000002,"15min":0.90200000000000002},"min":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":0.61299999999999999},"max":{"1min":1.0800000000000001,"5min":1.367,"15min":1.367},"last":0.78800000000000003}]},{"osd":3,"last update":"Fri Oct 6 03:31:24 2023","interfaces":[{"interface":"back","average":{"1min":0.877,"5min":1.002,"15min":1.0229999999999999},"min":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"max":{"1min":1.2070000000000001,"5min":1.4079999999999999,"15min":1.4079999999999999},"last":0.89100000000000001},{"interface":"front","average":{"1min":1.0620000000000001,"5min":1.0389999999999999,"15min":1.0349999999999999},"min":{"1min":0.872,"5min":0.69099999999999995,"15min":0.69099999999999995},"max":{"1min":1.3799999999999999,"5min":1.3799999999999999,"15min":1.3799999999999999},"last":1.1200000000000001}]},{"osd":4,"last update":"Fri Oct 6 03:31:24 2023","interfaces":[{"interface":"back","average":{"1min":0.94399999999999995,"5min":1.0620000000000001,"15min":1.0820000000000001},"min":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"max":{"1min":1.667,"5min":2.2050000000000001,"15min":2.2050000000000001},"last":0.63900000000000001},{"interface":"front","average":{"1min":1.1060000000000001,"5min":1.1699999999999999,"15min":1.181},"min":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"max":{"1min":1.7090000000000001,"5min":1.7090000000000001,"15min":1.7090000000000001},"last":1.024}]},{"osd":6,"last update":"Fri Oct 6 03:31:24 2023","interfaces":[{"interface":"back","average":{"1min":1.079,"5min":1.099,"15min":1.1020000000000001},"min":{"1min":0.82099999999999995,"5min":0.78200000000000003,"15min":0.78200000000000003},"max":{"1min":1.4850000000000001,"5min":1.4850000000000001,"15min":1.4850000000000001},"last":1.0649999999999999},{"interface":"front","average":{"1min":1.139,"5min":1.2170000000000001,"15min":1.23},"min":{"1min":0.872,"5min":0.872,"15min":0.872},"max":{"1min":1.514,"5min":2.3809999999999998,"15min":2.3809999999999998},"last":0.91800000000000004}]},{"osd":7,"last update":"Fri Oct 6 03:31:24 2023","interfaces":[{"interface":"back","average":{"1min":1.1970000000000001,"5min":1.2290000000000001,"15min":1.234},"min":{"1min":0.90200000000000002,"5min":0.79500000000000004,"15min":0.79500000000000004},"max":{"1min":1.6060000000000001,"5min":1.637,"15min":1.637},"last":0.95999999999999996},{"interface":"front","average":{"1min":1.2010000000000001,"5min":1.274,"15min":1.286},"min":{"1min":0.88100000000000001,"5min":0.88100000000000001,"15min":0.88100000000000001},"max":{"1min":1.635,"5min":2.3330000000000002,"15min":2.3330000000000002},"last":1.244}]}]},{"osd":4,"up_from":16,"seq":68719476766,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":297820,"kb_used_data":532,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074020,"statfs":{"total":96636764160,"available":96331796480,"internally_reserved":0,"allocated":544768,"data_stored":248482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Oct 6 03:31:28 2023","interfaces":[{"interface":"back","average":{"1min":0.77000000000000002,"5min":0.81299999999999994,"15min":0.81999999999999995},"min":{"1min":0.53700000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"max":{"1min":1.1479999999999999,"5min":1.1479999999999999,"15min":1.1479999999999999},"last":0.83699999999999997},{"interface":"front","average":{"1min":0.88600000000000001,"5min":0.94699999999999995,"15min":0.95699999999999996},"min":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":1.2,"5min":1.629,"15min":1.629},"last":0.90400000000000003}]},{"osd":1,"last update":"Fri Oct 6 03:31:28 2023","interfaces":[{"interface":"back","average":{"1min":0.88100000000000001,"5min":0.89000000000000001,"15min":0.89200000000000002},"min":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":1.1319999999999999,"5min":1.23,"15min":1.23},"last":0.87},{"interface":"front","average":{"1min":0.80000000000000004,"5min":0.85399999999999998,"15min":0.86299999999999999},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.0669999999999999,"5min":1.3680000000000001,"15min":1.3680000000000001},"last":0.93600000000000005}]},{"osd":2,"last update":"Fri Oct 6 03:31:28 2023","interfaces":[{"interface":"back","average":{"1min":0.93999999999999995,"5min":0.97999999999999998,"15min":0.98599999999999999},"min":{"1min":0.625,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":1.2,"5min":1.5549999999999999,"15min":1.5549999999999999},"last":0.80200000000000005},{"interface":"front","average":{"1min":0.89700000000000002,"5min":0.91600000000000004,"15min":0.91900000000000004},"min":{"1min":0.621,"5min":0.60799999999999998,"15min":0.60799999999999998},"max":{"1min":1.224,"5min":1.306,"15min":1.306},"last":1.2030000000000001}]},{"osd":3,"last update":"Fri Oct 6 03:31:28 2023","interfaces":[{"interface":"back","average":{"1min":0.96999999999999997,"5min":1.077,"15min":1.095},"min":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":0.72099999999999997},"max":{"1min":1.2070000000000001,"5min":1.5580000000000001,"15min":1.5580000000000001},"last":0.96899999999999997},{"interface":"front","average":{"1min":0.96599999999999997,"5min":1.0249999999999999,"15min":1.0349999999999999},"min":{"1min":0.70999999999999996,"5min":0.67500000000000004,"15min":0.67500000000000004},"max":{"1min":1.179,"5min":1.6699999999999999,"15min":1.6699999999999999},"last":1.0229999999999999}]},{"osd":5,"last update":"Fri Oct 6 03:31:28 2023","interfaces":[{"interface":"back","average":{"1min":0.88600000000000001,"5min":1.012,"15min":1.0329999999999999},"min":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"max":{"1min":1.0900000000000001,"5min":1.5169999999999999,"15min":1.5169999999999999},"last":0.61699999999999999},{"interface":"front","average":{"1min":1.016,"5min":1.216,"15min":1.2490000000000001},"min":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"max":{"1min":1.2669999999999999,"5min":2.2029999999999998,"15min":2.2029999999999998},"last":1.103}]},{"osd":6,"last update":"Fri Oct 6 03:31:28 2023","interfaces":[{"interface":"back","average":{"1min":1.0029999999999999,"5min":1.145,"15min":1.169},"min":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"max":{"1min":1.3129999999999999,"5min":2.0209999999999999,"15min":2.0209999999999999},"last":0.69999999999999996},{"interface":"front","average":{"1min":1.0469999999999999,"5min":1.1519999999999999,"15min":1.1699999999999999},"min":{"1min":0.745,"5min":0.71399999999999997,"15min":0.71399999999999997},"max":{"1min":1.2789999999999999,"5min":2.1440000000000001,"15min":2.1440000000000001},"last":1.075}]},{"osd":7,"last update":"Fri Oct 6 03:31:28 2023","interfaces":[{"interface":"back","average":{"1min":1.1399999999999999,"5min":1.2969999999999999,"15min":1.323},"min":{"1min":0.69199999999999995,"5min":0.69199999999999995,"15min":0.69199999999999995},"max":{"1min":1.48,"5min":2.0939999999999999,"15min":2.0939999999999999},"last":1.1659999999999999},{"interface":"front","average":{"1min":1.119,"5min":1.2709999999999999,"15min":1.296},"min":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"max":{"1min":1.5229999999999999,"5min":1.968,"15min":1.968},"last":1.1319999999999999}]}]},{"osd":3,"up_from":14,"seq":60129542174,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":301916,"kb_used_data":532,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94069924,"statfs":{"total":96636764160,"available":96327602176,"internally_reserved":0,"allocated":544768,"data_stored":248482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Oct 6 03:31:27 2023","interfaces":[{"interface":"back","average":{"1min":0.68899999999999995,"5min":0.63300000000000001,"15min":0.623},"min":{"1min":0.48399999999999999,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":1.071,"5min":1.071,"15min":1.071},"last":0.67300000000000004},{"interface":"front","average":{"1min":0.71099999999999997,"5min":0.745,"15min":0.751},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":1.149,"5min":1.149,"15min":1.149},"last":0.93600000000000005}]},{"osd":1,"last update":"Fri Oct 6 03:31:21 2023","interfaces":[{"interface":"back","average":{"1min":0.73699999999999999,"5min":0.81299999999999994,"15min":0.82499999999999996},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":1.1220000000000001,"5min":1.2210000000000001,"15min":1.2210000000000001},"last":1.0569999999999999},{"interface":"front","average":{"1min":0.68200000000000005,"5min":0.71199999999999997,"15min":0.71699999999999997},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":1.042,"5min":1.321,"15min":1.321},"last":0.57899999999999996}]},{"osd":2,"last update":"Fri Oct 6 03:31:21 2023","interfaces":[{"interface":"back","average":{"1min":0.76500000000000001,"5min":0.84699999999999998,"15min":0.86099999999999999},"min":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":1.085,"5min":1.228,"15min":1.228},"last":0.72999999999999998},{"interface":"front","average":{"1min":0.76400000000000001,"5min":0.79800000000000004,"15min":0.80400000000000005},"min":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":1.105,"5min":1.3680000000000001,"15min":1.3680000000000001},"last":0.83399999999999996}]},{"osd":4,"last update":"Fri Oct 6 03:31:27 2023","interfaces":[{"interface":"back","average":{"1min":0.95599999999999996,"5min":1.0009999999999999,"15min":1.0089999999999999},"min":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.74199999999999999},"max":{"1min":1.278,"5min":1.6359999999999999,"15min":1.6359999999999999},"last":1.2969999999999999},{"interface":"front","average":{"1min":0.91900000000000004,"5min":0.95699999999999996,"15min":0.96299999999999997},"min":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"max":{"1min":1.0900000000000001,"5min":1.409,"15min":1.409},"last":0.99199999999999999}]},{"osd":5,"last update":"Fri Oct 6 03:31:27 2023","interfaces":[{"interface":"back","average":{"1min":0.92500000000000004,"5min":0.97699999999999998,"15min":0.98499999999999999},"min":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"max":{"1min":1.1919999999999999,"5min":1.387,"15min":1.387},"last":1.2010000000000001},{"interface":"front","average":{"1min":0.98999999999999999,"5min":1.1060000000000001,"15min":1.1259999999999999},"min":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"max":{"1min":1.244,"5min":1.615,"15min":1.615},"last":1.159}]},{"osd":6,"last update":"Fri Oct 6 03:31:27 2023","interfaces":[{"interface":"back","average":{"1min":0.99199999999999999,"5min":1.0609999999999999,"15min":1.073},"min":{"1min":0.66800000000000004,"5min":0.66800000000000004,"15min":0.66800000000000004},"max":{"1min":1.3720000000000001,"5min":1.643,"15min":1.643},"last":0.91900000000000004},{"interface":"front","average":{"1min":1.0049999999999999,"5min":1.085,"15min":1.0980000000000001},"min":{"1min":0.79200000000000004,"5min":0.74399999999999999,"15min":0.74399999999999999},"max":{"1min":1.256,"5min":1.6819999999999999,"15min":1.6819999999999999},"last":1.3460000000000001}]},{"osd":7,"last update":"Fri Oct 6 03:31:27 2023","interfaces":[{"interface":"back","average":{"1min":1.0329999999999999,"5min":1.1539999999999999,"15min":1.1739999999999999},"min":{"1min":0.70799999999999996,"5min":0.70799999999999996,"15min":0.70799999999999996},"max":{"1min":1.3380000000000001,"5min":1.597,"15min":1.597},"last":1.397},{"interface":"front","average":{"1min":1.0449999999999999,"5min":1.097,"15min":1.1060000000000001},"min":{"1min":0.66000000000000003,"5min":0.66000000000000003,"15min":0.66000000000000003},"max":{"1min":1.3049999999999999,"5min":1.446,"15min":1.446},"last":1.248}]}]},{"osd":2,"up_from":14,"seq":60129542173,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":301916,"kb_used_data":532,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94069924,"statfs":{"total":96636764160,"available":96327602176,"internally_reserved":0,"allocated":544768,"data_stored":248482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Oct 6 03:31:23 2023","interfaces":[{"interface":"back","average":{"1min":0.73399999999999999,"5min":0.70799999999999996,"15min":0.70399999999999996},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":0.98899999999999999,"5min":1.1240000000000001,"15min":1.1240000000000001},"last":0.622},{"interface":"front","average":{"1min":0.748,"5min":0.71199999999999997,"15min":0.70599999999999996},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":1.1319999999999999,"5min":1.1559999999999999,"15min":1.1559999999999999},"last":0.90900000000000003}]},{"osd":1,"last update":"Fri Oct 6 03:31:23 2023","interfaces":[{"interface":"back","average":{"1min":0.752,"5min":0.72799999999999998,"15min":0.72399999999999998},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":1.0449999999999999,"5min":1.1060000000000001,"15min":1.1060000000000001},"last":0.84299999999999997},{"interface":"front","average":{"1min":0.748,"5min":0.73699999999999999,"15min":0.73499999999999999},"min":{"1min":0.52700000000000002,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":0.96499999999999997,"5min":1.2390000000000001,"15min":1.2390000000000001},"last":0.68400000000000005}]},{"osd":3,"last update":"Fri Oct 6 03:31:23 2023","interfaces":[{"interface":"back","average":{"1min":0.85699999999999998,"5min":0.79900000000000004,"15min":0.78900000000000003},"min":{"1min":0.495,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":1.1579999999999999,"5min":1.165,"15min":1.165},"last":0.79600000000000004},{"interface":"front","average":{"1min":0.78100000000000003,"5min":0.85399999999999998,"15min":0.86599999999999999},"min":{"1min":0.435,"5min":0.435,"15min":0.435},"max":{"1min":1.0589999999999999,"5min":1.2949999999999999,"15min":1.2949999999999999},"last":0.98499999999999999}]},{"osd":4,"last update":"Fri Oct 6 03:31:24 2023","interfaces":[{"interface":"back","average":{"1min":1.0369999999999999,"5min":1.083,"15min":1.091},"min":{"1min":0.621,"5min":0.61399999999999999,"15min":0.61399999999999999},"max":{"1min":1.3480000000000001,"5min":2.3450000000000002,"15min":2.3450000000000002},"last":1.0209999999999999},{"interface":"front","average":{"1min":0.93100000000000005,"5min":0.95099999999999996,"15min":0.95399999999999996},"min":{"1min":0.76400000000000001,"5min":0.74399999999999999,"15min":0.74399999999999999},"max":{"1min":1.2050000000000001,"5min":1.2050000000000001,"15min":1.2050000000000001},"last":1.1080000000000001}]},{"osd":5,"last update":"Fri Oct 6 03:31:24 2023","interfaces":[{"interface":"back","average":{"1min":1.0860000000000001,"5min":1.1279999999999999,"15min":1.135},"min":{"1min":0.80700000000000005,"5min":0.79900000000000004,"15min":0.79900000000000004},"max":{"1min":1.2949999999999999,"5min":2.5169999999999999,"15min":2.5169999999999999},"last":0.94099999999999995},{"interface":"front","average":{"1min":1.0089999999999999,"5min":1.087,"15min":1.1000000000000001},"min":{"1min":0.71799999999999997,"5min":0.71799999999999997,"15min":0.71799999999999997},"max":{"1min":1.411,"5min":1.7250000000000001,"15min":1.7250000000000001},"last":1.0720000000000001}]},{"osd":6,"last update":"Fri Oct 6 03:31:24 2023","interfaces":[{"interface":"back","average":{"1min":0.94799999999999995,"5min":1,"15min":1.0089999999999999},"min":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"max":{"1min":1.139,"5min":1.218,"15min":1.218},"last":0.95499999999999996},{"interface":"front","average":{"1min":1.1759999999999999,"5min":1.1870000000000001,"15min":1.1890000000000001},"min":{"1min":0.84499999999999997,"5min":0.84499999999999997,"15min":0.84499999999999997},"max":{"1min":1.47,"5min":2.2450000000000001,"15min":2.2450000000000001},"last":1.157}]},{"osd":7,"last update":"Fri Oct 6 03:31:23 2023","interfaces":[{"interface":"back","average":{"1min":1.091,"5min":1.149,"15min":1.159},"min":{"1min":0.70299999999999996,"5min":0.70299999999999996,"15min":0.70299999999999996},"max":{"1min":1.423,"5min":1.8140000000000001,"15min":1.8140000000000001},"last":1.1910000000000001},{"interface":"front","average":{"1min":1.1100000000000001,"5min":1.0900000000000001,"15min":1.087},"min":{"1min":0.71599999999999997,"5min":0.71599999999999997,"15min":0.71599999999999997},"max":{"1min":1.54,"5min":1.54,"15min":1.54},"last":1.2130000000000001}]}]},{"osd":1,"up_from":14,"seq":60129542173,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":301916,"kb_used_data":532,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94069924,"statfs":{"total":96636764160,"available":96327602176,"internally_reserved":0,"allocated":544768,"data_stored":248482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Oct 6 03:31:28 2023","interfaces":[{"interface":"back","average":{"1min":0.64600000000000002,"5min":0.61499999999999999,"15min":0.60999999999999999},"min":{"1min":0.48599999999999999,"5min":0.44,"15min":0.44},"max":{"1min":0.97199999999999998,"5min":0.97199999999999998,"15min":0.97199999999999998},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.77200000000000002,"5min":0.72699999999999998,"15min":0.71899999999999997},"min":{"1min":0.44500000000000001,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":1.119,"5min":1.5329999999999999,"15min":1.5329999999999999},"last":0.81200000000000006}]},{"osd":2,"last update":"Fri Oct 6 03:31:21 2023","interfaces":[{"interface":"back","average":{"1min":0.71399999999999997,"5min":0.72099999999999997,"15min":0.72199999999999998},"min":{"1min":0.48799999999999999,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":1.018,"5min":1.018,"15min":1.018},"last":0.63400000000000001},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.70899999999999996,"15min":0.71699999999999997},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":1.0489999999999999,"5min":1.278,"15min":1.278},"last":0.97599999999999998}]},{"osd":3,"last update":"Fri Oct 6 03:31:21 2023","interfaces":[{"interface":"back","average":{"1min":0.68500000000000005,"5min":0.71999999999999997,"15min":0.72599999999999998},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":0.94999999999999996,"5min":0.94999999999999996,"15min":0.94999999999999996},"last":0.73899999999999999},{"interface":"front","average":{"1min":0.81899999999999995,"5min":0.79100000000000004,"15min":0.78600000000000003},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":1.1120000000000001,"5min":1.3700000000000001,"15min":1.3700000000000001},"last":1.056}]},{"osd":4,"last update":"Fri Oct 6 03:31:28 2023","interfaces":[{"interface":"back","average":{"1min":0.98699999999999999,"5min":0.95099999999999996,"15min":0.94499999999999995},"min":{"1min":0.72899999999999998,"5min":0.66400000000000003,"15min":0.66400000000000003},"max":{"1min":1.2989999999999999,"5min":1.7549999999999999,"15min":1.7549999999999999},"last":1.173},{"interface":"front","average":{"1min":1.0229999999999999,"5min":0.98399999999999999,"15min":0.97799999999999998},"min":{"1min":0.68700000000000006,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":1.2250000000000001,"5min":1.415,"15min":1.415},"last":1.2190000000000001}]},{"osd":5,"last update":"Fri Oct 6 03:31:28 2023","interfaces":[{"interface":"back","average":{"1min":1.002,"5min":0.95199999999999996,"15min":0.94399999999999995},"min":{"1min":0.81799999999999995,"5min":0.54400000000000004,"15min":0.54400000000000004},"max":{"1min":1.222,"5min":1.4910000000000001,"15min":1.4910000000000001},"last":1.119},{"interface":"front","average":{"1min":1.143,"5min":1.032,"15min":1.014},"min":{"1min":0.82099999999999995,"5min":0.70499999999999996,"15min":0.70499999999999996},"max":{"1min":1.3680000000000001,"5min":1.4370000000000001,"15min":1.4370000000000001},"last":1.2929999999999999}]},{"osd":6,"last update":"Fri Oct 6 03:31:28 2023","interfaces":[{"interface":"back","average":{"1min":1.1399999999999999,"5min":1.0449999999999999,"15min":1.0289999999999999},"min":{"1min":0.79000000000000004,"5min":0.69199999999999995,"15min":0.69199999999999995},"max":{"1min":1.5309999999999999,"5min":1.603,"15min":1.603},"last":1.331},{"interface":"front","average":{"1min":1.143,"5min":1.115,"15min":1.1100000000000001},"min":{"1min":0.81100000000000005,"5min":0.69199999999999995,"15min":0.69199999999999995},"max":{"1min":1.581,"5min":1.581,"15min":1.581},"last":1.151}]},{"osd":7,"last update":"Fri Oct 6 03:31:28 2023","interfaces":[{"interface":"back","average":{"1min":1.083,"5min":1.048,"15min":1.042},"min":{"1min":0.78700000000000003,"5min":0.60799999999999998,"15min":0.60799999999999998},"max":{"1min":1.413,"5min":1.6160000000000001,"15min":1.6160000000000001},"last":1.2470000000000001},{"interface":"front","average":{"1min":1.169,"5min":1.0900000000000001,"15min":1.077},"min":{"1min":1.0009999999999999,"5min":0.67500000000000004,"15min":0.67500000000000004},"max":{"1min":1.3640000000000001,"5min":1.5700000000000001,"15min":1.5700000000000001},"last":1.3600000000000001}]}]},{"osd":0,"up_from":15,"seq":64424509471,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302496,"kb_used_data":1112,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94069344,"statfs":{"total":96636764160,"available":96327008256,"internally_reserved":0,"allocated":1138688,"data_stored":838850,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Oct 6 03:31:23 2023","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.626,"15min":0.628},"min":{"1min":0.47399999999999998,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.86499999999999999,"5min":1.0029999999999999,"15min":1.0029999999999999},"last":0.55000000000000004},{"interface":"front","average":{"1min":0.67000000000000004,"5min":0.72699999999999998,"15min":0.73699999999999999},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":0.94099999999999995,"5min":1.179,"15min":1.179},"last":0.60699999999999998}]},{"osd":2,"last update":"Fri Oct 6 03:31:23 2023","interfaces":[{"interface":"back","average":{"1min":0.71999999999999997,"5min":0.78500000000000003,"15min":0.79600000000000004},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":1.119,"5min":1.1879999999999999,"15min":1.1879999999999999},"last":0.74299999999999999},{"interface":"front","average":{"1min":0.66700000000000004,"5min":0.71299999999999997,"15min":0.72099999999999997},"min":{"1min":0.39000000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":1.1519999999999999,"5min":1.3879999999999999,"15min":1.3879999999999999},"last":0.67700000000000005}]},{"osd":3,"last update":"Fri Oct 6 03:31:23 2023","interfaces":[{"interface":"back","average":{"1min":0.73599999999999999,"5min":0.75600000000000001,"15min":0.76000000000000001},"min":{"1min":0.53200000000000003,"5min":0.442,"15min":0.442},"max":{"1min":0.89000000000000001,"5min":1.1930000000000001,"15min":1.1930000000000001},"last":0.78000000000000003},{"interface":"front","average":{"1min":0.79000000000000004,"5min":0.84899999999999998,"15min":0.85899999999999999},"min":{"1min":0.437,"5min":0.437,"15min":0.437},"max":{"1min":1.2410000000000001,"5min":1.2410000000000001,"15min":1.2410000000000001},"last":0.83799999999999997}]},{"osd":4,"last update":"Fri Oct 6 03:31:23 2023","interfaces":[{"interface":"back","average":{"1min":0.92800000000000005,"5min":1,"15min":1.012},"min":{"1min":0.63400000000000001,"5min":0.63400000000000001,"15min":0.63400000000000001},"max":{"1min":1.177,"5min":1.55,"15min":1.55},"last":0.97199999999999998},{"interface":"front","average":{"1min":0.98499999999999999,"5min":0.98899999999999999,"15min":0.98999999999999999},"min":{"1min":0.622,"5min":0.61799999999999999,"15min":0.61799999999999999},"max":{"1min":1.325,"5min":1.5269999999999999,"15min":1.5269999999999999},"last":1.155}]},{"osd":5,"last update":"Fri Oct 6 03:31:23 2023","interfaces":[{"interface":"back","average":{"1min":0.93400000000000005,"5min":0.97399999999999998,"15min":0.98099999999999998},"min":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"max":{"1min":1.2509999999999999,"5min":1.5409999999999999,"15min":1.5409999999999999},"last":1.401},{"interface":"front","average":{"1min":1.034,"5min":1.0760000000000001,"15min":1.083},"min":{"1min":0.82099999999999995,"5min":0.63100000000000001,"15min":0.63100000000000001},"max":{"1min":1.3049999999999999,"5min":1.353,"15min":1.353},"last":1.1890000000000001}]},{"osd":6,"last update":"Fri Oct 6 03:31:23 2023","interfaces":[{"interface":"back","average":{"1min":1.0680000000000001,"5min":1.129,"15min":1.139},"min":{"1min":0.80800000000000005,"5min":0.76400000000000001,"15min":0.76400000000000001},"max":{"1min":1.323,"5min":1.502,"15min":1.502},"last":1.0549999999999999},{"interface":"front","average":{"1min":1.079,"5min":1.119,"15min":1.1259999999999999},"min":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"max":{"1min":1.377,"5min":1.6439999999999999,"15min":1.6439999999999999},"last":0.92000000000000004}]},{"osd":7,"last update":"Fri Oct 6 03:31:23 2023","interfaces":[{"interface":"back","average":{"1min":1.079,"5min":1.095,"15min":1.097},"min":{"1min":0.64800000000000002,"5min":0.64800000000000002,"15min":0.64800000000000002},"max":{"1min":1.4319999999999999,"5min":1.4319999999999999,"15min":1.4319999999999999},"last":1.2689999999999999},{"interface":"front","average":{"1min":1.1419999999999999,"5min":1.1899999999999999,"15min":1.198},"min":{"1min":0.878,"5min":0.66600000000000004,"15min":0.66600000000000004},"max":{"1min":1.3029999999999999,"5min":1.573,"15min":1.573},"last":1.127}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-06T03:31:34.755 DEBUG:teuthology.orchestra.run.smithi110:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2023-10-06T03:31:34.867 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:34.859+0000 7fc576b03700 1 Processor -- start 2023-10-06T03:31:34.867 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:34.859+0000 7fc576b03700 1 -- start start 2023-10-06T03:31:34.870 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:34.863+0000 7fc576b03700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc570155a90 0x7fc570155e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:31:34.870 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:34.863+0000 7fc576b03700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fc5700af800 0x7fc5700c7820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:31:34.870 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:34.863+0000 7fc576b03700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5700c2600 con 0x7fc5700c7d60 2023-10-06T03:31:34.870 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:34.863+0000 7fc576b03700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc5700c17e0 con 0x7fc5700af800 2023-10-06T03:31:34.870 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:34.863+0000 7fc576b03700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5700c19f0 con 0x7fc570155a90 2023-10-06T03:31:34.870 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:34.863+0000 7fc575300700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fc5700af800 0x7fc5700c7820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:31:34.870 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:34.863+0000 7fc575300700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fc5700af800 0x7fc5700c7820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:43882/0 (socket says 172.21.15.110:43882) 2023-10-06T03:31:34.871 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:34.863+0000 7fc575300700 1 -- 172.21.15.110:0/3846238953 learned_addr learned my addr 172.21.15.110:0/3846238953 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:31:34.871 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:34.863+0000 7fc575b01700 1 --2- 172.21.15.110:0/3846238953 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc570155a90 0x7fc570155e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:31:34.871 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:34.863+0000 7fc575300700 1 -- 172.21.15.110:0/3846238953 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc570155a90 msgr2=0x7fc570155e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:31:34.871 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:34.863+0000 7fc575300700 1 --2- 172.21.15.110:0/3846238953 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc570155a90 0x7fc570155e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:34.871 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:34.863+0000 7fc575300700 1 -- 172.21.15.110:0/3846238953 >> v1:172.21.15.192:6789/0 conn(0x7fc5700c7d60 legacy=0x7fc5700c2a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:31:34.871 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:34.863+0000 7fc574aff700 1 -- 172.21.15.110:0/3846238953 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2238556050 0 0) 0x7fc5700c2600 con 0x7fc5700c7d60 2023-10-06T03:31:34.871 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:34.863+0000 7fc575300700 1 -- 172.21.15.110:0/3846238953 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5700c1cd0 con 0x7fc5700af800 2023-10-06T03:31:34.871 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:34.863+0000 7fc575300700 1 --2- 172.21.15.110:0/3846238953 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc5700af800 0x7fc5700c7820 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7fc56400bc50 tx=0x7fc56400b4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=52c5c46ba0af2513 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:31:34.872 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:34.863+0000 7fc574aff700 1 -- 172.21.15.110:0/3846238953 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc56400dbf0 con 0x7fc5700af800 2023-10-06T03:31:34.872 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:34.863+0000 7fc574aff700 1 -- 172.21.15.110:0/3846238953 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc56400dd50 con 0x7fc5700af800 2023-10-06T03:31:34.872 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:34.863+0000 7fc574aff700 1 -- 172.21.15.110:0/3846238953 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5640196b0 con 0x7fc5700af800 2023-10-06T03:31:34.872 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:34.863+0000 7fc576b03700 1 -- 172.21.15.110:0/3846238953 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc5700af800 msgr2=0x7fc5700c7820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:31:34.873 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:34.863+0000 7fc576b03700 1 --2- 172.21.15.110:0/3846238953 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc5700af800 0x7fc5700c7820 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7fc56400bc50 tx=0x7fc56400b4f0 comp rx=0 tx=0).stop 2023-10-06T03:31:34.873 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:34.867+0000 7fc576b03700 1 -- 172.21.15.110:0/3846238953 shutdown_connections 2023-10-06T03:31:34.873 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:34.867+0000 7fc576b03700 1 --2- 172.21.15.110:0/3846238953 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc5700af800 0x7fc5700c7820 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:34.873 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:34.867+0000 7fc576b03700 1 --2- 172.21.15.110:0/3846238953 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc570155a90 0x7fc570155e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:34.873 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:34.867+0000 7fc576b03700 1 -- 172.21.15.110:0/3846238953 >> 172.21.15.110:0/3846238953 conn(0x7fc57006e050 msgr2=0x7fc5700cca40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:31:34.873 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:34.867+0000 7fc576b03700 1 -- 172.21.15.110:0/3846238953 shutdown_connections 2023-10-06T03:31:34.873 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:34.867+0000 7fc576b03700 1 -- 172.21.15.110:0/3846238953 wait complete. 2023-10-06T03:31:34.874 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:34.867+0000 7fc576b03700 1 Processor -- start 2023-10-06T03:31:34.874 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:34.867+0000 7fc576b03700 1 -- start start 2023-10-06T03:31:34.875 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:34.867+0000 7fc576b03700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc5700c7d60 0x7fc570151d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:31:34.876 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:34.867+0000 7fc576b03700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc570155a90 0x7fc5700c3af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:31:34.876 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:34.867+0000 7fc576b03700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc5700c5650 0x7fc5700c5a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:31:34.876 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:34.867+0000 7fc576b03700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc570159f70 con 0x7fc5700c5650 2023-10-06T03:31:34.876 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:34.867+0000 7fc576b03700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc570159c20 con 0x7fc5700c7d60 2023-10-06T03:31:34.876 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:34.867+0000 7fc576b03700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc570152830 con 0x7fc570155a90 2023-10-06T03:31:34.876 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:34.871+0000 7fc576302700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc5700c5650 0x7fc5700c5a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:31:34.876 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:34.871+0000 7fc575300700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc570155a90 0x7fc5700c3af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:31:34.876 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:34.871+0000 7fc575b01700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc5700c7d60 0x7fc570151d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:31:34.876 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:34.871+0000 7fc575300700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc570155a90 0x7fc5700c3af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3301/0 says I am v2:172.21.15.110:53774/0 (socket says 172.21.15.110:53774) 2023-10-06T03:31:34.876 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:34.871+0000 7fc575300700 1 -- 172.21.15.110:0/2739780444 learned_addr learned my addr 172.21.15.110:0/2739780444 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:31:34.877 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:34.871+0000 7fc575300700 1 -- 172.21.15.110:0/2739780444 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc5700c7d60 msgr2=0x7fc570151d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:31:34.878 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:34.871+0000 7fc575300700 1 --2- 172.21.15.110:0/2739780444 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc5700c7d60 0x7fc570151d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:34.878 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:34.871+0000 7fc575300700 1 -- 172.21.15.110:0/2739780444 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc5700c5650 msgr2=0x7fc5700c5a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:31:34.878 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:34.871+0000 7fc575300700 1 --2- 172.21.15.110:0/2739780444 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc5700c5650 0x7fc5700c5a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:34.878 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:34.871+0000 7fc575300700 1 -- 172.21.15.110:0/2739780444 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc56400b790 con 0x7fc570155a90 2023-10-06T03:31:34.878 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:34.871+0000 7fc576302700 1 --2- 172.21.15.110:0/2739780444 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc5700c5650 0x7fc5700c5a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-06T03:31:34.878 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:34.871+0000 7fc575300700 1 --2- 172.21.15.110:0/2739780444 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc570155a90 0x7fc5700c3af0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fc57006a470 tx=0x7fc56401b000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:31:34.878 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:34.871+0000 7fc56dffb700 1 -- 172.21.15.110:0/2739780444 <== mon.2 v2:172.21.15.192:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc56401ae30 con 0x7fc570155a90 2023-10-06T03:31:34.878 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:34.871+0000 7fc56dffb700 1 -- 172.21.15.110:0/2739780444 <== mon.2 v2:172.21.15.192:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc564010680 con 0x7fc570155a90 2023-10-06T03:31:34.878 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:34.871+0000 7fc576b03700 1 -- 172.21.15.110:0/2739780444 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5700c65a0 con 0x7fc570155a90 2023-10-06T03:31:34.879 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:34.871+0000 7fc56dffb700 1 -- 172.21.15.110:0/2739780444 <== mon.2 v2:172.21.15.192:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc564010a40 con 0x7fc570155a90 2023-10-06T03:31:34.879 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:34.871+0000 7fc576b03700 1 -- 172.21.15.110:0/2739780444 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5701522c0 con 0x7fc570155a90 2023-10-06T03:31:34.879 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:34.871+0000 7fc576b03700 1 -- 172.21.15.110:0/2739780444 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5700c6aa0 con 0x7fc570155a90 2023-10-06T03:31:34.880 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:34.875+0000 7fc56dffb700 1 -- 172.21.15.110:0/2739780444 <== mon.2 v2:172.21.15.192:3301/0 4 ==== mgrmap(e 13) v1 ==== 80061+0+0 (secure 0 0 0) 0x7fc5640183e0 con 0x7fc570155a90 2023-10-06T03:31:34.881 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:34.875+0000 7fc56dffb700 1 --2- 172.21.15.110:0/2739780444 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fc55c060670 0x7fc55c062b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:31:34.881 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:34.875+0000 7fc56dffb700 1 -- 172.21.15.110:0/2739780444 <== mon.2 v2:172.21.15.192:3301/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 6171+0+0 (secure 0 0 0) 0x7fc564014070 con 0x7fc570155a90 2023-10-06T03:31:34.881 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:34.875+0000 7fc575b01700 1 --2- 172.21.15.110:0/2739780444 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fc55c060670 0x7fc55c062b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:31:34.881 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:34.875+0000 7fc575b01700 1 --2- 172.21.15.110:0/2739780444 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fc55c060670 0x7fc55c062b20 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fc560009d00 tx=0x7fc560009120 comp rx=0 tx=0).ready entity=mgr.5505 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:31:34.892 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:34.887+0000 7fc56dffb700 1 -- 172.21.15.110:0/2739780444 <== mon.2 v2:172.21.15.192:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7fc56404e330 con 0x7fc570155a90 2023-10-06T03:31:35.054 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:35.047+0000 7fc576b03700 1 -- 172.21.15.110:0/2739780444 --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_command([{prefix=config set, name=mon_health_to_clog}] v 0) v1 -- 0x7fc570071020 con 0x7fc570155a90 2023-10-06T03:31:35.066 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:35.059+0000 7fc56dffb700 1 -- 172.21.15.110:0/2739780444 <== mon.2 v2:172.21.15.192:3301/0 7 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7fc56404e330 con 0x7fc570155a90 2023-10-06T03:31:35.067 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:35.063+0000 7fc56dffb700 1 -- 172.21.15.110:0/2739780444 <== mon.2 v2:172.21.15.192:3301/0 8 ==== mon_command_ack([{prefix=config set, name=mon_health_to_clog}]=0 v16)=0 v16) v1 ==== 123+0+0 (secure 0 0 0) 0x7fc564016330 con 0x7fc570155a90 2023-10-06T03:31:35.071 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:35.063+0000 7fc576b03700 1 -- 172.21.15.110:0/2739780444 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fc55c060670 msgr2=0x7fc55c062b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:31:35.071 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:35.063+0000 7fc576b03700 1 --2- 172.21.15.110:0/2739780444 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fc55c060670 0x7fc55c062b20 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fc560009d00 tx=0x7fc560009120 comp rx=0 tx=0).stop 2023-10-06T03:31:35.071 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:35.063+0000 7fc576b03700 1 -- 172.21.15.110:0/2739780444 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc570155a90 msgr2=0x7fc5700c3af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:31:35.071 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:35.063+0000 7fc576b03700 1 --2- 172.21.15.110:0/2739780444 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc570155a90 0x7fc5700c3af0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fc57006a470 tx=0x7fc56401b000 comp rx=0 tx=0).stop 2023-10-06T03:31:35.071 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:35.063+0000 7fc576b03700 1 -- 172.21.15.110:0/2739780444 shutdown_connections 2023-10-06T03:31:35.071 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:35.063+0000 7fc576b03700 1 --2- 172.21.15.110:0/2739780444 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7fc55c060670 0x7fc55c062b20 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:35.071 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:35.063+0000 7fc576b03700 1 --2- 172.21.15.110:0/2739780444 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc5700c5650 0x7fc5700c5a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:35.071 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:35.063+0000 7fc576b03700 1 --2- 172.21.15.110:0/2739780444 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7fc570155a90 0x7fc5700c3af0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:35.072 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:35.063+0000 7fc576b03700 1 --2- 172.21.15.110:0/2739780444 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc5700c7d60 0x7fc570151d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:35.072 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:35.063+0000 7fc576b03700 1 -- 172.21.15.110:0/2739780444 >> 172.21.15.110:0/2739780444 conn(0x7fc57006e050 msgr2=0x7fc5700c81e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:31:35.072 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:35.063+0000 7fc576b03700 1 -- 172.21.15.110:0/2739780444 shutdown_connections 2023-10-06T03:31:35.072 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:35.063+0000 7fc576b03700 1 -- 172.21.15.110:0/2739780444 wait complete. 2023-10-06T03:31:35.081 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2023-10-06T03:31:35.082 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-06T03:31:35.184 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:35.179+0000 7f17e157c700 1 Processor -- start 2023-10-06T03:31:35.184 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:35.179+0000 7f17e157c700 1 -- start start 2023-10-06T03:31:35.185 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:35.179+0000 7f17e157c700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f17dc0c86d0 0x7f17dc0c8aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:31:35.185 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:35.179+0000 7f17e157c700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f17dc15d0a0 0x7f17dc15d540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:31:35.185 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:35.179+0000 7f17e157c700 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17dc0ac320 con 0x7f17dc06b4f0 2023-10-06T03:31:35.185 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:35.179+0000 7f17e157c700 1 -- --> v2:172.21.15.192:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f17dc0c7000 con 0x7f17dc0c86d0 2023-10-06T03:31:35.185 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:35.179+0000 7f17e157c700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17dc0abd10 con 0x7f17dc15d0a0 2023-10-06T03:31:35.186 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:35.179+0000 7f17db7fe700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f17dc15d0a0 0x7f17dc15d540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:31:35.186 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:35.179+0000 7f17daffd700 1 --2- >> v2:172.21.15.192:3300/0 conn(0x7f17dc0c86d0 0x7f17dc0c8aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:31:35.186 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:35.179+0000 7f17da7fc700 1 --1- >> v1:172.21.15.192:6789/0 conn(0x7f17dc06b4f0 0x7f17dc0c68f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.110:43006/0 (socket says 172.21.15.110:43006) 2023-10-06T03:31:35.186 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:35.179+0000 7f17da7fc700 1 -- 172.21.15.110:0/20458722 learned_addr learned my addr 172.21.15.110:0/20458722 (peer_addr_for_me v1:172.21.15.110:0/0) 2023-10-06T03:31:35.186 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:35.179+0000 7f17db7fe700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f17dc15d0a0 0x7f17dc15d540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3301/0 says I am v2:172.21.15.110:53776/0 (socket says 172.21.15.110:53776) 2023-10-06T03:31:35.188 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:35.179+0000 7f17daffd700 1 -- 172.21.15.110:0/20458722 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f17dc15d0a0 msgr2=0x7f17dc15d540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T03:31:35.188 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:35.179+0000 7f17daffd700 1 --2- 172.21.15.110:0/20458722 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f17dc15d0a0 0x7f17dc15d540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:35.188 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:35.179+0000 7f17daffd700 1 -- 172.21.15.110:0/20458722 >> v1:172.21.15.192:6789/0 conn(0x7f17dc06b4f0 legacy=0x7f17dc0c68f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:31:35.188 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:35.179+0000 7f17daffd700 1 -- 172.21.15.110:0/20458722 --> v2:172.21.15.192:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17dc0c71f0 con 0x7f17dc0c86d0 2023-10-06T03:31:35.188 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:35.179+0000 7f17db7fe700 1 --2- 172.21.15.110:0/20458722 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f17dc15d0a0 0x7f17dc15d540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-06T03:31:35.188 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:35.179+0000 7f17daffd700 1 --2- 172.21.15.110:0/20458722 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f17dc0c86d0 0x7f17dc0c8aa0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f17d00098e0 tx=0x7f17d00091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c43d4abf96faf3a3 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:31:35.188 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:35.179+0000 7f17d9ffb700 1 -- 172.21.15.110:0/20458722 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17d000b640 con 0x7f17dc0c86d0 2023-10-06T03:31:35.188 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:35.179+0000 7f17d9ffb700 1 -- 172.21.15.110:0/20458722 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7f17d000b7a0 con 0x7f17dc0c86d0 2023-10-06T03:31:35.188 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:35.179+0000 7f17d9ffb700 1 -- 172.21.15.110:0/20458722 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17d000bb60 con 0x7f17dc0c86d0 2023-10-06T03:31:35.188 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:35.179+0000 7f17e157c700 1 -- 172.21.15.110:0/20458722 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f17dc0c86d0 msgr2=0x7f17dc0c8aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:31:35.189 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:35.179+0000 7f17e157c700 1 --2- 172.21.15.110:0/20458722 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f17dc0c86d0 0x7f17dc0c8aa0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f17d00098e0 tx=0x7f17d00091a0 comp rx=0 tx=0).stop 2023-10-06T03:31:35.189 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:35.183+0000 7f17e157c700 1 -- 172.21.15.110:0/20458722 shutdown_connections 2023-10-06T03:31:35.189 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:35.183+0000 7f17e157c700 1 --2- 172.21.15.110:0/20458722 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f17dc15d0a0 0x7f17dc15d540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:35.189 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:35.183+0000 7f17e157c700 1 --2- 172.21.15.110:0/20458722 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f17dc0c86d0 0x7f17dc0c8aa0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:35.189 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:35.183+0000 7f17e157c700 1 -- 172.21.15.110:0/20458722 >> 172.21.15.110:0/20458722 conn(0x7f17dc067040 msgr2=0x7f17dc067410 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:31:35.189 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:35.183+0000 7f17e157c700 1 -- 172.21.15.110:0/20458722 shutdown_connections 2023-10-06T03:31:35.189 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:35.183+0000 7f17e157c700 1 -- 172.21.15.110:0/20458722 wait complete. 2023-10-06T03:31:35.190 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:35.183+0000 7f17e157c700 1 Processor -- start 2023-10-06T03:31:35.190 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:35.183+0000 7f17e157c700 1 -- start start 2023-10-06T03:31:35.193 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:35.187+0000 7f17e157c700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f17dc06b4f0 0x7f17dc0ba670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:31:35.194 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:35.187+0000 7f17e157c700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f17dc0c86d0 0x7f17dc0c9f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:31:35.194 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:35.187+0000 7f17e157c700 1 --2- >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f17dc15d0a0 0x7f17dc14cea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:31:35.194 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:35.187+0000 7f17e157c700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17dc0caf80 con 0x7f17dc0c86d0 2023-10-06T03:31:35.194 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:35.187+0000 7f17e157c700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17dc0cb1d0 con 0x7f17dc06b4f0 2023-10-06T03:31:35.194 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:35.187+0000 7f17e157c700 1 -- --> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17dc0cb4b0 con 0x7f17dc15d0a0 2023-10-06T03:31:35.194 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:35.187+0000 7f17da7fc700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f17dc0c86d0 0x7f17dc0c9f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:31:35.194 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:35.187+0000 7f17da7fc700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f17dc0c86d0 0x7f17dc0c9f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:43902/0 (socket says 172.21.15.110:43902) 2023-10-06T03:31:35.194 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:35.187+0000 7f17da7fc700 1 -- 172.21.15.110:0/1395937622 learned_addr learned my addr 172.21.15.110:0/1395937622 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-06T03:31:35.194 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:35.187+0000 7f17daffd700 1 --2- 172.21.15.110:0/1395937622 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f17dc06b4f0 0x7f17dc0ba670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:31:35.195 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:35.187+0000 7f17db7fe700 1 --2- 172.21.15.110:0/1395937622 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f17dc15d0a0 0x7f17dc14cea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:31:35.195 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:35.187+0000 7f17da7fc700 1 -- 172.21.15.110:0/1395937622 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f17dc15d0a0 msgr2=0x7f17dc14cea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:31:35.195 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:35.187+0000 7f17da7fc700 1 --2- 172.21.15.110:0/1395937622 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f17dc15d0a0 0x7f17dc14cea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:35.195 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:35.187+0000 7f17da7fc700 1 -- 172.21.15.110:0/1395937622 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f17dc06b4f0 msgr2=0x7f17dc0ba670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:31:35.195 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:35.187+0000 7f17da7fc700 1 --2- 172.21.15.110:0/1395937622 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f17dc06b4f0 0x7f17dc0ba670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:35.195 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:35.187+0000 7f17da7fc700 1 -- 172.21.15.110:0/1395937622 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17cc009420 con 0x7f17dc0c86d0 2023-10-06T03:31:35.195 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:35.187+0000 7f17db7fe700 1 --2- 172.21.15.110:0/1395937622 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f17dc15d0a0 0x7f17dc14cea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-06T03:31:35.195 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:35.187+0000 7f17da7fc700 1 --2- 172.21.15.110:0/1395937622 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f17dc0c86d0 0x7f17dc0c9f10 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f17d400cc70 tx=0x7f17d40019f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:31:35.195 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:35.187+0000 7f17c2ffd700 1 -- 172.21.15.110:0/1395937622 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17d400a8e0 con 0x7f17dc0c86d0 2023-10-06T03:31:35.195 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:35.187+0000 7f17c2ffd700 1 -- 172.21.15.110:0/1395937622 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7f17d400aa60 con 0x7f17dc0c86d0 2023-10-06T03:31:35.196 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:35.187+0000 7f17e157c700 1 -- 172.21.15.110:0/1395937622 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17d0009420 con 0x7f17dc0c86d0 2023-10-06T03:31:35.196 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:35.187+0000 7f17c2ffd700 1 -- 172.21.15.110:0/1395937622 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17d400ae40 con 0x7f17dc0c86d0 2023-10-06T03:31:35.196 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:35.187+0000 7f17e157c700 1 -- 172.21.15.110:0/1395937622 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17dc0c99f0 con 0x7f17dc0c86d0 2023-10-06T03:31:35.196 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:35.187+0000 7f17e157c700 1 -- 172.21.15.110:0/1395937622 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17dc064770 con 0x7f17dc0c86d0 2023-10-06T03:31:35.197 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:35.191+0000 7f17c2ffd700 1 -- 172.21.15.110:0/1395937622 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 13) v1 ==== 80061+0+0 (secure 0 0 0) 0x7f17d4007300 con 0x7f17dc0c86d0 2023-10-06T03:31:35.197 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:35.191+0000 7f17c2ffd700 1 --2- 172.21.15.110:0/1395937622 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f17c80606a0 0x7f17c8062b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T03:31:35.197 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:35.191+0000 7f17c2ffd700 1 -- 172.21.15.110:0/1395937622 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 6171+0+0 (secure 0 0 0) 0x7f17d407cea0 con 0x7f17dc0c86d0 2023-10-06T03:31:35.197 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:35.191+0000 7f17daffd700 1 --2- 172.21.15.110:0/1395937622 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f17c80606a0 0x7f17c8062b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T03:31:35.198 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:35.191+0000 7f17daffd700 1 --2- 172.21.15.110:0/1395937622 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f17c80606a0 0x7f17c8062b50 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f17d000a010 tx=0x7f17d00091a0 comp rx=0 tx=0).ready entity=mgr.5505 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T03:31:35.205 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:35.199+0000 7f17c2ffd700 1 -- 172.21.15.110:0/1395937622 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166856 (secure 0 0 0) 0x7f17d404d210 con 0x7f17dc0c86d0 2023-10-06T03:31:35.395 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:35.387+0000 7f17e157c700 1 -- 172.21.15.110:0/1395937622 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f17dc0abfb0 con 0x7f17dc0c86d0 2023-10-06T03:31:35.396 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:35.391+0000 7f17c2ffd700 1 -- 172.21.15.110:0/1395937622 <== mon.0 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 18 v18) v1 ==== 94+0+3535 (secure 0 0 0) 0x7f17d4050830 con 0x7f17dc0c86d0 2023-10-06T03:31:35.396 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-10-06T03:31:35.396 INFO:teuthology.orchestra.run.smithi110.stdout:{"epoch":18,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5107,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6834/884349347","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6833","nonce":884349347},{"type":"v1","addr":"172.21.15.192:6834","nonce":884349347}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":5110,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6835/4077204152","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6832","nonce":4077204152},{"type":"v1","addr":"172.21.15.192:6835","nonce":4077204152}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":5160,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/504583843","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6834","nonce":504583843},{"type":"v1","addr":"172.21.15.110:6835","nonce":504583843}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":5622,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6837/2027378146","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6836","nonce":2027378146},{"type":"v1","addr":"172.21.15.110:6837","nonce":2027378146}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":18}],"filesystems":[]} 2023-10-06T03:31:35.399 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:35.391+0000 7f17e157c700 1 -- 172.21.15.110:0/1395937622 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f17c80606a0 msgr2=0x7f17c8062b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:31:35.399 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:35.391+0000 7f17e157c700 1 --2- 172.21.15.110:0/1395937622 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f17c80606a0 0x7f17c8062b50 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f17d000a010 tx=0x7f17d00091a0 comp rx=0 tx=0).stop 2023-10-06T03:31:35.399 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:35.391+0000 7f17e157c700 1 -- 172.21.15.110:0/1395937622 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f17dc0c86d0 msgr2=0x7f17dc0c9f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T03:31:35.399 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:35.391+0000 7f17e157c700 1 --2- 172.21.15.110:0/1395937622 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f17dc0c86d0 0x7f17dc0c9f10 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f17d400cc70 tx=0x7f17d40019f0 comp rx=0 tx=0).stop 2023-10-06T03:31:35.400 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:35.391+0000 7f17e157c700 1 -- 172.21.15.110:0/1395937622 shutdown_connections 2023-10-06T03:31:35.400 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:35.391+0000 7f17e157c700 1 --2- 172.21.15.110:0/1395937622 >> [v2:172.21.15.110:6832/33268,v1:172.21.15.110:6833/33268] conn(0x7f17c80606a0 0x7f17c8062b50 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:35.400 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:35.391+0000 7f17e157c700 1 --2- 172.21.15.110:0/1395937622 >> [v2:172.21.15.192:3301/0,v1:172.21.15.192:6790/0] conn(0x7f17dc15d0a0 0x7f17dc14cea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:35.400 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:35.391+0000 7f17e157c700 1 --2- 172.21.15.110:0/1395937622 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f17dc0c86d0 0x7f17dc0c9f10 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:35.400 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:35.391+0000 7f17e157c700 1 --2- 172.21.15.110:0/1395937622 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f17dc06b4f0 0x7f17dc0ba670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T03:31:35.400 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:35.391+0000 7f17e157c700 1 -- 172.21.15.110:0/1395937622 >> 172.21.15.110:0/1395937622 conn(0x7f17dc067040 msgr2=0x7f17dc0aec10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T03:31:35.400 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:35.391+0000 7f17e157c700 1 -- 172.21.15.110:0/1395937622 shutdown_connections 2023-10-06T03:31:35.400 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-06T03:31:35.391+0000 7f17e157c700 1 -- 172.21.15.110:0/1395937622 wait complete. 2023-10-06T03:31:35.401 INFO:teuthology.orchestra.run.smithi110.stderr:dumped fsmap epoch 18 2023-10-06T03:31:35.410 DEBUG:tasks.cephfs.filesystem:already dead... 2023-10-06T03:31:35.410 INFO:teuthology.misc:Shutting down mds daemons... 2023-10-06T03:31:35.410 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-10-06T03:31:35.410 INFO:teuthology.orchestra.run:waiting for 300 2023-10-06T03:31:35.425 INFO:tasks.ceph.mds.a:Stopped 2023-10-06T03:31:35.426 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-10-06T03:31:35.426 INFO:teuthology.orchestra.run:waiting for 300 2023-10-06T03:31:35.441 INFO:tasks.ceph.mds.c:Stopped 2023-10-06T03:31:35.441 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2023-10-06T03:31:35.441 INFO:teuthology.orchestra.run:waiting for 300 2023-10-06T03:31:35.457 INFO:tasks.ceph.mds.b:Stopped 2023-10-06T03:31:35.458 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2023-10-06T03:31:35.458 INFO:teuthology.orchestra.run:waiting for 300 2023-10-06T03:31:35.469 INFO:tasks.ceph.mds.d:Stopped 2023-10-06T03:31:35.469 INFO:teuthology.misc:Shutting down osd daemons... 2023-10-06T03:31:35.470 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2023-10-06T03:31:35.470 INFO:teuthology.orchestra.run:waiting for 300 2023-10-06T03:31:35.537 INFO:tasks.ceph.osd.0:Stopped 2023-10-06T03:31:35.537 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2023-10-06T03:31:35.537 INFO:teuthology.orchestra.run:waiting for 300 2023-10-06T03:31:35.612 INFO:tasks.ceph.osd.1:Stopped 2023-10-06T03:31:35.613 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2023-10-06T03:31:35.613 INFO:teuthology.orchestra.run:waiting for 300 2023-10-06T03:31:35.696 INFO:tasks.ceph.osd.2:Stopped 2023-10-06T03:31:35.696 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2023-10-06T03:31:35.696 INFO:teuthology.orchestra.run:waiting for 300 2023-10-06T03:31:35.762 INFO:tasks.ceph.osd.3:Stopped 2023-10-06T03:31:35.762 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2023-10-06T03:31:35.763 INFO:teuthology.orchestra.run:waiting for 300 2023-10-06T03:31:35.847 INFO:tasks.ceph.osd.4:Stopped 2023-10-06T03:31:35.847 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2023-10-06T03:31:35.847 INFO:teuthology.orchestra.run:waiting for 300 2023-10-06T03:31:35.925 INFO:tasks.ceph.osd.5:Stopped 2023-10-06T03:31:35.925 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2023-10-06T03:31:35.925 INFO:teuthology.orchestra.run:waiting for 300 2023-10-06T03:31:35.988 INFO:tasks.ceph.osd.6:Stopped 2023-10-06T03:31:35.988 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2023-10-06T03:31:35.988 INFO:teuthology.orchestra.run:waiting for 300 2023-10-06T03:31:36.070 INFO:tasks.ceph.osd.7:Stopped 2023-10-06T03:31:36.070 INFO:teuthology.misc:Shutting down mgr daemons... 2023-10-06T03:31:36.070 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2023-10-06T03:31:36.070 INFO:teuthology.orchestra.run:waiting for 300 2023-10-06T03:31:36.104 INFO:tasks.ceph.mgr.y:Stopped 2023-10-06T03:31:36.105 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2023-10-06T03:31:36.105 INFO:teuthology.orchestra.run:waiting for 300 2023-10-06T03:31:36.131 INFO:tasks.ceph.mgr.x:Stopped 2023-10-06T03:31:36.131 INFO:teuthology.misc:Shutting down mon daemons... 2023-10-06T03:31:36.131 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2023-10-06T03:31:36.132 INFO:teuthology.orchestra.run:waiting for 300 2023-10-06T03:31:36.151 INFO:tasks.ceph.mon.a:Stopped 2023-10-06T03:31:36.151 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2023-10-06T03:31:36.151 INFO:teuthology.orchestra.run:waiting for 300 2023-10-06T03:31:36.193 INFO:tasks.ceph.mon.b:Stopped 2023-10-06T03:31:36.193 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2023-10-06T03:31:36.193 INFO:teuthology.orchestra.run:waiting for 300 2023-10-06T03:31:36.211 INFO:tasks.ceph.mon.c:Stopped 2023-10-06T03:31:36.211 INFO:tasks.ceph:Checking cluster log for badness... 2023-10-06T03:31:36.211 DEBUG:teuthology.orchestra.run.smithi110:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'overall HEALTH_' | egrep -v '\(FS_DEGRADED\)' | egrep -v '\(MDS_FAILED\)' | egrep -v '\(MDS_DEGRADED\)' | egrep -v '\(FS_WITH_FAILED_MDS\)' | egrep -v '\(MDS_DAMAGE\)' | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(FS_INLINE_DATA_DEPRECATED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | head -n 1 2023-10-06T03:31:36.211 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~0s 2023-10-06T03:31:36.211 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~0s 2023-10-06T03:31:36.230 INFO:teuthology.orchestra.run.smithi110.stdout:1696562969.3603275 mon.a (mon.0) 209 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2023-10-06T03:31:36.230 WARNING:tasks.ceph:Found errors (ERR|WRN|SEC) in cluster log 2023-10-06T03:31:36.231 DEBUG:teuthology.orchestra.run.smithi110:> sudo egrep '\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'overall HEALTH_' | egrep -v '\(FS_DEGRADED\)' | egrep -v '\(MDS_FAILED\)' | egrep -v '\(MDS_DEGRADED\)' | egrep -v '\(FS_WITH_FAILED_MDS\)' | egrep -v '\(MDS_DAMAGE\)' | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(FS_INLINE_DATA_DEPRECATED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | head -n 1 2023-10-06T03:31:36.243 DEBUG:teuthology.orchestra.run.smithi110:> sudo egrep '\[ERR\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'overall HEALTH_' | egrep -v '\(FS_DEGRADED\)' | egrep -v '\(MDS_FAILED\)' | egrep -v '\(MDS_DEGRADED\)' | egrep -v '\(FS_WITH_FAILED_MDS\)' | egrep -v '\(MDS_DAMAGE\)' | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(FS_INLINE_DATA_DEPRECATED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | head -n 1 2023-10-06T03:31:36.256 DEBUG:teuthology.orchestra.run.smithi110:> sudo egrep '\[WRN\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'overall HEALTH_' | egrep -v '\(FS_DEGRADED\)' | egrep -v '\(MDS_FAILED\)' | egrep -v '\(MDS_DEGRADED\)' | egrep -v '\(FS_WITH_FAILED_MDS\)' | egrep -v '\(MDS_DAMAGE\)' | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(FS_INLINE_DATA_DEPRECATED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | head -n 1 2023-10-06T03:31:36.317 INFO:teuthology.orchestra.run.smithi110.stdout:1696562969.3603275 mon.a (mon.0) 209 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2023-10-06T03:31:36.318 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi110.front.sepia.ceph.com 2023-10-06T03:31:36.318 DEBUG:teuthology.orchestra.run.smithi110:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2023-10-06T03:31:36.633 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi110.front.sepia.ceph.com 2023-10-06T03:31:36.633 DEBUG:teuthology.orchestra.run.smithi110:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2023-10-06T03:31:36.848 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi110.front.sepia.ceph.com 2023-10-06T03:31:36.848 DEBUG:teuthology.orchestra.run.smithi110:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2023-10-06T03:31:37.039 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi110.front.sepia.ceph.com 2023-10-06T03:31:37.039 DEBUG:teuthology.orchestra.run.smithi110:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2023-10-06T03:31:37.215 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi192.front.sepia.ceph.com 2023-10-06T03:31:37.216 DEBUG:teuthology.orchestra.run.smithi192:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2023-10-06T03:31:37.472 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi192.front.sepia.ceph.com 2023-10-06T03:31:37.473 DEBUG:teuthology.orchestra.run.smithi192:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2023-10-06T03:31:37.697 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi192.front.sepia.ceph.com 2023-10-06T03:31:37.697 DEBUG:teuthology.orchestra.run.smithi192:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2023-10-06T03:31:37.874 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi192.front.sepia.ceph.com 2023-10-06T03:31:37.874 DEBUG:teuthology.orchestra.run.smithi192:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2023-10-06T03:31:38.029 DEBUG:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-06T03:31:38.031 DEBUG:teuthology.orchestra.run.smithi192:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-06T03:31:38.067 INFO:tasks.ceph:Archiving mon data... 2023-10-06T03:31:38.068 DEBUG:teuthology.misc:Transferring archived files from smithi110:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/yuriw-2023-10-05_23:52:50-fs-wip-yuri-testing-2023-10-05-0730-quincy-distro-default-smithi/7412204/data/mon.a.tgz 2023-10-06T03:31:38.068 DEBUG:teuthology.orchestra.run.smithi110:> mktemp 2023-10-06T03:31:38.075 INFO:teuthology.orchestra.run.smithi110.stdout:/tmp/tmp.CAMoE7s4Wo 2023-10-06T03:31:38.075 DEBUG:teuthology.orchestra.run.smithi110:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.CAMoE7s4Wo 2023-10-06T03:31:38.241 DEBUG:teuthology.orchestra.run.smithi110:> sudo chmod 0666 /tmp/tmp.CAMoE7s4Wo 2023-10-06T03:31:38.267 DEBUG:teuthology.orchestra.remote:smithi110:/tmp/tmp.CAMoE7s4Wo is 601KB 2023-10-06T03:31:38.335 DEBUG:teuthology.orchestra.run.smithi110:> rm -fr /tmp/tmp.CAMoE7s4Wo 2023-10-06T03:31:38.342 DEBUG:teuthology.misc:Transferring archived files from smithi192:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/yuriw-2023-10-05_23:52:50-fs-wip-yuri-testing-2023-10-05-0730-quincy-distro-default-smithi/7412204/data/mon.b.tgz 2023-10-06T03:31:38.343 DEBUG:teuthology.orchestra.run.smithi192:> mktemp 2023-10-06T03:31:38.349 INFO:teuthology.orchestra.run.smithi192.stdout:/tmp/tmp.dTmbxlmPLN 2023-10-06T03:31:38.350 DEBUG:teuthology.orchestra.run.smithi192:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.dTmbxlmPLN 2023-10-06T03:31:38.511 DEBUG:teuthology.orchestra.run.smithi192:> sudo chmod 0666 /tmp/tmp.dTmbxlmPLN 2023-10-06T03:31:38.577 DEBUG:teuthology.orchestra.remote:smithi192:/tmp/tmp.dTmbxlmPLN is 850KB 2023-10-06T03:31:38.607 DEBUG:teuthology.orchestra.run.smithi192:> rm -fr /tmp/tmp.dTmbxlmPLN 2023-10-06T03:31:38.613 DEBUG:teuthology.misc:Transferring archived files from smithi192:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/yuriw-2023-10-05_23:52:50-fs-wip-yuri-testing-2023-10-05-0730-quincy-distro-default-smithi/7412204/data/mon.c.tgz 2023-10-06T03:31:38.614 DEBUG:teuthology.orchestra.run.smithi192:> mktemp 2023-10-06T03:31:38.660 INFO:teuthology.orchestra.run.smithi192.stdout:/tmp/tmp.1RJiocOTBg 2023-10-06T03:31:38.660 DEBUG:teuthology.orchestra.run.smithi192:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.1RJiocOTBg 2023-10-06T03:31:38.821 DEBUG:teuthology.orchestra.run.smithi192:> sudo chmod 0666 /tmp/tmp.1RJiocOTBg 2023-10-06T03:31:38.838 DEBUG:teuthology.orchestra.remote:smithi192:/tmp/tmp.1RJiocOTBg is 850KB 2023-10-06T03:31:38.908 DEBUG:teuthology.orchestra.run.smithi192:> rm -fr /tmp/tmp.1RJiocOTBg 2023-10-06T03:31:38.915 INFO:tasks.ceph:Cleaning ceph cluster... 2023-10-06T03:31:38.915 DEBUG:teuthology.orchestra.run.smithi110:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2023-10-06T03:31:38.917 DEBUG:teuthology.orchestra.run.smithi192:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2023-10-06T03:31:38.933 INFO:tasks.ceph:Checking for errors in any valgrind logs... 2023-10-06T03:31:38.933 DEBUG:teuthology.orchestra.run.smithi110:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2023-10-06T03:31:38.935 DEBUG:teuthology.orchestra.run.smithi192:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2023-10-06T03:31:38.953 INFO:teuthology.orchestra.run.smithi110.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2023-10-06T03:31:38.955 INFO:teuthology.orchestra.run.smithi192.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2023-10-06T03:31:38.961 INFO:tasks.ceph:Archiving crash dumps... 2023-10-06T03:31:38.963 DEBUG:teuthology.misc:Transferring archived files from smithi110:/var/lib/ceph/crash to /home/teuthworker/archive/yuriw-2023-10-05_23:52:50-fs-wip-yuri-testing-2023-10-05-0730-quincy-distro-default-smithi/7412204/remote/smithi110/crash 2023-10-06T03:31:38.964 DEBUG:teuthology.orchestra.run.smithi110:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2023-10-06T03:31:39.020 DEBUG:teuthology.misc:Transferring archived files from smithi192:/var/lib/ceph/crash to /home/teuthworker/archive/yuriw-2023-10-05_23:52:50-fs-wip-yuri-testing-2023-10-05-0730-quincy-distro-default-smithi/7412204/remote/smithi192/crash 2023-10-06T03:31:39.021 DEBUG:teuthology.orchestra.run.smithi192:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2023-10-06T03:31:39.038 INFO:tasks.ceph:Shutting down logrotate 2023-10-06T03:31:39.038 DEBUG:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-06T03:31:39.040 DEBUG:teuthology.orchestra.run.smithi192:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-06T03:31:39.085 DEBUG:teuthology.orchestra.run.smithi110:> sudo rm /etc/logrotate.d/ceph-test.conf 2023-10-06T03:31:39.100 DEBUG:teuthology.orchestra.run.smithi192:> sudo rm /etc/logrotate.d/ceph-test.conf 2023-10-06T03:31:39.115 INFO:tasks.ceph:Compressing logs... 2023-10-06T03:31:39.116 DEBUG:teuthology.orchestra.run.smithi110:> sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-10-06T03:31:39.118 DEBUG:teuthology.orchestra.run.smithi192:> sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-10-06T03:31:41.213 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b has been restored 2023-10-06T03:31:41.213 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c has been restored 2023-10-06T03:31:42.575 INFO:tasks.ceph:Archiving logs... 2023-10-06T03:31:42.575 DEBUG:teuthology.misc:Transferring archived files from smithi110:/var/log/ceph to /home/teuthworker/archive/yuriw-2023-10-05_23:52:50-fs-wip-yuri-testing-2023-10-05-0730-quincy-distro-default-smithi/7412204/remote/smithi110/log 2023-10-06T03:31:42.576 DEBUG:teuthology.orchestra.run.smithi110:> sudo tar c -f - -C /var/log/ceph -- . 2023-10-06T03:31:43.220 DEBUG:teuthology.misc:Transferring archived files from smithi192:/var/log/ceph to /home/teuthworker/archive/yuriw-2023-10-05_23:52:50-fs-wip-yuri-testing-2023-10-05-0730-quincy-distro-default-smithi/7412204/remote/smithi192/log 2023-10-06T03:31:43.221 DEBUG:teuthology.orchestra.run.smithi192:> sudo tar c -f - -C /var/log/ceph -- . 2023-10-06T03:31:43.597 DEBUG:teuthology.run_tasks:Unwinding manager install 2023-10-06T03:31:43.606 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2023-10-06T03:31:43.607 DEBUG:teuthology.orchestra.run.smithi110:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-10-06T03:31:43.609 DEBUG:teuthology.orchestra.run.smithi192:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-10-06T03:31:43.688 DEBUG:teuthology.run_tasks:Unwinding manager clock 2023-10-06T03:31:43.699 INFO:teuthology.task.clock:Checking final clock skew... 2023-10-06T03:31:43.699 DEBUG:teuthology.orchestra.run.smithi110:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-10-06T03:31:43.701 DEBUG:teuthology.orchestra.run.smithi192:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-10-06T03:31:43.725 INFO:teuthology.orchestra.run.smithi110.stdout: remote refid st t when poll reach delay offset jitter 2023-10-06T03:31:43.725 INFO:teuthology.orchestra.run.smithi110.stdout:============================================================================== 2023-10-06T03:31:43.725 INFO:teuthology.orchestra.run.smithi110.stdout:*hv01.front.sepi 44.4.53.2 2 u 20 64 77 0.117 1.243 0.805 2023-10-06T03:31:43.725 INFO:teuthology.orchestra.run.smithi110.stdout:+hv02.front.sepi 63.231.80.2 3 u 22 64 77 0.070 0.772 0.646 2023-10-06T03:31:43.725 INFO:teuthology.orchestra.run.smithi110.stdout:+hv03.front.sepi 74.6.168.72 3 u 25 64 77 0.105 0.676 0.832 2023-10-06T03:31:43.725 INFO:teuthology.orchestra.run.smithi110.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2023-10-06T03:31:43.727 INFO:teuthology.orchestra.run.smithi192.stdout: remote refid st t when poll reach delay offset jitter 2023-10-06T03:31:43.727 INFO:teuthology.orchestra.run.smithi192.stdout:============================================================================== 2023-10-06T03:31:43.727 INFO:teuthology.orchestra.run.smithi192.stdout:*hv01.front.sepi 44.4.53.2 2 u 25 64 77 0.073 0.432 0.169 2023-10-06T03:31:43.727 INFO:teuthology.orchestra.run.smithi192.stdout:+hv02.front.sepi 63.231.80.2 3 u 23 64 77 0.069 0.737 0.363 2023-10-06T03:31:43.727 INFO:teuthology.orchestra.run.smithi192.stdout:+hv03.front.sepi 74.6.168.72 3 u 26 64 77 0.105 -0.044 0.430 2023-10-06T03:31:43.727 INFO:teuthology.orchestra.run.smithi192.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2023-10-06T03:31:43.728 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2023-10-06T03:31:43.738 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2023-10-06T03:31:43.739 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2023-10-06T03:31:43.747 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2023-10-06T03:31:43.756 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2023-10-06T03:31:43.771 INFO:teuthology.task.internal:Duration was 706.690440 seconds 2023-10-06T03:31:43.771 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2023-10-06T03:31:43.780 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2023-10-06T03:31:43.781 DEBUG:teuthology.orchestra.run.smithi110:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-10-06T03:31:43.783 DEBUG:teuthology.orchestra.run.smithi192:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-10-06T03:31:43.835 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2023-10-06T03:31:43.836 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi110.front.sepia.ceph.com 2023-10-06T03:31:43.836 DEBUG:teuthology.orchestra.run.smithi110:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-10-06T03:31:43.848 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi192.front.sepia.ceph.com 2023-10-06T03:31:43.848 DEBUG:teuthology.orchestra.run.smithi192:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-10-06T03:31:43.860 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2023-10-06T03:31:43.860 DEBUG:teuthology.orchestra.run.smithi110:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-10-06T03:31:43.862 DEBUG:teuthology.orchestra.run.smithi192:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-10-06T03:31:43.891 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2023-10-06T03:31:43.891 DEBUG:teuthology.orchestra.run.smithi110:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-10-06T03:31:43.893 DEBUG:teuthology.orchestra.run.smithi192:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-10-06T03:31:44.000 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2023-10-06T03:31:44.010 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2023-10-06T03:31:44.010 DEBUG:teuthology.orchestra.run.smithi110:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-10-06T03:31:44.025 DEBUG:teuthology.orchestra.run.smithi192:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-10-06T03:31:44.041 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2023-10-06T03:31:44.054 DEBUG:teuthology.orchestra.run.smithi110:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-10-06T03:31:44.056 DEBUG:teuthology.orchestra.run.smithi192:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-10-06T03:31:44.069 INFO:teuthology.orchestra.run.smithi110.stdout:kernel.core_pattern = core 2023-10-06T03:31:44.070 INFO:teuthology.orchestra.run.smithi192.stdout:kernel.core_pattern = core 2023-10-06T03:31:44.085 DEBUG:teuthology.orchestra.run.smithi110:> test -e /home/ubuntu/cephtest/archive/coredump 2023-10-06T03:31:44.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-06T03:31:44.091 DEBUG:teuthology.orchestra.run.smithi192:> test -e /home/ubuntu/cephtest/archive/coredump 2023-10-06T03:31:44.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-06T03:31:44.169 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2023-10-06T03:31:44.183 INFO:teuthology.task.internal:Transferring archived files... 2023-10-06T03:31:44.184 DEBUG:teuthology.misc:Transferring archived files from smithi110:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2023-10-05_23:52:50-fs-wip-yuri-testing-2023-10-05-0730-quincy-distro-default-smithi/7412204/remote/smithi110 2023-10-06T03:31:44.184 DEBUG:teuthology.orchestra.run.smithi110:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-10-06T03:31:44.210 DEBUG:teuthology.misc:Transferring archived files from smithi192:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2023-10-05_23:52:50-fs-wip-yuri-testing-2023-10-05-0730-quincy-distro-default-smithi/7412204/remote/smithi192 2023-10-06T03:31:44.210 DEBUG:teuthology.orchestra.run.smithi192:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-10-06T03:31:44.234 INFO:teuthology.task.internal:Removing archive directory... 2023-10-06T03:31:44.234 DEBUG:teuthology.orchestra.run.smithi110:> rm -rf -- /home/ubuntu/cephtest/archive 2023-10-06T03:31:44.253 DEBUG:teuthology.orchestra.run.smithi192:> rm -rf -- /home/ubuntu/cephtest/archive 2023-10-06T03:31:44.260 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2023-10-06T03:31:44.272 INFO:teuthology.task.internal:Not uploading archives. 2023-10-06T03:31:44.272 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2023-10-06T03:31:44.283 INFO:teuthology.task.internal:Tidying up after the test... 2023-10-06T03:31:44.283 DEBUG:teuthology.orchestra.run.smithi110:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-10-06T03:31:44.286 DEBUG:teuthology.orchestra.run.smithi192:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-10-06T03:31:44.291 INFO:teuthology.orchestra.run.smithi110.stdout: 924112 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Oct 6 03:31 /home/ubuntu/cephtest 2023-10-06T03:31:44.309 INFO:teuthology.orchestra.run.smithi192.stdout: 924112 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Oct 6 03:31 /home/ubuntu/cephtest 2023-10-06T03:31:44.311 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2023-10-06T03:31:44.322 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2023-10-06T03:31:44.345 INFO:teuthology.nuke:Checking targets against current locks 2023-10-06T03:31:44.371 DEBUG:teuthology.nuke:shortname: smithi110 2023-10-06T03:31:44.372 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-10-06T03:31:44.395 DEBUG:teuthology.nuke:shortname: smithi192 2023-10-06T03:31:44.395 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-10-06T03:31:44.398 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi110.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-10-05_23:52:50-fs-wip-yuri-testing-2023-10-05-0730-quincy-distro-default-smithi/7412204', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '20.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-10-06 03:11:12.020242', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFyT/r7xuKfmg50w0OdQebr4A/ptJo1Zk6DyZNZNZM/q5rW+sznh+wvHqeT3e4YtCdZLaXyvMms8o8u8H5nXOYM='} 2023-10-06T03:31:44.418 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi192.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-10-05_23:52:50-fs-wip-yuri-testing-2023-10-05-0730-quincy-distro-default-smithi/7412204', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '20.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-10-06 03:11:12.018725', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFyT/r7xuKfmg50w0OdQebr4A/ptJo1Zk6DyZNZNZM/q5rW+sznh+wvHqeT3e4YtCdZLaXyvMms8o8u8H5nXOYM='} 2023-10-06T03:31:44.452 INFO:teuthology.orchestra.console.smithi110:Power off 2023-10-06T03:31:44.453 DEBUG:teuthology.orchestra.console.smithi110:pexpect command: ipmitool -H smithi110.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-10-06T03:31:44.471 INFO:teuthology.orchestra.console.smithi192:Power off 2023-10-06T03:31:44.471 DEBUG:teuthology.orchestra.console.smithi192:pexpect command: ipmitool -H smithi192.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-10-06T03:31:44.481 DEBUG:teuthology.orchestra.console.smithi110:power off output: Chassis Power Control: Down/Off 2023-10-06T03:31:44.481 DEBUG:teuthology.orchestra.console.smithi110:pexpect command: ipmitool -H smithi110.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-06T03:31:44.495 DEBUG:teuthology.orchestra.console.smithi192:power off output: Chassis Power Control: Down/Off 2023-10-06T03:31:44.496 DEBUG:teuthology.orchestra.console.smithi192:pexpect command: ipmitool -H smithi192.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-06T03:31:44.507 DEBUG:teuthology.orchestra.console.smithi110:check power output: Chassis Power is on 2023-10-06T03:31:44.518 DEBUG:teuthology.orchestra.console.smithi192:check power output: Chassis Power is on 2023-10-06T03:31:48.508 DEBUG:teuthology.orchestra.console.smithi110:pexpect command: ipmitool -H smithi110.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-06T03:31:48.519 DEBUG:teuthology.orchestra.console.smithi192:pexpect command: ipmitool -H smithi192.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-06T03:31:48.618 DEBUG:teuthology.orchestra.console.smithi110:check power output: Chassis Power is on 2023-10-06T03:31:48.628 DEBUG:teuthology.orchestra.console.smithi192:check power output: Chassis Power is on 2023-10-06T03:31:52.620 DEBUG:teuthology.orchestra.console.smithi110:pexpect command: ipmitool -H smithi110.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-06T03:31:52.628 DEBUG:teuthology.orchestra.console.smithi192:pexpect command: ipmitool -H smithi192.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-06T03:31:52.729 DEBUG:teuthology.orchestra.console.smithi110:check power output: Chassis Power is on 2023-10-06T03:31:52.737 DEBUG:teuthology.orchestra.console.smithi192:check power output: Chassis Power is on 2023-10-06T03:31:56.731 DEBUG:teuthology.orchestra.console.smithi110:pexpect command: ipmitool -H smithi110.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-06T03:31:56.738 DEBUG:teuthology.orchestra.console.smithi192:pexpect command: ipmitool -H smithi192.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-06T03:31:56.840 DEBUG:teuthology.orchestra.console.smithi110:check power output: Chassis Power is off 2023-10-06T03:31:56.845 DEBUG:teuthology.orchestra.console.smithi192:check power output: Chassis Power is off 2023-10-06T03:31:56.942 INFO:teuthology.orchestra.console.smithi110:Power off completed 2023-10-06T03:31:56.946 INFO:teuthology.orchestra.console.smithi192:Power off completed 2023-10-06T03:31:57.051 INFO:teuthology.run:Summary data: description: fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/snap_schedule_snapdir} duration: 706.6904401779175 failure_reason: '"1696562969.3603275 mon.a (mon.0) 209 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log' flavor: default owner: scheduled_yuriw@teuthology success: false 2023-10-06T03:31:57.051 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-10-06T03:31:57.152 INFO:teuthology.run:FAIL